The Design & Architecture of Smallstep

This is a technical design document for readers that are already familiar with public key infrastructure (PKI) and certificate management and want to get up to speed with step certificates quickly.

For readers who are less familiar check out...

About the step toolchain

step is the command-line interface for all things smallstep, and a swiss-army knife for day-to-day Production Identity operations.

step is a command-line tool for developers, operators, and security professionals to:

It's the open source core of everything smallstep does, and everything you need to work with certificates, tokens, JOSE structures (JWT, JWK, JWE, JWS), and many other common cryptography primitives.

step Characteristics

Things you can do with step

Step certificates extends the step command line tool, adding a certificate authority ( step-ca ) and subcommands to make secure automated certificate management easy. The goal is to reduce the cost of building an internal PKI to cryptographically identify people, devices, and production workloads (e.g., microservices) and use TLS.

While end-to-end cryptography and mutual authentication between services is valuable, it's too hard with legacy tools. This difficulty is not inherent -- PKI is not necessarily hard. But there is an infrastructure and tooling gap that makes internal PKI challenging.Step certificates fills that gap.

Step certificates has two core components:

Both of these programs are written in golang and use the excellent crypto and x509 libraries from the golang standard library. They also use square's go-jose library for JOSE primitives (JWT & JWK).

These tools make the arcane art of automated certificate management a simple, intuitive process. With step certificates you should be able to setup your own internal CA and automate the delivery of certificates to your production workloads in an afternoon.

Key Features

Installing

Installing step and step-ca is easy. If you're on macOS you can simply

For Linux installation and custom builds see the installation instructions in the step and step certificates repositories.

If you're using Docker or Kubernetes we also have pre-built containers and helm charts available.

We're not currently testing step and step-ca on Microsoft operating systems, but they should mostly work (though you will need to build the binaries yourself). We are interested in issues if you encounter any.

Overview & Design

Let's consider a simple concrete example: using step certificates to issue certificates to two microservices so they can communicate securely using HTTPS.

Assume our two services -- foo.internal and bar.internal -- currently communicate using plain HTTP:

Our goal is to switch to mutually authenticated HTTPS (HTTP over mutual TLS). Most proxies and standard library HTTP stacks support mutual HTTPS. But each service will need three artifacts to speak HTTPS:

The certificate and private key are credentials used for authentication. The root certificate verifies these credentials.

Once everything is deployed our system will look like:

Let's use step certificates ( step and step-ca ) to generate and deploy these artifacts.

Creating PKI artifacts and starting step-ca

Before we can deploy a root certificate we need to create one, along with a corresponding private key. The step ca init command does this. It also generates a configuration file for step-ca and registers a provisioner (discussed in the next section):

This command generates all of the PKI artifacts and configuration files needed to run step-ca :

The --name flag specifies a human-readable name for your CA infrastructure. It forms part of the subject name in your newly generated root and intermediate certificates:

The use of an intermediate certificate and signing key is a PKI best practice that enables certain advanced PKI use cases. You can safely ignore this aspect of the system until you need it: the key generation and bundling required to make intermediates work is handled for you by step and step-ca .

It's important to keep the root and intermediate signing keys secure. Both keys are encrypted on disk using a password read from --password-file (if this flag isn't provided you'll be prompted for a password). You can change the password on these keys using step crypto change-pass .

The root signing key is not actually needed for step-ca to operate. You can remove it from $(step path)/secrets/root_ca_key and keep it securely offline, if you'd prefer.

The --dns flag specifies the DNS host name for the CA. The CA exposes an HTTPS API and issues itself an end-entity certificate using this name. The CA listens on the IP address and port specified by the --address flag.

Note: you'll need to configure ca.internal to resolve to the right IP address in DNS. You can use localhost as your CA's DNS name for testing.

We're ready to start step-ca :

Note: step path is a helper command that outputs the filesystem path used to store step configuration files, certificates, and keys. This will default to ~/.step but can be configured using the STEPPATH environment variable.

Automated certificate management

The step command line utility integrates with the step-ca HTTPS APIs to help:

Distributing your Root Certificate

Like any other TLS/HTTPS client, step needs your CA's root certificate to verify issued certificates and establish secure connections. This presents a challenge: we'd like to use HTTPS to connect to the CA and obtain our root certificate, but we need the root certificate to establish this secure connection.

When we ran step ca init our root certificate fingerprint was printed. This fingerprint is a hash over our root certificate. step can use this fingerprint to securely obtain the root certificate from the CA:

Note: you can regenerate this fingerprint by running step certificate fingerprint $(step path)/certs/root_ca.crt on your CA.

Technically, the step ca root command proceeds by making an HTTPS request to the CA with certificate path validation turned off. This is usually insecure. However, step immediately re-computes the fingerprint over the received root certificate and compares it to the expected value. Next, step runs certificate path validation on the end-entity certificate presented by the CA, using the newly validated root certificate. If there's a fingerprint mismatch, or if certificate path validation fails, step ca root fails.

The upshot is: if step ca root exits successfully you can be sure you've connected to ca.internal and obtained the correct root certificate (identified by --fingerprint ).

This command can be baked into an AMI, run in a cloud VM startup script, a kubernetes init container, or by a configuration management tool like Ansible, Chef, or Puppet. Alternatively, you can bake your root certificate directly into an AMI. But using step ca root makes federation and root certificate rotation easier.

With the root certificate securely distributed, our next task is to have foo.local generate a key pair and obtain a certificate from step-ca . To do that, we'll need to use a provisioner.

Provisioners

The certificate signing API exposed by step-ca requires mutual authentication. The CA authenticates to the client using a certificate (which the client verifies using the root certificate we obtained in the last step). But the client must also authenticate to the CA. Again, this presents a challenge: the client obviously can't use a certificate for authentication because it doesn't have one yet.

To verify the legitimacy of certificate signing requests, we need some entity that the CA trusts to measure and attest to the identity of our workloads. In the step certificates architecture this entity is called a provisioner, because it helps to provision certificates.

In general, once a provisioner authenticates a workload's identity it issues the workload a bearer token to submit to the CA, along with a certificate signing request (CSR), to obtain a certificate. The bearer token authenticates the workload and authorizes the certificate request.

The details of how a provisioner interacts with a workload (or device or person) and the CA vary by provisioner type. Smallstep supports a number of provisioner types including, AWS Instance Identity Documents, Google Cloud Instance Identity Tokens, the Azure Instance Metadata Service, OAuth/OIDC Single Sign-on, and JWT for building custom integrations.

IID Provisioner

For cloud VMs running on AWS, GCP, or Azure step-ca can be configured to use instance identity documents (IIDs) to authorize certificate signing requests. IIDs make integrating step certificates even easier: instead of building and integrating a JWK provisioner that issues tokens to workloads, step can request tokens directly from your cloud provider's metadata API with no additional infrastructure requirements:

Once you've configured step-ca to use IIDs for authentication, step will detect the provisioner type, obtain an IID token from your cloud provider, and use it to obtain a certificate from step-ca automatically.

First, bootstrap foo.internal if you haven't already:

Then request a certificate, specifying the name of your IID provisioner as the --issuer (in this case "Google Cloud" ):

OAuth/OIDC Single Sign-on Provisioner

Occasionally it's useful to issue certificates to people (e.g., for testing and debugging). To facilitate this, step-ca can be configured to use OAuth/OpenID Connect (OIDC) identity tokens for authentication. In other words, you can use single sign-on with GSuite, Okta, Active Directory, or any other OAuth OIDC provider to get a certificate:

Again, step detects the provisioner type selected and initiates the OAuth flow automatically:

When an OIDC provisioner is selected for use by step ca token or step ca certificate an OAuth flow is automatically initiated in your local web browser.

By default the issued certificate will use the subject ( sub ) claim from the identity token as the certificate subject. The value of the email claim is also included as an email SAN in the certificate.

When you configure an OIDC provisioner at the CA you can also specify a list of admins . These users will be allowed to request certificates with any name (non-admins can only get certificates binding their own ID and email address).

JWK Provisioner

The JWK provisioner is the most generic provisioner type. It's useful for building custom integrations with infrastructure like Puppet, Chef, Ansible, and Kubernetes. JWK provisioners issue workloads single-use short-lived JSON Web Tokens (JWTs) to authorize certificate signing requests.

Because the JWTs issued by JWT provisioners are short-lived and single-use they have reasonably good security characteristics: a token that's intercepted and misused is easily detected by the intended recipient when their certificate request fails.

To create a JWK provisioner you generate a key pair, represented as a JSON Web Key (JWK), and register the public key with the CA. Optionally, the CA can escrow the corresponding private key (symmetrically encrypted with a password).

A JWK provisioner was created automatically for us when we ran step ca init . We named this provisioner admin using the --provisioner flag. The private key was encrypted using the password from --provisioner-password-file and escrowed with the CA.

To add another JWK provisioner see step help ca provisioner add .

The step ca token subcommand integrates with step-ca to make token generation easy. Before we can use this subcommand, our provisioner needs to be bootstrapped into our PKI:

Like step ca root , the step ca bootstrap command securely fetches the root certificate from the CA. It stores the certificate in a well-known location on disk where step can find it (under ~/.step by default). It also configures step to connect to your CA by default, so you don't need to pass the --ca-url and --root flags all the time.

Once bootstrapped, we can use step ca token to generate a bearer token for foo.local :

This token can be securely transmitted to foo.local (e.g., using SSH) and used to obtain a certificate:

At this point, foo.local has everything it needs to speak HTTPS!

The root_ca.crt we obtained earlier via step ca root :

Along with a private key, and a certificate issued by our CA, generated and obtained via step ca certificate using a JWK provisioner token issued using step ca token :

JWTs are also easy to generate programmatically, without using step . The JWT audience ( aud ) should be the /1.0/sign endpoint for your CA. The JWT subject ( sub ) specifies the subject common name for the issued certificate. The sha and sans claims are non-standard: they specify the root certificate fingerprint and any subject alternative name(s) that are authorized for inclusion in the issued certificate.

What happens when step ca certificate runs?

The step ca bootstrap and step ca certificate commands belie a lot of complexity. Let's look at how they work in more depth.

step ca bootstrap --ca-url <ca-url> --fingerprint <fingerprint> downloads the root certificate and configures step to connect to your CA:

step ca certificate <identity> <crt-file> <key-file> generates a keypair, constructs a CSR, and obtains a certificate from the CA:

Renewing a certificate

Certificates expire. By default, step certificates follows PKI best practices by issuing short-lived certificates that expire after 24 hours.

But your workloads will likely live longer than 24 hours. To keep your workloads up you'll need to renew certificates before expiration. Again, step can help:

The --daemon flag makes step ca renew even more useful. Instead of running once and exiting, this flag tells step ca renew to stay running, continuously monitoring the certificate and renewing it before expiration:

The default behavior in this mode is to wait until two-thirds of the certificate lifetime has passed (e.g., 16 hours for a 24 hours certificate) then attempt a renewal. Some jitter is added to this schedule to prevent a thundering herd of renewals sent to the CA by many VMs that were provisioned at the same time.

If renewal fails (e.g., if the CA is down) step will retry periodically ahead of expiration. Thus, the default behavior of step certificates makes an outage of less than one-third of your certificate lifetime (8 hours for a 24 hour lifetime) non-critical. The system will recover on its own.

In many cases, workloads read certificates from disk at startup and will not notice that a renewal has occurred. To assist in these situations, the --exec flag tells step ca renew to execute a command after each renewal. This can be used to restart a process, or a send a signal to a process telling it to reload its certificate.

Alternatively, your workloads can be written to watch for filesystem changes or simply read the certificate file from disk periodically (e.g., every minute).

Revoking a certificate

Currently, step certificates does not support active revocation mechanisms like CRL or OCSP. These features are on the roadmap, but even with infrastructure support they are challenging to implement properly.

Instead, step certificates is designed to make passive revocation easy. Rather than actively marking a certificate as revoked, and relying on each workload to notice and enforce the revocation, you can mark a certificate as revoked at the CA. The CA will not renew a revoked certificate, allowing the certificate will naturally expire:

Federation

Suppose foo.internal is running in AWS and bar.internal is running in GCP. To communicate, traffic must transit the public internet. Traditionally this sort of cross-connectivity is secured using a network-level tunnel (e.g., an SSN tunnel or a VPN). These tunnels are often configured and managed ad-hoc. They're also hard to scale, hard to operate, and can introduce a single-point of failure into an otherwise resilient system.

With good automated certificate management you can leverage mTLS for secure cross-connectivity without a VPN.

In this scenario, you may want to run two CAs: one in AWS and one in GCP. That way you don't need to expose step-ca to the public internet. To make this work these CAs must federate -- foo.internal and bar.internal must trust both CAs to issue certificates.

The step ca federate subcommand can be used to setup this federation, and step ca works to distribute the bundle of federated roots to workloads in both clouds. Learn more about federation and step support in our Step v0.8.3 brings Federation & Root Rotation blog post.

Get Started Today

We encourage you to visit smallstep on Github to build your own internal PKI and empower mTLS between your distributed services. We would love to hear about your experience and welcome feedback and/or questions on our community gitter. Or give us your opinions by taking the PKI Survey. The team continues to innovate on our offerings and new features are coming every couple of weeks so please check back often to follow our progress. Alternatively you can subscribe to our updates by entering your email below.