Production considerations when running a certificate authority server
step-ca is built for robust certificate management in distributed systems. As
with any entity in your infrastructure, running step-ca effectively in
production requires some knowledge of its strengths and limitations. This
document addresses the important production considerations that operators
should know about when running step-ca as a certificate authority server.
In this section we recommend a few best practices when it comes to running, deploying, and managing your own online certificate authority server and PKI.
Security is a moving target.
We expect our recommendations to change and evolve as well.
Safeguard Your Root And Intermediate CA Keys
When you initialize a two-tier CA, two private keys are generated:
one intermediate private key, and one root private key.
It is very important that these private keys be kept secret.
The intermediate key is used by the CA to sign certificates. The root key is not needed for day-to-day CA operation and should be stored offline.
The keys can be generated on an air-gapped device or on a Hardware Security Module (HSM).
Here's an example key protection strategy for a high-security production PKI.
In this example, step-ca acts as a subordinate CA to an offline root CA.
Generate a root CA (private key and certificate) on a Hardware Security Module (HSM) or air-gapped device that is kept in "cold storage", off the internet.
HSMs are ideal for storing private keys and performing signing operations securely.
For durability, keep at least two copies of your root key, in separate locations.
Generate intermediate key(s) on a separate, online cloud HSM or in a key management service (KMS) that will be used by the CA for signing operations in production
Generate Certificate Signing Requests (CSRs) for your intermediate CA(s)
Sign the generated CSR using the root HSM
Configure step-ca to use the signed root and intermediate certificates
Configure step-ca to access the cloud HSM or KMS intermediate key for online signing operations
You'll use this new intermediate key password to start step-ca.
Never use your password in a command line argument.
To keep sensitive values from leaking into .bash_history and system state, put your intermediate key password into a file by itself. Then use the--password-file flag when calling step-ca.
Once you've changed the intermediate private key password,
you should never have to use the root private key password again.
So, then what should you do with it?
Bury it in a cave high in the mountains.
Or, store it in a password manager or secrets manager.
There are many to choose from and the choice will depend on the risk & security profile of your organization.
In addition to using a password manager to store all passwords
(private key, provisioner password, etc.) we recommend using a threshold cryptography algorithm like
Shamir's Secret Sharing
to divide the root private key password across a handful of trusted parties.
Avoid Storing Passwords in Environment Variables
systemd discourages using the environment for secrets
because it doesn't consider it secure and exposes a unit's environment over dbus. From
Note that environment variables are not suitable for passing secrets (such as passwords, key material, …) to service processes.
Environment variables set for a unit are exposed to unprivileged clients via D-Bus IPC, and generally not understood as being data that requires protection.
Moreover, environment variables are propagated down the process tree,
including across security boundaries (such as setuid/setgid executables),
and hence might leak to processes that should not have access to the secret data.
For some isolated environments, we could see an argument for the convenience of an environment variable.
Even then, there can be subtle issues.
For example, anyone with access to the Docker daemon can view all of the environment variables of running Docker containers, using docker inspect.
For posterity, however, if you've secured your environment and rely on it for secrets, there is a way to pass a password into step-ca from an environment variable in Bash:
This method is known as Bash Process Subsitution,
and on most systems the password will not appear in ps output.
However, this approach is not recommended simply because it's so difficult to ensure security with environment variables.
When you initialize your PKI,
a default JWK provisioner will be created.
If you're not going to use this provisioner,
we recommend deleting it.
If you are going to use this provisioner,
secure it with a different password than your CA's signing keys.
You can do this by passing separate --password-file and --provisioner-password-file files when running step ca init.
Short-lived certificates are not a replacement for active revocation using certificate revocation lists (CRL) or Online Certificate Status Protocol (OCSP). Automatic active revocation is not available in step-ca, but you can manually manage a CRL for your CA certificates. See below.
One mayfly species, dolania americana, lives for five minutes or less! So do some certificates. But it can be difficult to operationalize such short-lived certificates.
Enable Active Revocation On Your Intermediate CA
The value of a two-tiered PKI is that you can add your root CA certificate to the certificate trust store on all of your nodes, and store your root private key completely offline.
A leaf certificate signed by the CA always comes in a bundle that contains the intermediate CA certificate alongside the leaf certificate.
With this bundle, any client that trusts your root CA can verify the complete chain of trust.
Now, what if one day your intermediate CA key is compromised?
You could issue a new intermediate using your root CA key, but your old intermediate has a 10 year validity period!
So, you're stuck having to rotate your root CA too, and that's a much bigger project because you have to distribute the new root certificate to everyone and ensure the old one is no longer trusted.
To avoid this scenario, you can use active revocation on your intermediate CA certificate,
making it possible to immediately revoke a compromised intermediate.
While step-ca doesn't directly support active revocation mechanisms like Certificate Revocation Lists (CRLs) or the Online Certificate Status Portocol (OCSP),
you can independently manage your own CRL if you like.
Create an intermediate CA with a CRL distribution endpoint
Let's make it possible to revoke your intermediate CA down the road if necessary.
This setup is more complex than the default step-ca PKI, but it offers an insurance policy for a compromised intermediate CA.
Create an intermediate CA that includes a CRL endpoint. Save the following template to intermediate.tpl:
You'll need this template to manually create your intermediate CA.
The CRL endpoint here should be an HTTP URL; the CRL file itself is signed.
The CRL will be a static file, so you you might choose an object storage or CDN endpoint here.
Use the template to create your intermediate CA. You will need your root CA certificate and key:
Upload the DER-formatted ca.crl file to the distribution point URL you specified in the template.
Finally, configure your step-ca server to use the intermediate CA you created.
Your CRL will expire, so you will need to generate and push a new empty CRL file regularly. We recommend updating the CRL once two-thirds of its lifetime has elapsed. Configuring automated CRL renewal is beyond the scope of this document.
Revoke A Certificate
To revoke a certificate, add it to the index.txt file before regenerating the CRL file. The format for this CRL database file is:
One certificate per line
Each line is tab-delimited
The tab-delimited fields are:
Entry type. May be V (valid), R (revoked) or E (expired).
An expired certificate may have the type V because the type has not been updated.
openssl ca updatedb does such an update.
Expiration datetime. Format is yymmddHHMMSSZ.
Revokation datetime and optional revocation reason. Must be set for any entry of the type R. Format is yymmddHHMMSSZ[,reason].
Certificate serial number in uppercase hexidecimal, eg. 804A72D941DB451A0123BA4706446D1F.
File name This doesn't seem to be used, ever, so use the value unknown.
Because certificate templates add a lot of flexibility to step-ca,
they can be a source of subtle vulnerabilities in your PKI.
If you use custom certificate templates, be sure they are tightly restricted for your use case.
Use extreme caution when referencing user-supplied data marked as Insecure in templates.
Create a Service User to Run step-ca
Make sure that the configuration folders, private keys, and password file used by the CA are only
accessible by this user. If you're running step-ca on port 443, you'll need the step-ca binary
to be able to bind to that port. See Running step-ca as a Daemon for details.
Running step-ca as a Daemon
Note: This section requires a Linux OS running systemd version 245 or greater.
Add a service user for the CA.
The service user will only be used by systemd to manage the CA. Run:
The following are a few useful commands for
checking the status of your CA,
enabling it on system startup,
and starting your CA.
# Rescan the systemd unit files
$sudo systemctl daemon-reload# Check the current status of the step-ca service
$sudo systemctl status step-ca# Enable and start the `step-ca` process
$systemctl enable --now step-ca# Follow the log messages for step-ca
$journalctl --follow --unit=step-ca
step-ca is built to scale horizontally. However, the creators and maintainers do not regularly test in an HA environment using multiple instances. You may run into issues we did not plan for. If this happens, please open an issue.
A few things to consider / implement when running multiple instances of step-ca:
Use a MySQL database. The default Badger database has no concurrency support.
The only integrated DB that can support multiple instances is MySQL.
See the database documentation to learn how to configure step-ca for MySQL.
Respect concurrency limits. The ACME server has known concurrency limitations when using the same account to
manage multiple orders. The recommended temporary workaround is to generate
an ephemeral account keypair for each new ACME order, or to ensure that ACME
orders owned by the same account are managed serially. The issue tracking
this limitation can be found here.
Synchronize ca.json across instances.step-ca reads all of it's
configuration (and all of the provisioner configuration) from the ca.json file
specified on the command line. If the ca.json of one instance is modified
(either manually or using a command like step ca provisioner (add | remove))
the other instances will not pick up on this change until the ca.json is
copied over to the correct location for each instance and the instance is
sent SIGHUP or restarted. It's recommended to use a configuration management
tool (ansible, chef, salt, puppet, etc.) to synchronize ca.json across instances.
Load balancing or proxying step-ca traffic
If you need to place a load balancer or reverse proxy downstream from the CA, we recommend using layer 4 (TCP)
load balancing or proxying (aka "TLS passthrough").
Layer 7 proxying is not recommended, becase the step toolchain is built around TLS:
step expects to be able to perform a TLS handshake with step-ca, using the CA's root certificate
to complete the trust chain.
Certificate renewal uses mutual TLS by default.
step-ca authenticates the client using the expiring certificate, in order to issue a new one.
Mutual TLS requires a direct, end-to-end TLS handshake between step and step-ca.
(See step ca renew --help for more details.)
By design, step-ca does not have an option to run in HTTP only.
Philosophically, we value perimeterless security and we believe people should use authenticated encryption (e.g. mutual TLS) everywhere.
Making mTLS easy, and helping people get away from the "perimeter security" anti-pattern, are motivating goals behind the project.
That said, lots of folks have legacy issues to contend with, some of these decisions are out of their control, and every threat model is different.
See certificates#246 for more details.
Nginx has a stream module that allows it to pass TLS traffic directly to step-ca.
But it comes with a price: Unlike typical reverse proxy configurations, source IPs are not visible to step-ca (there is no X-Forwarded-For header), and traffic is also not logged to the nginx access log.
See this blog post for an example of TLS passthrough.
Caddy doesn't natively support TLS passthrough, but there is an experimental caddy-l4 module that can do it.
Automate X.509 Certificate Lifecycle Management
We recommend automating certificate renewal when possible.
Renewal can be easily automated in many environments.
See our Renewal documentation for details.
X.509 Certificate Revocation
By default, step-ca uses passive revocation.
Certificates can be revoked using the step ca revoke subcommand.
See our Revocation documentation for details.
Sane Cryptographic Defaults
The step ecosystem uses sane defaults so that you don't have to be a security
engineer to use our step-ca safely. Our defaults align with best current
practices in the industry for using cryptographic primitives and higher order
abstractions, like JWTs.
This section describes our defaults and explains the rationale behind them. Our
selections and guidance will change and evolve over time as security and
cryptography are constantly changing in response to real world pressures.
We use JWTs (JSON Web Tokens) to prove authenticity and identity within the step ecosystem. When configured well, JWTs are a great way to sign and encode data. It's easy to use JWTs insecurely, though, so you must be deliberate about how you validate and verify them (see RFC7519).
step-ca produces JWTs that:
are short-lived (5 minute lifespan)
are one-time-use tokens (during the lifetime of the step-ca)
We've chosen the AES encryption algorithm for writing private keys to disk because it was the official choice of the Advanced Encryption Standard contest.
All supported key sizes are considered to be unbreakable for the foreseeable future. We chose 128 bits as our default because the performance is better as compared to the greater key sizes, and because 128 bits are sufficient for most security needs.
Root CA Certificate
The Root CA certificate is generated once, when you run step ca init.
Validity (10 year window)
Not Before: Now
Not After: Now + 10 years
A 10 year window is advisable until software and tools can be written for rotating the root certificate.
The root certificate is a certificate authority and will be used to sign other Certificates.
Path Length: 1
The Path Length constraint expresses the number of possible intermediate CA certificates in a path built from an end-entity certificate up to the CA certificate.
The default step PKI has only one intermediate CA certificate between end-entity certificates and the root CA certificate.
Key Usage describes how the certificate can be used.
Certificate Sign: indicates that our root public key will be used to verify a signature on certificates.
CRL Sign: indicates that our root public key will be used to verify a signature on revocation information, such as CRL.
Intermediate CA Certificate
The Intermediate CA certificate is generated once, when you run step ca init. It is signed by the Root CA certificate.
The Path Length of the intermediate certificate is 0. Otherwise it uses the same defaults as the root certificate.
A Path Length of zero indicates that there can be no additional intermediary certificates in the path between the intermediate CA certificate and end-entity certificates.
Leaf (End Entity) Certificate
These are the certificates issued by the step-ca server.
Validity (24 hour window)
Not Before: Now
Not After: Now + 24 hours
The default is a 24hr window. This value is somewhat arbitrary. However, our goal is to have seamless end-entity certificate rotation. Rotating certificates frequently is a good security measure because it gives attackers very little time to form an attack and limits the usefulness of any single private key in the system.
We will continue to work towards decreasing this window because we believe it significantly reduces the probability and effectiveness of any attack.
Key Usage describes how the certificate can be used.
Key Encipherment: indicates that a certificate will be used with a protocol that encrypts keys.
Digital Signature: indicates that this public key may be used as a digital signature to support security services that enable entity authentication and data origin authentication with integrity.
Extended Key Usage
TLS Web Server Authentication: certificate can be used as the server side certificate in the TLS protocol.
TLS Web Client Authentication: certificate can be used as the client side certificate in the TLS protocol.
These are the defaults used for communication between step and step-ca.
Min TLS Version: TLS 1.2 Max TLS Version: TLS 1.2
The PCI Security Standards Council required all payment processors and merchants to move to TLS 1.2 and above by June 30, 2018. By setting TLS 1.2 as the default for all TLS protocol negotiation, we encourage our users to adopt the same security conventions.
TLS renegotiation significantly complicates the state machine and has been the source of numerous, subtle security issues. Therefore, by default we disable it.
The default 'ciphersuites' are a list of two cipher combinations. For communication between services running step there is no need for cipher suite negotiation. The server can specify a single cipher suite which the client is already known to support.
Reasons for selecting TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305:
ECDHE key exchange algorithm has perfect forward secrecy
ECDSA has smaller keys and better performance than RSA
CHACHA20 with POLY1305 is the cipher mode used by Google.
CHACHA20's performance is better than GCM and CBC.
The http2 spec requires the TLS_ECDHE_(RSA|ECDSA)_WITH_AES_128_GCM_SHA256 ciphersuite be accepted by the server, therefore it makes our list of default ciphersuites.
Above is a list of step-approved cipher suites. Not all communication can be
resolved with step TLS functionality. For those connections, the list of
server supported cipher suites must have more options in case older clients do
not support our favored cipher suite.
Reasons for selecting these cipher suites can be found in the following