Kubernetes Ingress TLS — Practical Zero Trust

How to get and renew Kubernetes Ingress TLS certificates

Written September 21, 2021

Zero Trust or BeyondProd approaches require authenticated and encrypted communications everywhere. TLS is the cryptographic protocol that powers encryption for all your technologies. For TLS, you need certificates. This practitioner's tutorial provides instructions for automating Kubernetes Ingress TLS certificate renewal and enabling server-side encryption.

Set up a private certificate authority

Before you can configure Kubernetes Ingress TLS, you will need to be able to request a certificate issued by a trusted certificate authority (CA). If you already have a private CA and root certificate, you can skip to the automated renewal section below. If you need to create a CA, you can:

Request a copy of your CA root certificate, which will be used to make sure each application can trust certificates presented by other applications.

step ca root ca.crt

Your certificate will be saved in ca.crt.

Select a provisioner

Smallstep CAs use provisioners to authenticate certificate requests using passwords, one-time tokens, single sign-on, and a variety of other mechanisms.

  • ACME (RFC8555) is an open standard, used by Let's Encrypt, for authenticating certificate requests. To use ACME on a private network you need to run an ACME server. ACME is harder to setup, but has a large client ecosystem (some software even has built-in support).
  • Other provisioners use the open source step CLI and do not require a local network agent. The instructions below focus on the JWK provisioner, but can be repurposed with small tweaks to operationalize all non-ACME provisioners.

To learn more, see Configuring step-ca Provisioners.

Show me instructions for...

The right provisioner depends on your operational environment.

The JWK provisioner is the most general-purpose provisioner. It supports password and one-time token-based authentication. To add a JWK provisioner called kubernetes-ingress to a hosted Certificate Manager authority (if you haven't already), run:

step ca provisioner add kubernetes-ingress --type JWK --create --x509-default-dur 720h

For instructions on adding provisioners to open source step-ca, or to learn more about other provisioner types, see Configuring step-ca Provisioners.

The ACME protocol requires access to your internal network or DNS in order to satisfy ACME challenges. For hosted Certificate Manager CAs, you'll need to configure an ACME Registration Authority on your network that will act as an ACME agent to Certificate Manager.

Configure Automated Kubernetes Ingress TLS Renewal

There are various approaches for configuring TLS certificate renewal in Kubernetes. For this tutorial, we will assume you are using Helm and will manage certificate renewal using the open source cert-manager project. You may repurpose various pieces for your own configuration by following the cert-manager or Kubernetes Ingress documentation.

First, install cert-manager in your cluster with Helm:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true

In cert-manager, a custom resource type Certificate represents each TLS certififcate and specifies how to store it.

Create a new Certificate resource myserver-certificate.yaml for your Kubernetes Ingress server. Make sure you specify the same DNS name configured above, and keep track of the Kubernetes secret name where cert-manager will store your certificate and private key.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myserver
spec:
  secretName: myserver-tls
  duration: 720h
  dnsNames:
    - myserver.example.net
  issuerRef:
    name: my-ca
    kind: StepIssuer
    group: certmanager.step.sm

Customize the Certificate name, secretName, and issuer name to suit your own naming style in your Kubernetes cluster, then apply it with kubectl.

kubectl apply -f myserver-certificate.yaml

This configuration specifies that cert-manager should issue and renew a TLS certificate with the DNS name myserver.example.net and store the certificate and private key in a Kubernetes secret named myserver-tls. The certificate is valid for 720 hours, and cert-manager will automatically renew it before expiration and update the myserver-tls secret.

You'll also notice the above Certificate resource has an issuerRef specifying a cert-manager StepIssuer resource named my-ca. Before cert-manager knows how to use your CA to issue and renew your certificate, we'll need to create that StepIssuer resource and configure it to point to your private CA.

First, let's grab the name and key kid from the JWK provisioner we created earlier.

step ca provisioner list

Next, get a base64-encoded version of your CA root certificate:

step ca root -f >(step base64) 2> /dev/null

Create my-ca-issuer.yaml and fill in the CA URL and provisioner details with your own. caBundle refers to the base64-encoded version of your root certificate.

apiVersion: certmanager.step.sm/v1beta1
kind: StepIssuer
metadata:
  name: my-ca
  namespace: default
spec:
  url: https://my-ca-url.my-team.ca.smallstep.com
  caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpekNDQVRHZ0F3SUJBZ0lRTytFQWg4eS8wVjlQMFhwSHJWajVOVEFLQmdncWhrak9QUVFEQWpBa01TSXcKSUFZRFZRUURFeGxUZEdWd0lFTmxjblJwWm1sallYUmxjeUJTYjI5MElFTkJNQjRYRFRFNU1EZ3hNekU1TVRVdwpNbG9YRFRJNU1EZ3hNREU1TVRVd01sb3dKREVpTUNBR0ExVUVBeE1aVTNSbGNDQkRaWEowYVdacFkyRjBaWE1nClVtOXZkQ0JEUVRCWk1CTUdCeXFHU000OUFnRUdDQ3FHU000OUF3RUhBMElBQkFNVkw3VzBQbTNvSlVmSTR3WGQKa2xERW5uNVhTbWo4NlgwYW1DQTBnY08xdElUUG1DVzNCcGU0cE9vV1V2WlZlUWRvU2NxN3pua1V0Mi9HMnQxTgo3MWlqUlRCRE1BNEdBMVVkRHdFQi93UUVBd0lCQmpBU0JnTlZIUk1CQWY4RUNEQUdBUUgvQWdFQk1CMEdBMVVkCkRnUVdCQlJ1Y1ByVm5QdlpOMHI0QVU5TGcyL2VCcng3a2pBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCUlJBdGsKNXpMY0doQ2FobVBuVzIwZExpdEMzRVdNaVE0bERwN2FFeitFUEFJaEFJOWZWczVxb0l0bVQ4anA2WktVNVEydQphRFBrOGsyQ25OMjdyRnNZV3VwTAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  provisioner:
    name: kubernetes-ingress
    kid: N6I99Yuk7iGDMk_eW3QaN2admCsrC9UuDN27dlFXUOs
    passwordRef:
      name: my-ca-provisioner-password
      key: password

You'll notice the passwordRef configuration above, which declares a secret where step-issuer can expect to find your JWK provisioner password. You can name this whatever you'd like. Create the secret in your cluster, then apply your StepIssuer resource.

kubectl create secret generic my-ca-provisioner-password --from-literal=password=Y4nys7f11
kubectl apply -f my-ca-issuer.yaml

If all went as expected now that both your Certificate and your StepIssuer are configured, cert-manager should have reached out to your Smallstep CA, issued (or begun issuing) the certificate, and created the myserver-tls secret with your certificate and private key.

You'll also notice the above Certificate resource has an issuerRef specifying a cert-manager Issuer resource named my-ca. Before cert-manager knows how to use your CA to issue and renew your certificate, we'll need to create that Issuer resource and configure it to point to your private ACME CA (or RA if you've configured one). Create my-ca-issuer.yaml and fill in the CA URL with your own. The Issuer will keep an account key in a Kubernetes secret which is configurable here as well.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: my-ca
spec:
  acme:
    server: https://my-ca-url.my-team.ca.smallstep.com/acme/acme/directory
    privateKeySecretRef:
      name: my-ca-account-key
    solvers:
    - http01:
        ingress:
          class: nginx

Then apply:

kubectl apply -f my-ca-issuer.yaml

To verify that the requester of the certificate actually controls the DNS name specified in your ACME request (from dnsName in your Certificate), the ACME CA will send an HTTP challenge request to that DNS name and expects to receive particular response. For this reason, your Kubernetes cluster must have some HTTP ingress configured, and your DNS name must resolve to that ingress. In the above example, we assume your ingress has a class name nginx, and cert-manager will automatically intercept ACME requests to that ingress to solve the HTTP01 ACME challenge sent by your CA. This effectively "authenticates" your ACME client with the CA, and the CA will then sign the certificate.

If all went as expected now that both your Certificate and your Issuer are configured, cert-manager should have issued (or begun issuing) the certificate and created the myserver-tls secret with your certificate and private key. We'll reference this secret later when we configure Kubernetes Ingress to use this certificate and private key for TLS.

Telling your ingress to make use of the certificate and private key in the myserver-tls secret is as easy as adding a tls block to your exsting Ingress resources.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: myserver
  labels: ...
  annotations: ...
spec:
  rules:
    - host: myserver.example.net
      http:
        paths:
          - backend:
              serviceName: my-service
              servicePort: 5000
            path: /
  tls:
    - hosts:
        - myserver.company.net
      secretName: myserver-tls

Voila! Your ingress controller will pick up this tls configuration and automatically serve the renewed certificate each time the myserver-tls secret is updated.

Use curl to check that your TLS configuraiton is working as expected. You'll need to pass your CA root certificate as an argument so curl can verify the ingress certificate.

$ curl --cacert ca.crt https://myserver.example.net
HTTP/2 200
...

Distribute your root certificate to end users and systems

Once Kubernetes Ingress TLS is configured, you'll need to make sure that clients know to trust certificates signed by your CA. For certificates signed by a public CA (like Let's Encrypt), most clients already include the CA root certificate in their trust stores for certificate verification. But, for a private CA, you will need to explicitly add your CA's root certificate to your clients' trust stores.

The step CLI includes a utility command for this purpose on many systems:

step certificate install ca.crt

Rather than manually running the above for each machine that needs to trust your CA, most teams will use some form of automation to distribute the root certificate. Depending on your needs and your IT or DevOps team's approach, this may be a configuration management tool (like Ansible or Puppet), a Mobile Device Management (MDM) solution, or something else. Some examples:

  • Use Ansible to add ca.crt directly to the ca-ceritficates bundle on linux VMs so running applications trust the API servers they call
  • Bake ca.crt directly into base Docker images for gRPC so gRPC clients can always reference the trusted CA
  • Store ca.crt in a Kubernetes Secret and inject it into an environment variable for access from application code
  • Use Jamf to install ca.crt in the trust stores of every employee Macbook so their web browsers trust internal websites
  • Use Puppet to run step certificate install ca.crt on target machines that want curl to implicity trust the CA
  • Store ca.crt in a Kubernetes ConfigMap and mount it to pods for reference on the filesystem

Alternatively, many clients support passing the CA root certificate as a flag or argument at runtime.

Contribute to this document

The Practical Zero Trust project is a collection of living documents detailing TLS configuration across a broad spread of technologies. We'd love to make this document better. Feel free to contribute any improvements directly on GitHub.

Subscribe
Subscribe to updates

Unsubscribe anytime. See our privacy policy.

Learn
Blog
Try For Free
Register for Demo
Products
Certificate Manager
Smallstep SSH
ACME Registration Authority
Integrations
Pricing
Certificate Manager
Smallstep SSH
Documentation
Certificate Manager
Smallstep SSH
step-ca
Tutorials
step Command Reference
Open Source
step CLI
step-ca
Company
About
Support
Status
Careers

© 2023 Smallstep Labs, Inc. All rights reserved.

Security

Privacy

Terms & Conditions

Website Preferences

Do Not Sell My Information