Surviving the 90-Day Cert: Architecting Zero-Downtime TLS Rotations

The landscape of Public Key Infrastructure (PKI) and SSL/TLS certificate management is undergoing a massive, forced evolution. For years, DevOps and security teams could rely on multi-year certificate lifespans, treating rotation as an infrequent, albeit stressful, annual chore. Those days are over.

Driven by Google’s "Moving Forward, Together" initiative for the Chrome Root Program, the industry is bracing for a reduction in maximum public TLS certificate validity from 398 days to just 90 days. Simultaneously, the explosion of Zero Trust Architectures and service meshes has introduced internal mTLS certificates with lifespans measured in hours.

The central challenge for modern Platform Engineering teams is no longer whether to automate certificate management, but how to execute high-frequency certificate rotations without dropping a single client connection. When certificates rotate every few weeks—or every few hours—manual intervention guarantees an outage.

In this comprehensive guide, we will explore the architectural strategies, configuration patterns, and modern tooling required to achieve true zero-downtime certificate rotation.

The Cost of Failure in the Ephemeral Era

Manual rotation processes inevitably lead to human error, and the financial impact of those errors is staggering. According to industry reports, 77% of organizations have experienced at least one severe outage due to an expired certificate in the past two years, with the average cost of downtime exceeding $300,000 per hour.

We do not have to look far for high-profile examples. In 2023, a global Starlink outage affecting users worldwide was traced back to an expired ground station certificate. Similarly, Cisco issued multiple advisories throughout 2023 and 2024 regarding internal certificate expirations in their SD-WAN and Meraki products, requiring emergency patching. Even gaming giant Epic Games famously suffered a cascading failure across their network due to a single expired service-to-service certificate.

When the 90-day mandate takes effect, the frequency of rotations will quadruple. A simple cron job executing a bash script is no longer a resilient strategy. You need state-aware automation that handles both the issuance of the new cryptographic material and the graceful transition of active network traffic.

The Anatomy of a Zero-Downtime Rotation

To achieve zero downtime during a rotation, your infrastructure must seamlessly transition to the new certificate and private key without terminating active TCP/TLS sessions. Dropping active connections during a rotation results in 502 Bad Gateway errors, interrupted API calls, and degraded user experiences.

There are three primary architectural approaches to solving this problem.

Strategy 1: Graceful Process Reloading (The Standard Approach)

Modern web servers and reverse proxies like Nginx and HAProxy support "hot reloads." This is the most common method for achieving zero downtime on traditional virtual machines or bare-metal servers.

When a hot reload is triggered, the master process reads the new certificate from disk, validates the configuration, and spawns new worker processes equipped with the new cryptographic material. Crucially, the master process sends a "graceful shutdown" signal to the old worker processes. These old workers stop accepting new connections but continue serving their active connections until they naturally terminate.

Implementation Example: Nginx Safe Reload

A naive rotation script might just restart the Nginx service, immediately killing active connections. A resilient script validates the configuration first, then triggers a graceful reload.

#!/bin/bash
# post-rotation-hook.sh

# 1. Test the Nginx configuration with the new certificate
if nginx -t; then
    echo "Configuration test passed. Triggering graceful reload."
    # 2. Send the SIGHUP signal to the master process
    nginx -s reload
    echo "Zero-downtime rotation complete."
else
    echo "CRITICAL: Nginx configuration test failed. Aborting reload."
    # Alerting logic here
    exit 1
fi

If you are using an ACME client like Certbot, you can natively hook this into your renewal process:

certbot renew --post-hook "/usr/local/bin/post-rotation-hook.sh"

Strategy 2: Dynamic Secret Discovery (The Cloud-Native Approach)

While graceful reloads are effective, spawning new worker processes still consumes system resources and can cause latency spikes under extremely high load. Cloud-native proxies like Envoy Proxy take zero-downtime rotation a step further using the Secret Discovery Service (SDS) API.

Envoy can dynamically fetch new certificates from a control plane (such as Istio, HashiCorp Vault, or a custom SDS server) and apply them to active listeners without restarting the process or spawning new workers. The proxy updates its internal memory state and immediately begins using the new certificate for subsequent TLS handshakes.

Implementation Example: Envoy SDS Configuration

Instead of hardcoding the path to the certificate on disk, you configure the Envoy listener to fetch the TLS context from an SDS cluster:

tls_context:
  common_tls_context:
    tls_certificate_sds_secret_configs:
      - name: "my_service_cert"
        sds_config:
          api_config_source:
            api_type: GRPC
            grpc_services:
              envoy_grpc:
                cluster_name: sds_cluster_vault

When the certificate is nearing expiration, your control plane generates a new one and pushes it to Envoy via gRPC. Envoy seamlessly swaps the certificate in memory. This is the exact mechanism that service meshes like Istio and Linkerd use to rotate internal mTLS certificates every few hours with zero dropped packets.

Strategy 3: Blue/Green Infrastructure Rotation (The Immutable Approach)

In highly mature DevOps environments, certificates are treated as immutable infrastructure components. Instead of rotating the certificate on a running server, you spin up a completely new environment (Green) provisioned with the newly issued certificate.

1. Provision the Green environment with the new certificate.
2. Update your load balancer (e.g., AWS Application Load Balancer or Cloudflare) to route a small percentage of traffic to Green.
3. Monitor for TLS handshake errors.
4. If successful, shift 100% of traffic to Green.
5. Allow the Blue environment's active connections to drain naturally, then destroy the Blue instances.

This approach guarantees zero downtime and provides an instant rollback mechanism if the new certificate is misconfigured.

Automating Rotation in Kubernetes

For containerized workloads, Kubernetes has standardized around cert-manager as the undisputed industry solution for certificate automation.

cert-manager acts as a native Kubernetes controller that integrates with ACME providers (like Let's Encrypt), internal CAs (like HashiCorp Vault), and enterprise PKI platforms (like Venafi). It handles the entire lifecycle: issuing the Certificate Signing Request (CSR), solving the ACME challenge, and storing the resulting certificate as a Kubernetes Secret.

Implementation Example: Kubernetes Certificate Manifest

Here is how you define an automated, auto-renewing certificate in Kubernetes:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-gateway-cert
  namespace: ingress-nginx
spec:
  # The secret name where the cert will be stored
  secretName: api-gateway-tls
  # Trigger rotation 30 days before the 90-day expiration
  renewBefore: 720h 
  duration: 2160h # 90 days
  dnsNames:
    - api.yourdomain.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Achieving Zero Downtime in Kubernetes:
When cert-manager updates the api-gateway-tls Secret, how does the application know to use it without downtime?

If you are using the popular ingress-nginx controller, the controller continuously watches the Kubernetes API for changes to Secrets referenced by Ingress resources. When it detects a change, the controller automatically triggers a graceful Nginx reload internally.

For standard application pods that mount the certificate as a volume, you can use tools like Reloader to watch for Secret updates and gracefully perform a rolling restart of your deployments, ensuring at least one pod is always available to handle traffic.

Industry Best Practices for Bulletproof Rotation

Achieving zero downtime requires more than just the right code; it requires robust operational processes.

1. Adopt the 2/3 Lifespan Rule

Never wait until the last minute to rotate a certificate. If a certificate is valid for 90 days, automated rotation should trigger at day 60. This leaves a 30-day buffer to detect and remediate automation failures, DNS issues, or CA outages before an actual expiration occurs.

2. Decouple Issuance from Deployment

Fetching a certificate and applying it are two distinct operational steps. Ensure your CI/CD pipeline handles both safely. Use a centralized secrets engine like HashiCorp Vault to act as the single source of truth. Vault can generate the certificate, and your configuration management tools (Ansible, Chef, or Kubernetes Operators) can pull the certificate and handle the delicate deployment and reload phases.

3. Secure Your Private Keys During Rotation

During the rotation process, private keys must never be written to disk in plaintext or pushed to version control. If a script must write a key to a filesystem, ensure it uses tmpfs (RAM disks) so the key is never persisted to physical storage. For high-security environments, leverage Hardware Security Modules (HSMs) or Trusted Platform Modules (TPMs) to generate the private key directly on the hardware, making it physically impossible to exfiltrate during rotation.

4. Implement Independent Synthetic Monitoring

This is where most organizations fail. Relying solely on your PKI dashboard or cert-manager logs is dangerous. A deployment pipeline might report "success" because it successfully fetched the certificate and restarted a service, but a misconfigured load balancer might still be serving the old, expiring certificate to the public edge.

You must implement independent, external monitoring that actively performs TLS handshakes against your public endpoints to verify the actual certificate being served to clients.

This is where Expiring.at becomes a critical component of your zero-downtime architecture. By continuously monitoring your domains from the outside in, Expiring.at acts as your final line of defense. It tracks your expiration dates, verifies that your automated rotations actually propagated to the edge, and alerts your team via Slack