Architecting Kubernetes Certificate Automation for North-South and East-West Traffic

As Kubernetes solidifies its position as the de facto operating system for cloud infrastructure, the volume of machine identities it generates has exploded. The shift toward microservices and Zero Trust Architecture means that every pod, node, and ingress point requires a cryptographic identity. Today, the average enterprise manages over 250,000 machine identities, with Kubernetes being the primary driver of this growth.

Simultaneously, the lifespan of these identities is shrinking. Google’s proposal to reduce the maximum validity of public TLS certificates to 90 days has created a trickle-down effect, pushing internal Kubernetes Public Key Infrastructure (PKI) toward similar—or even shorter—lifecycles.

When network downtime costs average $300,000 per hour, and certificate expirations remain a leading cause of preventable Kubernetes outages, manual certificate management is no longer a viable option. A single expired webhook certificate can prevent new pods from spinning up, causing a cascading failure that takes down the entire cluster.

This post details how to architect a robust, fully automated certificate management system in Kubernetes, splitting the infrastructure into two distinct planes: North-South ingress traffic and East-West pod-to-pod communication.

The Two Planes of Kubernetes Certificate Management

A resilient Kubernetes certificate architecture must address two fundamentally different traffic patterns. Treating them as a single problem often leads to architectural bottlenecks and security vulnerabilities.

1. North-South Traffic: Traffic entering the cluster from external clients. This requires public-facing or enterprise-wide internal certificates, managed via an Ingress Controller or the newer Kubernetes Gateway API.
2. East-West Traffic: Internal communication between microservices within the cluster. This requires highly ephemeral, short-lived certificates utilized for mutual TLS (mTLS), managed by a service mesh.

Automating North-South Traffic with cert-manager and the Gateway API

For traffic entering your cluster, cert-manager is the CNCF-graduated standard. It operates as a Kubernetes add-on, automating the issuance and renewal of TLS certificates from various issuing sources.

While many legacy environments still use the Ingress API, modern architectures are migrating to the Kubernetes Gateway API. The Gateway API offers superior Role-Based Access Control (RBAC), allowing infrastructure teams to manage the Gateway (and its certificates) while developers manage the HTTPRoutes.

Decoupling Root CAs from Kubernetes Secrets

A critical anti-pattern in Kubernetes PKI is storing Root Certificate Authority (CA) private keys directly inside Kubernetes Secrets. If a cluster is compromised, the root of trust is compromised.

Instead, use an external, hardened PKI backend like HashiCorp Vault or AWS Certificate Manager (ACM). The external PKI issues an Intermediate CA to the Kubernetes cluster, keeping the Root CA offline and secure.

Here is an example of configuring a ClusterIssuer in cert-manager to authenticate with an external Vault server using a Kubernetes Service Account:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.internal.example.com:8200
    path: pki_int/sign/k8s-ingress
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: cert-manager-role
        secretRef:
          name: cert-manager-vault-token
          key: token

With the ClusterIssuer established, you can automate certificate provisioning for a Gateway API resource. When you define a Certificate resource, cert-manager automatically handles the Certificate Signing Request (CSR), retrieves the signed certificate from Vault, and stores it as a Kubernetes Secret for the Gateway to use.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-gateway-cert
  namespace: gateway-system
spec:
  secretName: api-gateway-tls
  duration: 2160h # 90 days
  renewBefore: 360h # 15 days
  issuerRef:
    name: vault-issuer
    kind: ClusterIssuer
  dnsNames:
    - api.example.com

This configuration achieves zero-touch automation. If a certificate requires a Jira ticket or human intervention to renew, the architecture is fundamentally flawed.

Securing East-West Traffic with Service Meshes and eBPF

Internal workload certificates should live for hours, not years. Ephemeral certificates limit the blast radius of a compromised private key and eliminate the need for complex Certificate Revocation Lists (CRLs) or Online Certificate Status Protocol (OCSP) infrastructure, which often fail at scale.

To handle the massive volume and rapid rotation of East-West certificates, you need a Service Mesh.

The SPIFFE/SPIRE Standard

Modern workload identity is governed by SPIFFE (Secure Production Identity Framework for Everyone). SPIFFE moves trust away from static IP addresses or long-lived API tokens and instead issues cryptographic identities based on the workload's runtime characteristics.

The mesh's control plane acts as a subordinate Intermediate CA. It automatically injects and rotates leaf certificates into the workloads.

The Shift from Sidecars to eBPF

Historically, tools like Istio injected a sidecar proxy (Envoy) into every pod to handle mTLS and certificate management. While effective, this doubled the number of containers in a cluster, increasing memory overhead and latency.

In 2024 and beyond, the industry is rapidly shifting toward eBPF-based service meshes like Cilium. Instead of sidecars, Cilium handles mTLS and certificate management at the Linux kernel level using eBPF maps. This drastically reduces overhead while ensuring that pod-to-pod communication is encrypted transparently.

In an eBPF architecture, the node-level agent securely requests short-lived certificates (often valid for just 1 to 24 hours) from the control plane and handles the cryptographic handshakes in the kernel, entirely abstracting the PKI complexity away from the application developers.

Preventing the "Thundering Herd" Problem

Automating certificates at scale introduces a new failure mode: the Thundering Herd.

This occurs when a widely used Intermediate CA expires or a cluster experiences a mass restart. Thousands of microservices may attempt to renew their certificates at the exact same millisecond. This sudden spike in cryptographic requests can easily crash the cert-manager webhooks, the external Vault API, or the service mesh control plane.

To mitigate this:
1. Implement Jitter: Ensure that certificate renewal clients introduce randomized jitter (e.g., renewing anywhere between 15 and 20 days before expiration) so requests are spread out over time.
2. Rate Limiting: Enforce strict rate limits on your external PKI APIs to ensure they degrade gracefully rather than crashing outright.
3. Scale Control Planes Independently: Ensure your cert-manager and service mesh control plane pods have Horizontal Pod Autoscalers (HPA) configured to handle sudden spikes in CSRs.

Monitoring, Alerting, and Avoiding the Blind Spot

Automation is only as reliable as its monitoring. "Shadow PKI"—where developers spin up self-signed certificates or use unauthorized issuers to bypass IT processes—creates massive blind spots.

You must proactively monitor certificate lifespans. If you are using cert-manager, it exposes Prometheus metrics out of the box. The most critical metric is certmanager_certificate_expiration_timestamp_seconds.

Here is a PromQL alert rule that triggers a critical warning when a certificate is within 7 days of expiration:

groups:
- name: CertificateAlerts
  rules:
  - alert: CertificateExpiringSoon
    expr: certmanager_certificate_expiration_timestamp_seconds - time() < (7 * 24 * 3600)
    for: 1h
    labels:
      severity: critical
    annotations:
      summary: "Certificate {{ $labels.name }} in namespace {{ $labels.namespace }} is expiring in less than 7 days."
      description: "Automated renewal has likely failed. Investigate the cert-manager logs and Issuer status."

The In-Cluster Monitoring Trap

Relying solely on Prometheus for certificate monitoring contains a fatal flaw. If an internal webhook certificate (such as the one controlling the Prometheus operator or the Kubernetes API server itself) expires, the cluster control plane can lock up.

When the cluster fails, Prometheus cannot send the alert that the cluster is failing.

This is why out-of-band monitoring is a strict requirement for enterprise environments. Using a dedicated external platform like Expiring.at ensures that your external endpoints, API gateways, and critical infrastructure certificates are monitored from outside the cluster. If your internal PKI fails and brings down your ingress, Expiring.at will still detect the expiration or unreachable endpoint and alert your incident response teams immediately, bypassing the internal failure domain.

Compliance and Post-Quantum Cryptography (PQC)

Architecting a robust certificate lifecycle is no longer just a best practice; it is becoming a strict regulatory requirement.

The Digital Operational Resilience Act (DORA), taking effect in the EU in January 2025, mandates severe penalties for financial institutions that suffer outages due to unmanaged ICT risks—including expired certificates. Similarly, NIST SP 800-207 (Zero Trust Architecture) mandates continuous verification, which translates directly to automated mTLS and frequent certificate rotation in Kubernetes environments.

Furthermore, infrastructure teams must now plan for "crypto-agility." In August 20