How to Monitor Certificate Health Across Production Environments

In April 2023, Starlink experienced a massive global outage that disconnected users worldwide. The root cause was not a sophisticated cyberattack, a severed undersea cable, or a complex routing failure. It was a single expired TLS certificate in a ground station.

Starlink is not alone. Over the years, highly sophisticated engineering organizations—including Cisco, Epic Games, and Microsoft—have suffered catastrophic downtime due to mismanaged public key infrastructure (PKI). According to the 2024 Keyfactor State of Machine Identity Management Report, over 80% of organizations have experienced at least one certificate-related outage in the past 24 months.

The reality of modern infrastructure is that the volume of machine identities (microservices, IoT devices, cloud workloads) is exploding. Ephemeral infrastructure and service meshes rely heavily on mutual TLS (mTLS), meaning a single Kubernetes cluster might generate thousands of internal certificates daily. Relying on calendar reminders, spreadsheets, or ad-hoc scripts to track expiration dates is a guaranteed path to production failure.

Monitoring certificate health in production requires a fundamental shift from reactive expiration tracking to proactive, automated observability. This involves tracking cryptographic agility, chain of trust validity, revocation status, and protocol health across your entire infrastructure.

The Anatomy of Certificate Health

When engineering teams talk about certificate monitoring, they usually mean checking the NotAfter date. While expiration is the most common cause of outages, true certificate health encompasses much more.

The 90-Day Lifespan Reality

Google’s "Moving Forward, Together" initiative has proposed reducing the maximum validity of public TLS certificates from 398 days to just 90 days. While the exact enforcement date across all major browsers is pending, the industry has already begun the transition.

This shift fundamentally breaks manual certificate management. When certificates expire every three months, full automation via the ACME protocol and continuous monitoring become mandatory. If your monitoring system relies on human intervention to action a 30-day expiration warning, your operations team will soon be overwhelmed by alert fatigue.

Cryptographic Agility and PQC Readiness

In August 2024, NIST finalized the first three Post-Quantum Cryptography (PQC) algorithms (FIPS 203, 204, and 205). Organizations must now monitor their certificate inventory for cryptographic algorithms to prepare for the inevitable migration away from vulnerable RSA and ECC keys. A healthy certificate today might be considered a security liability tomorrow. Monitoring tools must be able to parse and alert on the specific cipher suites, hashing algorithms (flagging deprecated algorithms like SHA-1), and key sizes used across your environment.

Revocation Status and Trust Chains

A certificate is only as healthy as its chain of trust. Monitoring only the leaf (end-entity) certificate ignores the health of the Intermediate and Root Certificate Authorities (CAs). If a Root CA expires or is cross-signed improperly, the entire chain fails, often resulting in obscure ERR_CERT_AUTHORITY_INVALID errors that are difficult to debug.

Furthermore, active monitoring must include revocation status. If a certificate is compromised and revoked by the CA via an Online Certificate Status Protocol (OCSP) response or a Certificate Revocation List (CRL), but your infrastructure continues to serve it, you have a silent security failure. Ensuring OCSP Stapling is enabled and functioning correctly is a critical metric for production health.

Real-World Failures: When Monitoring Falls Short

Understanding why certificate monitoring fails in production helps architect better observability pipelines. Most failures stem from two critical blind spots:

1. Shadow IT and Unmanaged Secrets: Developers spinning up cloud resources often provision certificates via cloud provider APIs (like AWS Certificate Manager or Azure Key Vault) outside of central IT's purview. You cannot monitor what you cannot see. When these unmanaged certificates inevitably expire, the resulting outage is often misdiagnosed as a network issue, increasing Time to Resolution (TTR).
2. The "Spreadsheet" Anti-Pattern: Relying on static documentation or basic calendar alerts leads to human error. If the person who set the calendar reminder leaves the company, the alert fires into the void.

Implementing a Tiered Monitoring Strategy

To prevent alert fatigue and ensure actionable responses, certificate monitoring should employ a tiered alerting strategy based on time-to-expiration and infrastructure criticality.

Do not wait until a certificate is 7 days from expiration to trigger your first alert. A best-practice alerting threshold looks like this:

• 30 Days (Warning): Trigger an automated renewal workflow. For certificates managed by ACME clients, this is when the renewal attempt should silently succeed. If it fails, log a warning to a non-paging Slack channel or ticketing system.
• 15 Days (Critical): Page the infrastructure or platform engineering team during business hours. A failure at this stage indicates that automated renewal mechanisms are broken and require human intervention (e.g., DNS validation is failing, or firewall rules are blocking ACME HTTP-01 challenges).
• 7 Days (Fatal): Page the on-call engineer immediately, regardless of the time of day, and escalate to security management.

Technical Implementation: Building the Observability Pipeline

Effective certificate monitoring requires a combination of Blackbox (external) and Whitebox (internal) approaches.

Blackbox Monitoring with Prometheus

Blackbox monitoring involves simulating a client handshake from the outside to verify the certificate that is actually being served to users. This catches issues where a certificate was renewed on disk but the web server (e.g., NGINX or HAProxy) was never reloaded to serve the new file.

The open-source standard for this is the Prometheus Blackbox Exporter. You can configure it to probe your HTTPS endpoints and extract TLS metrics.

First, configure the blackbox.yml module to enforce SSL validation:

modules:
  http_2xx:
    prober: http
    http:
      preferred_ip_protocol: "ip4"
      fail_if_ssl: false
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false

Once Prometheus is scraping the exporter, you can use PromQL to create highly specific alerts. The most critical metric is probe_ssl_earliest_cert_expiry.

Here is a PromQL query that triggers an alert when a certificate has less than 30 days of validity remaining:

# Calculate days until expiration (30 days * 24 hours * 3600 seconds)
probe_ssl_earliest_cert_expiry{job="blackbox"} - time() < (30 * 24 * 3600)

You can also monitor for weak TLS versions. If your security policy mandates TLS 1.2 or higher, you can alert on any endpoint negotiating older protocols:

probe_ssl_last_chain_info{tls_version=~"TLS 1.0|TLS 1.1"} > 0

Whitebox Monitoring with Kubernetes and cert-manager

Whitebox monitoring queries the certificate store, ingress controllers, or Kubernetes secrets directly. This allows you to monitor certificates before they are even deployed or exposed externally.

In Kubernetes environments, cert-manager is the de facto standard for certificate lifecycle management. It natively exposes Prometheus metrics that provide deep visibility into your internal PKI and ACME queues.

To monitor certificate expiration directly from the Kubernetes control plane, use this PromQL query based on cert-manager metrics:

certmanager_certificate_expiration_timestamp_seconds - time() < (30 * 24 * 3600)

Additionally, you should monitor the certmanager_certificate_ready_status metric. If this metric returns 0 (False), it indicates that cert-manager is failing to issue or renew the certificate, allowing you to catch renewal pipeline failures weeks before the actual expiration date.

Shift-Left: CI/CD Integration

Certificate health shouldn't only be monitored in production; it should be validated before deployment. Integrating TLS checks into your CI/CD pipeline prevents misconfigured ingress controllers or weak ciphers from ever reaching production.

Command-line tools like testssl.sh can be integrated into GitHub Actions or GitLab CI. You can configure your pipeline to fail the build if the target environment negotiates weak ciphers (like RC4 or 3DES) or lacks proper OCSP stapling configurations. For deep, one-off health and security grading of external endpoints, Qualys SSL Labs remains the industry standard for verifying protocol health.

Security, Compliance, and Zero Trust

Certificate monitoring is not just an operational necessity; it is a strict compliance requirement.

The latest Payment Card Industry standard, PCI DSS v4.0, mandates strict cryptographic architecture. It requires organizations to maintain an active inventory of all trusted keys and certificates, and to respond quickly to vulnerabilities in cryptographic protocols. Failing to monitor certificate health can result in failed compliance audits and severe financial penalties.

Furthermore, the foundation of Zero Trust Architecture (ZTA) relies entirely on machine identity. In a Zero Trust model, IP addresses are irrelevant; every machine, microservice, and workload must authenticate itself, typically using mutual TLS (mTLS). If certificate health monitoring fails in a Zero Trust environment, the architecture collapses. Expired or revoked certificates