Building a Real-Time Threat Detection Pipeline with Certificate Transparency Logs

The timeline for modern phishing attacks has compressed drastically. Today, a threat actor can register a typosquatted domain, request a free TLS certificate, and launch a weaponized credential-harvesting campaign in under 15 minutes. For security teams and DevOps engineers, traditional reactive measures—like waiting for threat intelligence feeds to update or relying on user reports—are no longer fast enough.

To detect these threats before the first phishing email is even opened, security teams are turning to Certificate Transparency (CT) logs.

Originally designed by Google to detect rogue Certificate Authorities (CAs) issuing unauthorized certificates, CT has evolved into a critical pillar of External Attack Surface Management (EASM). Because every publicly trusted SSL/TLS certificate must be logged in a public, append-only cryptographic ledger, CT logs provide a real-time, unalterable feed of newly created infrastructure across the internet.

This post explores the technical mechanics of Certificate Transparency, how threat actors exploit automated certificate issuance, and how you can build a real-time monitoring pipeline to detect typosquatting, shadow IT, and subdomain takeovers.

The Technical Mechanics of Certificate Transparency

To effectively build a monitoring pipeline, you must first understand the cryptographic guarantees that make CT logs reliable. Governed by IETF standard RFC 9162, the Certificate Transparency ecosystem relies on three core components:

1. Precertificates: When a user or automated client requests a certificate, the CA does not immediately issue the final certificate. Instead, it creates a "precertificate" containing all the cryptographic data and sends it to a CT log.
2. Signed Certificate Timestamps (SCTs): Upon receiving the precertificate, the CT log returns an SCT. This is a cryptographically signed promise that the log will incorporate the certificate into its ledger within a specific timeframe (usually 24 hours, known as the Maximum Merge Delay). The CA embeds this SCT into the final certificate and delivers it to the client. Modern browsers like Chrome and Safari will reject any certificate that does not contain valid SCTs from multiple independent logs.
3. Merkle Hash Trees: CT logs use Merkle Trees to ensure the ledger is append-only and immutable. If a compromised CA attempts to retroactively delete a logged certificate to hide its tracks, the tree's root hash will change, immediately alerting global auditors to the tampering.

Because browsers mandate SCTs, threat actors cannot bypass the CT logging process if they want their phishing sites to display the trusted padlock icon. Every time they secure a malicious domain, they ring an alarm bell in the CT logs.

The Threat Landscape: Automation and the 90-Day Lifespan

The widespread adoption of automated, free CAs like Let's Encrypt and ZeroSSL has fundamentally changed the internet. While this automation is a massive win for privacy, it has been heavily weaponized. Recent industry data shows that over 85% of phishing sites now use valid HTTPS certificates, primarily issued by automated CAs.

Furthermore, Google's push to reduce the maximum validity of public TLS certificates from 398 days to 90 days means the sheer volume of certificates being issued is about to quadruple.

Manual monitoring via spreadsheets or ad-hoc database queries is mathematically unsustainable. Ingesting and analyzing this data requires connecting to real-time streaming APIs and applying highly optimized matching algorithms at the edge.

Building a Real-Time Monitoring Pipeline

CT logs process thousands of certificates per second. This "firehose" of data requires a structured pipeline to ingest, parse, match, and act upon the information.

1. Ingestion and Parsing

Rather than batch-processing historical logs, the most efficient way to monitor CT logs is via WebSockets. The open-source project CertStream aggregates data from various CT logs and provides a unified, real-time WebSocket feed.

Here is a practical Python implementation that connects to the CertStream feed, parses the Subject Alternative Name (SAN) and Common Name (CN), and checks for suspicious activity targeting a specific brand.

import certstream
import re
import logging
import json
import requests

# Setup logging
logging.basicConfig(
    format='[%(levelname)s:%(name)s] %(asctime)s - %(message)s', 
    level=logging.INFO
)

# Configuration
TARGET_BRAND = "examplebank"
# Regex to catch basic typosquatting (e.g., example-bank, examp1ebank)
SUSPICIOUS_REGEX = re.compile(rf".*{TARGET_BRAND}.*", re.IGNORECASE)
SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

def alert_security_team(domain, issuer):
    """Push high-confidence alerts to a webhook (e.g., Slack, Teams, SIEM)."""
    payload = {
        "text": f"🚨 *Suspicious Certificate Detected*\n*Domain:* {domain}\n*Issuer:* {issuer}"
    }
    try:
        requests.post(SLACK_WEBHOOK_URL, json=payload)
    except Exception as e:
        logging.error(f"Failed to send alert: {e}")

def process_certificate(message, context):
    """Callback function to process incoming certs from the WebSocket."""
    if message['message_type'] == "heartbeat":
        return

    if message['message_type'] == "certificate_update":
        cert_data = message['data']['leaf_cert']
        all_domains = cert_data['all_domains']
        issuer = cert_data['issuer']['O']

        for domain in all_domains:
            # Check for regex match and exclude authorized domains
            if SUSPICIOUS_REGEX.match(domain) and not domain.endswith(".examplebank.com"):
                logging.warning(f"Match found: {domain} (Issued by: {issuer})")
                alert_security_team(domain, issuer)

# Start listening to the real-time feed
logging.info("Starting CT log monitoring pipeline...")
certstream.listen_for_events(process_certificate, url='wss://certstream.calidog.io/')

2. Advanced Matching: Defeating Homograph Attacks

Basic Regex is sufficient for simple typosquatting (like example-login.com), but sophisticated attackers use Internationalized Domain Name (IDN) homograph attacks. By substituting a Cyrillic "а" for a Latin "a", the domain looks identical to the human eye but registers entirely differently in DNS.

To detect these, your matching engine must decode Punycode (the encoding used for IDNs, which starts with xn--) and calculate visual similarity. You can enhance the Python script above by integrating the idna library and calculating the Levenshtein distance between the newly registered domain and your protected brand assets.

import idna
import Levenshtein

def check_homoglyph(domain, target="examplebank"):
    try:
        # Convert punycode back to unicode for visual comparison
        if domain.startswith("xn--"):
            decoded_domain = idna.decode(domain)
        else:
            decoded_domain = domain

        # Calculate Levenshtein distance on the core domain string
        # (Assuming you strip TLDs before comparison in a production environment)
        distance = Levenshtein.distance(decoded_domain, target)

        # If the visual distance is less than 2, it's highly suspicious
        if distance > 0 and distance <= 2:
            return True
    except idna.IDNAError:
        pass
    return False

3. Detecting Subdomain Takeovers

CT logs are not just for finding external attackers; they are invaluable for securing your own infrastructure. Subdomain takeovers occur when a DNS record points to a de-provisioned cloud resource (like an AWS S3 bucket, an Azure Web App, or a GitHub Pages site).

If an attacker identifies a dangling CNAME record for dev-portal.yourcompany.com pointing to a deleted Azure instance, they can claim that instance name in their own Azure account. The moment they provision a TLS certificate for dev-portal.yourcompany.com using Let's Encrypt, the CT log broadcasts it.

By monitoring CT logs for exact matches of your corporate domains, you can detect unauthorized internal issuances. If a certificate is issued for a corporate subdomain, but your internal Configuration Management Database (CMDB) shows no active deployment for that asset, you are likely witnessing a subdomain takeover in progress.

Filtering the Noise and Managing False Positives

The biggest challenge in CT monitoring is alert fatigue. If you monitor *.yourcompany.com, your DevOps teams will likely trigger alerts every time they spin up a new ephemeral environment or auto-scaling cluster.

To distinguish between Agile IT and Shadow IT (or active attacks), you must integrate your monitoring pipeline with your internal tooling:

1. CI/CD Integration: When a Terraform script or GitHub Action provisions a new environment and requests a certificate via the ACME protocol, it should simultaneously register that expected issuance in an internal allowlist.
2. Dynamic Filtering: Your CT monitoring script should query this allowlist before firing an alert. If api-staging-v2.yourcompany.com appears in the CT log and matches a recent Jenkins build, it is silently logged and ignored.
3. Enrichment: If a match is found and not allowlisted, enrich the alert before sending it to a human. Automatically query DNS (A/AAAA/CNAME records), WHO