Magnifying glass over two nearly identical brass keys, one subtly counterfeit

Lookalike domain detection: build a pipeline on CT logs

•11 min read

Lookalike domain detection is the engineering half of brand protection: a system that notices yourbrand-pagos.com or yourbrnad.com the day it appears, collects evidence automatically and hands a clean case to whoever files the takedown. If you need the response playbook (what to do once a fake site is live), read our guide on how to take down a fake website impersonating your business. This article is about building the detector: data sources, data model, matching, enrichment, failure handling and the trade-offs at each step.

Why Certificate Transparency is the backbone

Most impersonation sites want the padlock, because browsers shame plain HTTP. That is good news for defenders. Under the Chrome Certificate Transparency Policy, all publicly-trusted TLS certificates must be CT compliant to validate in CT-enforcing versions of Chrome. As MDN explains, when a certificate is submitted to a CT log the log returns a signed certificate timestamp (SCT) as proof that it will be added. Chrome has enforced this since Chrome 68 (July 24, 2018) for certificates issued after April 30, 2018, according to the US government's IDManagement notice.

The practical consequence: an attacker who wants a trusted certificate for a lookalike hostname has to publish that hostname in public, append-only logs, often before the phishing page is even finished. Your system reads those logs.

CT is not complete coverage, and your design should say so explicitly:

  • Sites served over plain HTTP never appear.
  • A wildcard certificate (*.example-host.net) hides the actual subdomain used for the attack.
  • Impersonation on shared platforms (a page on a free site builder, a social profile, a marketplace listing) uses someone else's certificate.
  • Registered-but-unused domains have no certificate yet.

That is why a solid design combines CT with at least one other source.

Architecture of a lookalike domain detection pipeline

Think of it as four stages connected by a queue:

  1. Sources: a real-time CT stream, periodic CT search for backfill, and a permutation generator for domains that exist but have no certificate.
  2. Matcher: cheap, fast filtering of every hostname against your watch terms. This is the hot path.
  3. Enrichment: expensive checks only for candidates (DNS, HTTP fetch, screenshot, content similarity).
  4. Cases: scoring, deduplication, human review, evidence storage and alerting.

For the real-time feed, the certstream library receives Certificate Transparency list updates in real time and connects to the public certstream network. For backfill and ad-hoc queries, crt.sh, hosted by Sectigo, queries the approved CT logs and exposes the certificates via web and PostgreSQL; its JSON output can be scripted, for example https://crt.sh/?q=%25.example.com&output=json. For permutations, dnstwist generates typo, homograph and brand-impersonation variants of your domain and can check which are registered.

The key trade-off is where you get the stream. A public aggregator is free and fast to adopt, but it is a dependency you do not control: if it drops, you miss events. Tailing CT logs yourself removes that dependency but means tracking which logs are current, handling each log's API and storing a lot more data. A reasonable path is to start with an aggregator plus scheduled backfill, and only run your own log tailers if detection gaps become a measured problem.

Data model

Keep raw observations separate from the things a human acts on. One domain can produce many certificates, and one case can span several domains run by the same actor.

CREATE TABLE watch_terms (
  id          SERIAL PRIMARY KEY,
  brand_id    INT NOT NULL,
  term        TEXT NOT NULL,          -- normalized: "acmepay"
  kind        TEXT NOT NULL,          -- 'brand' | 'product' | 'executive'
  min_len_for_fuzzy INT NOT NULL DEFAULT 5
);

CREATE TABLE allowlist (
  domain      TEXT PRIMARY KEY        -- your own domains, partners, agencies
);

CREATE TABLE observations (
  id            BIGSERIAL PRIMARY KEY,
  hostname      TEXT NOT NULL,
  registrable   TEXT NOT NULL,        -- eTLD+1, e.g. acme-pay.co
  source        TEXT NOT NULL,        -- 'ct_stream' | 'ct_backfill' | 'permutation'
  cert_sha256   TEXT,                 -- null for permutation hits
  issuer        TEXT,
  seen_at       TIMESTAMPTZ NOT NULL,
  match_reason  JSONB NOT NULL,       -- which term, which rule, distance
  UNIQUE (hostname, cert_sha256)      -- idempotent inserts
);

CREATE TABLE candidates (
  registrable   TEXT PRIMARY KEY,
  first_seen    TIMESTAMPTZ NOT NULL,
  last_checked  TIMESTAMPTZ,
  score         INT NOT NULL DEFAULT 0,
  status        TEXT NOT NULL DEFAULT 'new'  -- new|watching|case|dismissed
);

CREATE TABLE evidence (
  id            BIGSERIAL PRIMARY KEY,
  registrable   TEXT REFERENCES candidates,
  captured_at   TIMESTAMPTZ NOT NULL,
  kind          TEXT NOT NULL,        -- dns|html|screenshot|whois|cert
  storage_uri   TEXT NOT NULL,        -- object storage, write-once
  sha256        TEXT NOT NULL
);

Two details matter. First, the UNIQUE constraint makes ingestion idempotent, so replays and backfills never create duplicates. Second, candidates is keyed by the registrable domain (eTLD+1, computed with the Public Suffix List), because that is the unit a registrar or host acts on. login.acme-pay.co and secure.acme-pay.co are one candidate.

Matching: fast, explainable, tunable

The matcher sees every hostname in the stream, so it must be cheap. Normalize first, then apply rules in order of cost.

import unicodedata, idna

# Small illustrative map; extend it from a proper confusables source.
HOMOGLYPHS = str.maketrans({"0": "o", "1": "l", "3": "e", "5": "s",
                            "\u0430": "a", "\u0435": "e", "\u043e": "o"})

def normalize_label(label: str) -> str:
    if label.startswith("xn--"):
        try:
            label = idna.decode(label)
        except idna.IDNAError:
            pass
    label = unicodedata.normalize("NFKC", label).lower()
    label = label.translate(HOMOGLYPHS)
    return label.replace("-", "").replace("_", "")

def damerau(a: str, b: str, cap: int = 2) -> int:
    # Optimal string alignment distance, early exit above cap.
    if abs(len(a) - len(b)) > cap:
        return cap + 1
    d = [[0] * (len(b) + 1) for _ in range(len(a) + 1)]
    for i in range(len(a) + 1): d[i][0] = i
    for j in range(len(b) + 1): d[0][j] = j
    for i in range(1, len(a) + 1):
        for j in range(1, len(b) + 1):
            cost = a[i-1] != b[j-1]
            d[i][j] = min(d[i-1][j] + 1, d[i][j-1] + 1, d[i-1][j-1] + cost)
            if i > 1 and j > 1 and a[i-1] == b[j-2] and a[i-2] == b[j-1]:
                d[i][j] = min(d[i][j], d[i-2][j-2] + 1)
    return d[-1][-1]

def match(hostname: str, terms: list[dict]) -> dict | None:
    labels = [normalize_label(l) for l in hostname.lstrip("*.").split(".")]
    joined = "".join(labels)
    for t in terms:
        if t["term"] in joined:
            return {"term": t["term"], "rule": "contains"}
        if len(t["term"]) >= t["min_len_for_fuzzy"]:
            for l in labels:
                if damerau(l, t["term"]) <= (1 if len(t["term"]) < 8 else 2):
                    return {"term": t["term"], "rule": "fuzzy", "label": l}
    return None

Design notes:

  • Store the reason. match_reason lets an analyst see why something fired and lets you tune rules with data instead of guesses.
  • Scale the fuzzy threshold with term length. An edit distance of 2 on a four-letter brand matches half the dictionary. Short brands should rely on contains plus context keywords.
  • Use context keywords (login, pago, pagos, soporte, factura, verify, account) as score boosters, not as standalone triggers.
  • Check the allowlist after matching, so your own new subdomains and your agency's staging hosts never page anyone.
  • If you watch many terms, replace the loop with an Aho-Corasick automaton for contains and a BK-tree or n-gram index for fuzzy lookups.

Enrichment and scoring

Only candidates reach this stage, so you can afford slower checks. Run them as independent jobs, each with its own timeout and retry policy, and write every result as evidence.

  • DNS: does it resolve? Does it have MX records (a sign it may send email as you)? Which hosting provider and nameservers?
  • HTTP: status, final URL after redirects, page title, forms that post credentials or card data.
  • Content similarity: dnstwist's approach is to fetch the page, normalize the HTML and compare its fuzzy hash with your real site. Add a favicon hash and a perceptual hash of a screenshot for pages that were rebuilt rather than copied.
  • Your own logs: cloned sites often hotlink your images, fonts or scripts. A Referer header from an unknown host in your CDN or web server logs is a strong, cheap signal, and it is something you can add to the websites you build and operate today.

Keep scoring as configuration, not code, so analysts can tune it:

signals:
  match.contains: 30
  match.fuzzy: 20
  keyword.payment_or_login: 15
  dns.has_mx: 10
  http.has_credential_form: 25
  content.similar_to_origin: 35
  logs.hotlinks_our_assets: 40
thresholds:
  alert: 70
  watch: 30

These weights are a starting point to tune against your own reviewed cases, not a benchmark.

Failure handling

Every external dependency in this pipeline will fail at some point. Plan for it explicitly.

on stream disconnect:
    record gap_start = last_event_time
    reconnect with exponential backoff + jitter
    on reconnect: enqueue backfill(gap_start, now) via CT search

consumer loop:
    event -> match() -> if hit: INSERT ... ON CONFLICT DO NOTHING
    ack only after commit           # at-least-once, idempotent writes

enrichment job:
    timeout per check, max N retries, then mark check 'unknown' (not 'clean')

scheduler:
    re-check 'watching' candidates on a decaying schedule

Lessons that are easy to learn the hard way:

  • Unknown is not clean. A timeout must never lower a score.
  • Domains get weaponized later. Many lookalikes sit parked before going live. Re-check watched candidates over time instead of judging them once.
  • Cloaking is common. Phishing kits often show a harmless page to scanners and data-center IPs. Fetch with a mobile user agent and, where possible, from an egress in your customers' country; record which vantage point saw what.
  • Rate-limit shared services. crt.sh is a public resource. Cache responses, schedule backfill and do not hammer it from every worker.

Security of the scanner itself

Your enrichment workers deliberately visit hostile websites. Treat them as untrusted:

  • Run the headless browser in an isolated container with no credentials, no access to internal networks and an egress-only firewall.
  • Block redirects to private and link-local IP ranges to avoid server-side request forgery against your own infrastructure.
  • Cap response size and execution time; disable downloads.
  • Store captures in write-once object storage with a SHA-256 per file and a UTC timestamp, so the evidence holds up when you hand it to a registrar, host or lawyer.

Build, buy or combine

Commercial brand-protection services bundle data sources, analysts and takedown relationships. Building gives you control over matching rules, integration with your own logs and customer-support tools, and no per-domain pricing. A common middle ground is to build the detection and evidence layer, which depends on knowledge only you have (your brands, assets and markets), and use external services for takedown execution. If you want that layer integrated with your existing stack, that is the kind of custom software we design and build.

Checklist

  • Watch terms include brands, products and common misspellings, normalized.
  • Allowlist covers your domains, partners and agencies.
  • Real-time CT stream plus scheduled CT backfill plus permutation scan.
  • Idempotent inserts keyed by hostname and certificate fingerprint.
  • Candidates grouped by registrable domain.
  • Match reasons stored and reviewable.
  • Enrichment isolated, time-boxed, and "unknown" never treated as "clean".
  • Referer monitoring on your CDN and web server logs.
  • Evidence hashed, timestamped and immutable.
  • Re-check schedule for parked domains.
  • Scoring weights in config, reviewed against real cases.

Common mistakes

  • Treating CT as complete coverage and ignoring HTTP-only sites and shared platforms.
  • Fuzzy matching short brand names and drowning analysts in noise.
  • Matching on the full hostname instead of per label, which misses acme-pay.secure-login.net patterns or over-matches random strings.
  • Alerting on every certificate instead of every new registrable domain.
  • Taking screenshots by hand, with no hash or timestamp, after the site is already gone.

Frequently asked questions

Can I detect lookalike domains that have no TLS certificate?

Not through CT. Use a permutation tool such as dnstwist to check which variants of your domain are registered, and complement it with Referer monitoring and customer reports.

How fast will Certificate Transparency show a new certificate?

CT requires the certificate to be submitted to logs to be trusted by Chrome, so it is public around issuance time. A real-time stream surfaces it as logs publish entries; a scheduled backfill through a search service such as crt.sh catches anything the stream missed during a disconnect.

Should I register common typo domains defensively?

It helps for the most obvious variants, but the permutation space is too large to buy outright, as the dnstwist documentation notes when it warns that checking every permutation can require millions of DNS lookups. Monitoring scales; defensive registration does not.

Comments

No comments yet. Be the first to share what you think.

Leave a comment

It is never published.