Managing Redirects at Scale: Architecture for Site Migrations

Managing redirects at scale is the part of a site migration that usually gets handed to whoever has time, and then breaks quietly: a config file grows past a platform limit, an editor adds a rule that creates a loop, or a chain of three hops appears after the second redesign. This article treats redirects as a small production system with its own data model, validation, runtime, failure modes and tests. If you need the SEO process side (URL inventory, mapping, launch-day checks), start with our website migration checklist for keeping your SEO; here we focus on the engineering behind it.

Why a list in a config file stops working

Most frameworks let you declare redirects in configuration, and for a few dozen rules that is the right answer. The problems show up as the list grows.

• Platform limits. The Next.js docs warn that redirects may have a limit on platforms, and on Vercel there is a limit of 1,024 redirects; for 1,000+ they recommend a custom solution. That limit applies to routes in general: Vercel's community answer on the "Maximum number of routes exceeded" error points out that other routes count too, and one GitHub discussion reports roughly 700 redirects being counted as more than 1,024 when i18n is configured without locale: false.
• Build-time evaluation. As Agility CMS's Next.js guide explains, redirects() is evaluated once at build time, so a redirect an editor publishes afterwards does nothing until the next deploy.
• No validation. A plain array accepts duplicates, loops and redirects to pages that no longer exist. Sanity's redirects course notes that without validation, authors can publish a redirect that breaks the deploy or creates circular redirects.

The fix is not a bigger file. It is to separate where redirects are stored and validated from where they are resolved.

Data model: redirects are data, not code

Store redirects in a table with enough metadata to answer "why does this exist and is it still used?" A workable starting schema:

CREATE TABLE redirects (
  id             BIGSERIAL PRIMARY KEY,
  source_path    TEXT NOT NULL,          -- normalized, e.g. /blog/old-post
  match_type     TEXT NOT NULL DEFAULT 'exact'
                 CHECK (match_type IN ('exact', 'prefix', 'pattern')),
  destination    TEXT NOT NULL,          -- path or absolute URL on an allowed host
  status_code    SMALLINT NOT NULL DEFAULT 301
                 CHECK (status_code IN (301, 302, 307, 308, 410)),
  preserve_query BOOLEAN NOT NULL DEFAULT TRUE,
  origin         TEXT NOT NULL,          -- 'migration-2025', 'editor', 'import'
  created_by     TEXT NOT NULL,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  hit_count      BIGINT NOT NULL DEFAULT 0,
  last_hit_at    TIMESTAMPTZ,
  UNIQUE (source_path, match_type)
);

Design notes:

• match_type keeps exact rules separate from patterns. Exact rules are a hash lookup. Prefix and regex rules are a different, slower problem, and you want to know how many you have.
• 410 lives in the same table. Some old URLs have no equivalent; recording that decision explicitly is better than letting them fall into a generic 404.
• origin and created_by make audits possible. When a migration batch turns out to be wrong, you can roll back that batch without touching editor rules.
• hit_count and last_hit_at tell you which rules still receive traffic. Update them asynchronously from logs, never in the request path.

Normalize once, at write time

Most redirect bugs are matching bugs: /Blog/Post/, /blog/post, and /blog/post?utm_source=x are different strings. Pick one canonical form and apply the same function when writing rules and when looking them up.

export function normalizePath(input: string): string {
  const url = new URL(input, 'https://placeholder.local');
  let path = decodeURIComponent(url.pathname);
  path = path.replace(/\/{2,}/g, '/');          // collapse double slashes
  if (path.length > 1 && path.endsWith('/')) {
    path = path.slice(0, -1);                    // drop trailing slash
  }
  return path.toLowerCase();                     // only if your old site was case-insensitive
}

Two decisions here deserve a conscious choice rather than a default. Lowercasing is safe only if the old platform treated paths case-insensitively; otherwise you can merge URLs that used to be distinct. Query strings are excluded from the key on purpose: match on the path, then carry the query string over to the destination when preserve_query is true, so tracking parameters survive.

Validate the graph before it ships

A redirect set is a directed graph. Treat every write (single rule or bulk import) as a graph operation and reject anything that would leave it in a bad state.

type Rule = { source: string; destination: string; status: number };

export function flatten(rules: Map<string, Rule>, maxHops = 5) {
  const errors: string[] = [];
  for (const [source, rule] of rules) {
    const seen = new Set([source]);
    let target = rule.destination;
    let hops = 0;
    while (rules.has(target)) {
      if (seen.has(target)) { errors.push(`loop: ${[...seen].join(' -> ')}`); break; }
      if (++hops > maxHops) { errors.push(`chain too long from ${source}`); break; }
      seen.add(target);
      target = rules.get(target)!.destination;
    }
    rule.destination = target;   // every rule now points at its final target
  }
  return errors;
}

On top of loop and chain detection, run these checks in CI or in the admin API:

1. Destination exists. Request each unique destination and require a 200 from the new site. A redirect to a 404 is worse than no redirect, because it hides the problem.
2. Source is not a live page. If a source path also exists as a real route, one of them is wrong.
3. No self-redirects after normalization (/about/ to /about).
4. Allowed hosts only for absolute destinations (more on this under security).

Flattening at write time means the runtime never follows chains: every request gets one hop to the final URL.

Where to resolve redirects: the trade-offs

There are four common places to run the lookup, and large sites usually combine them.

• CDN or reverse proxy (edge rules, Nginx map files). Fastest, and the request never reaches your app. The cost is a separate deploy pipeline and, on managed CDNs, plan-dependent limits you should check in your provider's docs before committing.
• Framework config (redirects() in Next.js, equivalents elsewhere). Zero runtime code and matched before the app renders. Good for a small, stable set of exact rules; limited by platform route caps and build-time evaluation.
• Middleware with a key-value store. Rules are read at request time, so editors' changes apply without a deploy. The Next.js docs suggest reading from a store such as Vercel's Edge Config or Redis. The cost is a lookup on every request unless you short-circuit it.
• Application fallback. Only check the redirect table when the app is about to return a 404. Cheap, because it runs on misses only, but slower for redirected users and easy to bypass by accident with catch-all routes.

A pattern that holds up well: exact migration rules at the edge or in a compiled map, editor-managed rules in middleware with a KV store, and the 404 handler as a last-chance lookup that also logs misses.

Making middleware lookups cheap with a Bloom filter

The cost of middleware lookups is that they run on every request, including the vast majority that are not redirects. The Next.js guide on managing redirects at scale recommends using a Bloom filter to check whether a redirect might exist before reading the larger redirects file or database, and avoiding importing a large redirects file into middleware because it slows every request.

A Bloom filter can return false positives but never false negatives, so a "no" lets you skip the lookup entirely and a "maybe" triggers the real one:

// middleware.ts (proxy.ts in recent Next.js versions)
import { NextResponse, type NextRequest } from 'next/server';
import { ScalableBloomFilter } from 'bloom-filters';
import filterJson from './redirects/bloom-filter.json';
import { normalizePath } from './lib/normalize';

const filter = ScalableBloomFilter.fromJSON(filterJson as any);

export async function middleware(req: NextRequest) {
  const key = normalizePath(req.nextUrl.pathname);
  if (!filter.has(key)) return NextResponse.next();       // definite miss

  const res = await fetch(
    new URL(`/api/redirects?pathname=${encodeURIComponent(key)}`, req.nextUrl.origin),
    { signal: AbortSignal.timeout(300) }
  ).catch(() => null);

  if (!res || !res.ok) return NextResponse.next();        // false positive or failure
  const { destination, status } = await res.json();
  const target = new URL(destination, req.nextUrl.origin);
  target.search = req.nextUrl.search;                     // preserve query string
  return NextResponse.redirect(target, status);
}

The filter file is generated at build time from the redirects table. The route handler must handle false positives, and the same docs say you should validate requests to it to prevent malicious use. Note that in the current Next.js documentation, middleware is called Proxy (proxy.ts).

Bloom filters only help with exact matches. A Next.js GitHub discussion on 1,000+ redirects makes the point that they do not work for wildcards or regex, and suggests a precompiled, prefix-indexed matcher so each request only tests a small subset of pattern rules. In practice: index pattern rules by their first static path segment (/blog, /shop) and compile the regexes once at startup.

Failure handling

Decide in advance what happens when the redirect store is slow or down.

• Fail open. If the lookup times out, pass the request through. Users see a 404 on old URLs for a while, which is recoverable; a 500 on every request is not.
• Short timeouts and a local cache. Cache hits and misses in memory for a short time per instance; misses are the bulk of traffic.
• Snapshot fallback. Keep the last exported JSON of exact rules in the deploy artifact so the edge or app can resolve the critical migration set even if the live store is unavailable.
• Idempotent bulk imports. Imports should upsert on (source_path, match_type) inside a transaction, so rerunning a failed import never creates duplicates.

Security: avoid building an open redirect

A redirect service that accepts arbitrary destinations is an open redirect waiting to be abused for phishing.

• Store destinations as paths where possible. For absolute URLs, check the host against an allowlist at write time and again at read time.
• Never build the destination from request input (query parameters, headers). The only input is the normalized path used as a lookup key.
• Protect the admin API and the bulk importer with authentication and an audit log; created_by in the schema exists for this.
• Rate-limit and validate the internal lookup endpoint, since it is reachable from the public internet.

Observability and testing

Redirects fail silently, so make them visible:

• Log every redirect served (source, destination, status) and aggregate into hit_count and last_hit_at in a background job.
• Log 404s with their path and referrer, and review the top entries weekly after a launch. They are the rules you forgot.
• Add a CI test that replays the full URL inventory against a preview deployment and asserts one hop, the expected status, and a final 200:

while IFS=, read -r src expected; do
  out=$(curl -s -o /dev/null -w '%{http_code} %{redirect_url}' "$PREVIEW$src")
  code=${out%% *}; loc=${out#* }
  [ "$code" = "301" ] && [ "$loc" = "$PREVIEW$expected" ] || echo "FAIL $src -> $out"
  final=$(curl -s -o /dev/null -w '%{http_code}' "$loc")
  [ "$final" = "200" ] || echo "FAIL final $loc -> $final"
done < redirects.csv

Checklist

1. Redirects stored in a database or versioned file with metadata, not hand-edited config.
2. One normalization function shared by writes and lookups.
3. Loops, chains, self-redirects and dead destinations rejected before deploy.
4. Chains flattened so every request is a single hop.
5. Exact rules resolved at the edge or from a compiled map; patterns indexed by prefix.
6. Middleware lookups short-circuited with a Bloom filter or cache.
7. Fail-open behavior and a snapshot fallback defined.
8. Destination host allowlist and authenticated admin API.
9. Hit and 404 logging in place from launch day.
10. Replay test of the full URL inventory in CI.

Common mistakes

• Putting thousands of rules in framework config and discovering the platform limit during the launch build.
• Using temporary status codes (302/307) for permanent moves because they were the default in a helper function.
• Dropping query strings, which breaks campaign tracking on old links.
• Updating hit counters synchronously in the request path.
• Deleting old rules after the migration "settles" without checking whether they still get traffic.

If you are planning a replatform and want the redirect layer designed alongside the new site, that is the kind of work we do in our website design and development projects, and for larger systems with their own admin tools, through custom software development.

Frequently asked questions

How many redirects can I put in next.config.js?

It depends on the hosting platform. On Vercel, the Next.js docs cite a limit of 1,024 redirects, and other route types count toward the same cap. Beyond roughly a thousand, move to middleware with a data store.

Should redirects run at the CDN or in the application?

Exact, stable migration rules are best resolved as early as possible, at the edge or from a compiled map. Rules that editors change often fit better in middleware backed by a key-value store, so changes apply without a deploy.

Does a Bloom filter replace the redirects database?

No. It only answers "definitely not here" or "maybe here". You still need the real lookup for positives, and it does not help with wildcard or regex rules.