Your Bugzilla is down and nginx says it's fine. Here is why, and the config that fixes it.

AI crawler swarms don't kill a site with volume. They kill it by walking the parameterised endpoints. show_bug.cgi?id=N is a cheap indexed lookup. buglist.cgi with an arbitrary query string is an unbounded scan. report.cgi is a group-by over the whole table. An aggregate 5 req/s spread across 40,000 residential IPs is nothing for nginx and fatal for the database, because every request is a distinct, uncacheable, expensive query — and per-IP rate limiting never fires, because no single IP is doing anything wrong.

So don't rate limit the site. Rate limit by cost class, and key on the network prefix, not the address.

The config (working, drop-in, nginx 1.18+)

# 1. classify by how expensive the request is to serve
map $request_uri $cost_class {
    default                            cheap;
    "~*/buglist\.cgi"                  costly;
    "~*/query\.cgi"                    costly;
    "~*/report\.cgi"                   costly;
    "~*/show_activity\.cgi"            costly;
    "~*/attachment\.cgi.*action=diff"  costly;
}

# 2. key on /24 and /64 - the farms rotate inside a prefix
map $remote_addr $prefix {
    default                                              $remote_addr;
    "~^(?<v4>\d+\.\d+\.\d+)\."                           $v4;
    "~^(?<v6>[0-9a-f]*:[0-9a-f]*:[0-9a-f]*:[0-9a-f]*):"   $v6;
}

# 3. an EMPTY key is not counted by limit_req, so this zone only ever sees
#    expensive requests. No `if` needed inside the location block.
map $cost_class $costly_key { costly $prefix; default ""; }

limit_req_zone  $prefix      zone=all:20m    rate=10r/s;
limit_req_zone  $costly_key  zone=costly:20m rate=6r/m;
limit_conn_zone $prefix      zone=conns:10m;

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=anon:64m
                 max_size=2g inactive=10m use_temp_path=off;

server {
    limit_req        zone=all    burst=40 nodelay;
    limit_req        zone=costly burst=5;          # queued, NOT nodelay
    limit_conn       conns 8;
    limit_req_status 429;
    add_header       Retry-After 120 always;

    # 4. cache the cheap class hard for anonymous readers only
    location / {
        proxy_cache            anon;
        proxy_cache_key        "$scheme$host$uri$is_args$args";
        proxy_cache_valid      200 60s;
        proxy_cache_bypass     $http_cookie;   # logged-in users always pass through
        proxy_no_cache         $http_cookie;
        proxy_cache_lock       on;             # one origin fetch per hot URL
        proxy_cache_use_stale  updating error timeout http_500 http_502 http_503;
        add_header             X-Cache $upstream_cache_status;
        proxy_pass             http://backend;
    }
}

Three things that matter more than the numbers

1. Return 429 with Retry-After, not 403. Most of these crawlers are ordinary HTTP clients with retry logic. A 429 makes them back off. A 403 makes them rotate to a fresh IP and try again — strictly worse for you.
2. Cache the cheap class. Swarms re-fetch the same popular pages endlessly. proxy_cache_lock on plus a 60s TTL typically removes 80–90% of database load and costs logged-in users nothing.
3. If you add proof-of-work (Anubis and friends), put it on the costly class only. Challenging show_bug.cgi de-indexes you from every search engine and breaks every link anyone ever pasted into an email. Challenging buglist.cgi costs a human one puzzle per session and costs a crawler its entire economics.

User-agent blocklists and robots.txt Crawl-delay are not a defence against this traffic. The volume comes from clients that lie about both. ASN blocks work for about a week. Cost-class limiting keeps working because it targets the thing that actually hurts.

Want this tuned to your stack?

If it saved you an afternoon

Tip jar, entirely optional, any amount: