An nginx rate limit zone key cannot be changed, only renamed

The change that looked harmless

Our per-application rate limit was keyed on the client address alone. Behind a CDN that address belongs to an edge node, so a load test against one application consumed the budget of every application behind the same node. The obvious correction is to add the server name to the key. We made it, the configuration test passed, and we shipped it.

What actually happened

A rate limit zone lives in the shared memory of the running nginx master. Its key is fixed for the lifetime of that memory. Changing the key in the file and reloading does not replace the zone; it collides with it.

[emerg] limit_req "velticAppRate" uses the "$binary_remote_addr$server_name" key
        while previously it used the "$binary_remote_addr" key

Why nothing caught it

This is the part worth remembering. Three separate checks all reported success, and each one was correct about the thing it actually tests.

• The configuration test

  It parses the file. It has no access to the state of the running process, so it reports success.

• A fresh test instance

  No earlier version of the zone exists in its memory, so the collision cannot occur there.

• Unit tests

  They compare generated text against expected text. The failure is not in the text.

The symptom was the confusing part

Running sites kept working, because the old configuration was still in memory and untouched. What stopped was activation: a new application would deploy, its container would start, its virtual host would be written, and HTTPS would never come up. The end-to-end suite reported a failed fetch. Nothing pointed at the reload.

The fix is a rename, not a repair

The old zone keeps its old key so the existing virtual hosts still parse and the running master sees no conflict. The new rule gets a new name. This makes the fix self-healing: the next deployment writes the configuration, the reload succeeds, and nobody has to restart nginx by hand. A rollback is safe for exactly the same reason.

• Never re-key a zone in place, give it a new name
• A configuration test is not a reload test
• A test instance without prior state cannot reproduce a state collision
• The mapping from zone name to key is now frozen by a test, and every zone a virtual host consumes must be declared