One certificate per application was a platform-wide lock
Twenty deployments in an afternoon made it impossible for anyone else to ship. The limit was never per customer, and we had not noticed.
What broke
Every application on Veltic gets a public hostname, and every hostname got its own TLS certificate. That looks per-tenant. It is not. Let's Encrypt counts fifty issuances per registered domain per week, and for every customer on the platform that registered domain is the same one. Around twenty new applications in a single dogfooding session were enough to exhaust the whole week for everybody.
How it surfaced
Not as a certificate error. A deployment would build, start its container, write its virtual host, and then never come up over HTTPS. The command line reported a network problem. The real message was sitting in the nginx error log, and it was unambiguous once we looked.
too many certificates (50) already issued for "veltic.app" in the last 168hThe fix is one certificate, not fifty
A wildcard certificate covers every application hostname at once. A deployment now costs zero issuances. Only a customer's own custom domain still takes one, which is the case where the count genuinely belongs to that customer.
- Application hostnames are served by a single wildcard lineage
- A custom domain still gets its own certificate over the HTTP challenge
- A wildcard is only obtainable over DNS-01, so the DNS provider plugin and an API token became a hard requirement
- If either is missing the manager falls back to the old behaviour instead of breaking the deployment
What we took away from it
A quota that is shared across tenants but charged per action is invisible until it is exhausted, because nothing in a single deployment looks expensive. The general form of the lesson: when you multiply a per-resource cost by the number of tenants, check what the denominator of the provider's limit actually is. Ours was the registered domain, not the subdomain, not the account, and not the customer.
A detail that cost us an hour
The first attempt to issue the wildcard failed against the very limit we were fixing. A failed issuance does not consume quota, so the second attempt fifty-one minutes later succeeded. In between, no merge to the main branch was allowed, because the end-to-end suite creates an application on every run and would have taken the slot that had just opened.