Naming things well saved us more time than any framework migration. Tests are documentation that fails loudly when it goes stale. Security is mostly about saying no to defaults that were convenient in 2010.
What didn’t
If the error message does not say what to do next, it is half an error message. We measured first, and the bottleneck was not where anyone expected. Tests are documentation that fails loudly when it goes stale.
Every retry needs a budget, otherwise it is just a slower outage. The rollback plan should exist before the rollout plan is approved.
The short version
Naming things well saved us more time than any framework migration. Security is mostly about saying no to defaults that were convenient in 2010. Most production incidents start as a small assumption nobody wrote down. Small pull requests get real reviews, large ones get approvals.
- Measure p95 and p99, not just the mean
- Write the failure mode down before writing code
- Alert on symptoms, not causes
- Pin versions, update on purpose
Latency percentiles tell a story that averages politely hide. If the error message does not say what to do next, it is half an error message.
What worked
Tests are documentation that fails loudly when it goes stale. Small pull requests get real reviews, large ones get approvals. Latency percentiles tell a story that averages politely hide. A boring deploy pipeline is a feature, not a lack of ambition.
Our on-call rotation improved the day we deleted half of the alerts. If the error message does not say what to do next, it is half an error message.
Caches are easy to add and surprisingly hard to remove. A boring deploy pipeline is a feature, not a lack of ambition.
Leave a Reply