Latency percentiles tell a story that averages politely hide. Our on-call rotation improved the day we deleted half of the alerts. Every retry needs a budget, otherwise it is just a slower outage.
The short version
Most production incidents start as a small assumption nobody wrote down. Logs answer questions you knew to ask; traces answer the ones you did not. Every retry needs a budget, otherwise it is just a slower outage. A boring deploy pipeline is a feature, not a lack of ambition.
Tests are documentation that fails loudly when it goes stale. Naming things well saved us more time than any framework migration.
What worked
Small pull requests get real reviews, large ones get approvals. The rollback plan should exist before the rollout plan is approved. Caches are easy to add and surprisingly hard to remove. Tests are documentation that fails loudly when it goes stale.
- Measure p95 and p99, not just the mean
- Delete dead code the same week
- Alert on symptoms, not causes
- Pin versions, update on purpose
Naming things well saved us more time than any framework migration. Most production incidents start as a small assumption nobody wrote down.
What didn’t
Latency percentiles tell a story that averages politely hide. Tests are documentation that fails loudly when it goes stale. The best architecture diagram is the one the new hire can redraw from memory. The fastest query is the one you never send, but the second fastest has a good index.
Caches are easy to add and surprisingly hard to remove. A boring deploy pipeline is a feature, not a lack of ambition.
Logs answer questions you knew to ask; traces answer the ones you did not. Caches are easy to add and surprisingly hard to remove.
Leave a Reply