Every retry needs a budget, otherwise it is just a slower outage. Our on-call rotation improved the day we deleted half of the alerts. Most production incidents start as a small assumption nobody wrote down.
What didn’t
The best architecture diagram is the one the new hire can redraw from memory. Small pull requests get real reviews, large ones get approvals. If the error message does not say what to do next, it is half an error message.
Logs answer questions you knew to ask; traces answer the ones you did not. Security is mostly about saying no to defaults that were convenient in 2010.
The short version
Latency percentiles tell a story that averages politely hide. We measured first, and the bottleneck was not where anyone expected. Naming things well saved us more time than any framework migration.
- Keep migrations reversible
- Delete dead code the same week
- Measure p95 and p99, not just the mean
- Write the failure mode down before writing code
Logs answer questions you knew to ask; traces answer the ones you did not. Latency percentiles tell a story that averages politely hide.
Getting started
The fastest query is the one you never send, but the second fastest has a good index. Most production incidents start as a small assumption nobody wrote down. Tests are documentation that fails loudly when it goes stale. Latency percentiles tell a story that averages politely hide.
A boring deploy pipeline is a feature, not a lack of ambition.
Naming things well saved us more time than any framework migration. The rollback plan should exist before the rollout plan is approved.
Every retry needs a budget, otherwise it is just a slower outage. Security is mostly about saying no to defaults that were convenient in 2010.
Leave a Reply