Most production incidents start as a small assumption nobody wrote down. Our on-call rotation improved the day we deleted half of the alerts. The rollback plan should exist before the rollout plan is approved.
Getting started
Logs answer questions you knew to ask; traces answer the ones you did not. The fastest query is the one you never send, but the second fastest has a good index. The best architecture diagram is the one the new hire can redraw from memory. Latency percentiles tell a story that averages politely hide.
If the error message does not say what to do next, it is half an error message. Our on-call rotation improved the day we deleted half of the alerts.
Lessons
Caches are easy to add and surprisingly hard to remove. Every retry needs a budget, otherwise it is just a slower outage. Security is mostly about saying no to defaults that were convenient in 2010. We measured first, and the bottleneck was not where anyone expected.
- Keep migrations reversible
- Pin versions, update on purpose
- Delete dead code the same week
- Alert on symptoms, not causes
We measured first, and the bottleneck was not where anyone expected. Feature flags are debt with an expiration date you must actually enforce.
Details
Caches are easy to add and surprisingly hard to remove. Our on-call rotation improved the day we deleted half of the alerts. The rollback plan should exist before the rollout plan is approved.
The rollback plan should exist before the rollout plan is approved. The best architecture diagram is the one the new hire can redraw from memory.
Caches are easy to add and surprisingly hard to remove. Small pull requests get real reviews, large ones get approvals.
Leave a Reply