Most production incidents start as a small assumption nobody wrote down. Small pull requests get real reviews, large ones get approvals. The fastest query is the one you never send, but the second fastest has a good index.
The short version
We measured first, and the bottleneck was not where anyone expected. Latency percentiles tell a story that averages politely hide. The rollback plan should exist before the rollout plan is approved.
A boring deploy pipeline is a feature, not a lack of ambition. We wrote the migration twice: once to learn, once to ship.
Lessons
The rollback plan should exist before the rollout plan is approved. Every retry needs a budget, otherwise it is just a slower outage. We measured first, and the bottleneck was not where anyone expected.
- Write the failure mode down before writing code
- Pin versions, update on purpose
- Alert on symptoms, not causes
- Measure p95 and p99, not just the mean
The rollback plan should exist before the rollout plan is approved. Most production incidents start as a small assumption nobody wrote down.
What didn’t
The rollback plan should exist before the rollout plan is approved. The best architecture diagram is the one the new hire can redraw from memory. Security is mostly about saying no to defaults that were convenient in 2010. Small pull requests get real reviews, large ones get approvals.
Naming things well saved us more time than any framework migration. Latency percentiles tell a story that averages politely hide.
The rollback plan should exist before the rollout plan is approved. We measured first, and the bottleneck was not where anyone expected.
Leave a Reply