What I Learned About Error Budgets the Hard Way

Abstract code gradient artwork

Latency percentiles tell a story that averages politely hide. Security is mostly about saying no to defaults that were convenient in 2010. Naming things well saved us more time than any framework migration.

Details

Caches are easy to add and surprisingly hard to remove. The fastest query is the one you never send, but the second fastest has a good index. We wrote the migration twice: once to learn, once to ship. Every retry needs a budget, otherwise it is just a slower outage.

Feature flags are debt with an expiration date you must actually enforce. The best architecture diagram is the one the new hire can redraw from memory.

What didn’t

Small pull requests get real reviews, large ones get approvals. Most production incidents start as a small assumption nobody wrote down. We wrote the migration twice: once to learn, once to ship. Feature flags are debt with an expiration date you must actually enforce.

  • Keep migrations reversible
  • Delete dead code the same week
  • Pin versions, update on purpose
  • Write the failure mode down before writing code

We measured first, and the bottleneck was not where anyone expected. Latency percentiles tell a story that averages politely hide.

Going further

The rollback plan should exist before the rollout plan is approved. We measured first, and the bottleneck was not where anyone expected. Small pull requests get real reviews, large ones get approvals. The fastest query is the one you never send, but the second fastest has a good index.

Naming things well saved us more time than any framework migration. Security is mostly about saying no to defaults that were convenient in 2010.

The rollback plan should exist before the rollout plan is approved. Most production incidents start as a small assumption nobody wrote down.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *