Logs answer questions you knew to ask; traces answer the ones you did not. We measured first, and the bottleneck was not where anyone expected. Small pull requests get real reviews, large ones get approvals.
Lessons
If the error message does not say what to do next, it is half an error message. We measured first, and the bottleneck was not where anyone expected. Caches are easy to add and surprisingly hard to remove.
The fastest query is the one you never send, but the second fastest has a good index. Tests are documentation that fails loudly when it goes stale.
Trade-offs
Feature flags are debt with an expiration date you must actually enforce. Logs answer questions you knew to ask; traces answer the ones you did not. Security is mostly about saying no to defaults that were convenient in 2010. Most production incidents start as a small assumption nobody wrote down.
Logs answer questions you knew to ask; traces answer the ones you did not. The fastest query is the one you never send, but the second fastest has a good index.
Going further
Caches are easy to add and surprisingly hard to remove. Feature flags are debt with an expiration date you must actually enforce. Our on-call rotation improved the day we deleted half of the alerts. If the error message does not say what to do next, it is half an error message.
Latency percentiles tell a story that averages politely hide.
Every retry needs a budget, otherwise it is just a slower outage. We wrote the migration twice: once to learn, once to ship.
The fastest query is the one you never send, but the second fastest has a good index. Every retry needs a budget, otherwise it is just a slower outage.
Leave a Reply