Operations · 5 min read

Signals that survive an incident

Dashboards are useful until they are not. During a real incident, the usual assumptions often stop holding: traffic shifts, a dependency disappears, and the metric that looked stable becomes the least trustworthy thing on the screen.

Prefer signals with a clear denominator

A count without a denominator is easy to misread. Requests, errors, and retries become far more useful when paired with the population they came from. The ratio tells you whether the system changed or whether the observation window did.

Record the decision, not only the result

Logs that capture a chosen path, a timeout, or a rejected request are often more valuable than a final status code. They preserve the reasoning at the moment it mattered, which is exactly what a later investigator lacks.

Make the quiet case visible

Success is easy to overlook. A small, steady signal for the healthy path — one that is cheap to collect and hard to confuse with noise — gives the incident a baseline. Without it, every anomaly looks equally urgent.

Back to the notes