Monitoring and Alerting
Monitoring collects evidence. Alerting interrupts humans. That distinction matters because human attention is scarce operational capacity. A dashboard can contain hundreds of signals; a page should represent probable user impact or imminent data loss.
Symptoms Over Causes
Alert first on symptoms users experience: elevated error rate, unacceptable latency, failed critical jobs, or exhausted capacity. Cause alerts can help route work, but they are often noisy. A single slow disk may not matter. A checkout latency objective being violated does.
Alert Quality
Every page should have an owner, severity, runbook, dashboard, and clear action. If the correct response is “wait and see,” it should probably not wake someone. If the alert fires repeatedly without action, the monitoring system is training engineers to ignore it.
Exercise
Review five recent alerts. For each, write the user impact, the action taken, and whether the alert arrived before or after users noticed. Delete, demote, or rewrite alerts that have no useful action.
Alert Routing
Alert routing is part of system design. An alert should reach the people who can act, with enough context to decide quickly. Routing everything to one central channel creates diffusion of responsibility. Routing every low-priority warning to a pager burns out the team. Severity should reflect urgency, user impact, and required response time.
Good routing also understands dependencies. If a database outage causes twenty services to fail, paging twenty service teams may create chaos. Dependency-aware alerting or incident coordination can reduce duplicate noise while still notifying affected owners.
Runbooks
A runbook should be written for a tired engineer under pressure. It should explain what the alert means, how to verify impact, what dashboards to open, which recent changes to inspect, which mitigations are safe, and when to escalate. A runbook that says “investigate” is not a runbook.
Runbooks decay. They should be reviewed after incidents and when architecture changes. If the on-call repeatedly ignores or rewrites the runbook during incidents, the runbook is evidence of outdated knowledge.
Alert Testing
Teams should test alerts deliberately. A synthetic failure in staging or a controlled production game day can prove whether the signal fires, routes correctly, includes useful context, and leads to mitigation. Untested alerting is a belief system.
Study Task
Rewrite one noisy alert. Define the user symptom, threshold, time window, owner, severity, dashboard link, runbook action, and suppression rule. Then decide whether it should page, open a ticket, or appear only on a dashboard.
Operational Review
A strong monitoring design should be reviewed against three questions. First, does it detect user harm before support channels become the monitoring system? Second, does it give the responder enough context to act without searching through unrelated dashboards? Third, does it protect attention by routing only urgent, actionable conditions to a pager?
Teams should also review negative space: what failure would not be detected? A service may have excellent CPU and uptime dashboards while missing failed background jobs, delayed messages, certificate expiry, or silent data corruption. Monitoring should cover critical user journeys and critical business processes, not only server processes.
The final test is operational rehearsal. Trigger a safe failure, watch the alert route, open the runbook, follow the mitigation, and record where the process hesitates. Every hesitation is a design requirement: clearer ownership, better dashboards, richer labels, safer automation, or a more precise threshold.