Instrument what the goal depends on
Dashboards full of vanity charts are not observation. We instrument the paths tied to the agreed outcomes: the API that must stay under a latency budget, the queue that must not grow unbounded, the backup that must actually restore. If a metric cannot change a decision, it does not deserve a permanent panel.
Watch signals, not only averages
Averages hide the weekend spike and the single tenant that is failing. We look at error budgets, saturation, saturation of people as well as machines, and the odd signals operators already know: a log line that means trouble, a certificate age, a replication lag that always precedes pain. Good observation mixes product metrics with the operational tells.
Close the loop from alert to action
An alert without an owner and a next step is noise. We wire signals to runbooks, on-call paths, and severity that matches the business impact. Day to day that means fewer pages, clearer pages, and a habit of fixing the alert itself when it cries wolf. Feedback loops only work when someone is expected to act.
Review incidents without theatre
After something breaks, we write what happened, what we missed, and what changes as a result — short, technical, without blame theatre. The point is to improve detection and response, not to produce a polished narrative. If the same class of incident repeats, the observation system or the mitigation plan is incomplete.
Keep runbooks beside the work
Runbooks that live in a forgotten wiki rot. We keep the steps to diagnose and recover close to the systems they describe, and we update them when the architecture moves. A new engineer should be able to follow the path without archaeology. Observation without operable instructions is just sightseeing.
Feed observation back into goals
What you learn in production should change the next outcome. Slow restores push backup goals; noisy alerts push reliability work; rising cost curves push architecture reviews. We treat monitoring as input to planning, not as a separate afterthought. That is how goals, risk, and observation stay one system instead of three slide sections.