← BlogBen LaiSubscribe via RSS ↗

Observability is a tax you pay before you owe it

The first incident is when you discover you don't have logs. The second is when you discover the logs aren't searchable. By the third you have observability, and a story about how expensive it would have been not to.

The first incident is the one where you discover you don’t have logs. The second is the one where you discover the logs you have aren’t searchable. By the third, you have observability, and a story about how expensive it would have been to not have it.

Observability is a tax. It costs money in tooling, time to instrument, and discipline to keep current. There is no quarter where adding tracing to a new service is the feature the customer asked for. There is no demo for a histogram of p99 latencies. The work is unglamorous, and like most unglamorous work it pays in incidents you don’t have.

The minimum you owe yourself

Three things, in the order I’d add them to a fresh project:

  1. A request ID that travels. Every request gets one. Every log line carries it. Every error carries it. The first thing a customer sends you when something is wrong is a screenshot — the request ID is what turns the screenshot into a trace.
  2. One graph that says “is the site working right now.” Not ten. Not a dashboard with twenty panels. One number a non-engineer can look at and answer yes or no. The rest is debugging support.
  3. An alert on user-visible failure, not on system noise. “Error rate above 1%” with a sane window. Not “CPU above 70%” — that wakes you up for a thing your customers don’t care about.

What you can wait on

Three things, in roughly the order most teams obsess over and shouldn’t:

  1. Tracing every internal call. You don’t need spans inside spans inside spans. You need the request boundary, the database boundary, and the external-API boundary.
  2. Custom metrics for every business event. Add them when an incident shows you you needed one. Anticipating is more expensive than the incident.
  3. The fancy SaaS tier. Most teams hit revenue-relevant problems on the free tier. The paid features are real, but they’re cheaper to add later than to maintain early.

The trick is to instrument early enough that you have ground to stand on during an incident, and late enough that you didn’t waste six months gold-plating telemetry for a service that got rewritten.