7 minute read
The service that hurts is not the service that broke
Why every graph points at the wrong component, and what to look at instead.
Damage runs downhill and blame runs uphill
When something goes wrong, everything that depends on it goes wrong too — and there are usually more of those than there are of the thing itself. So the loudest signal in your console is almost never the cause. It is the largest crowd of victims.
Worse, the victims are often the services you know best and watch most closely, so they are where you look first, and they will show you something genuinely alarming.
Ask of every degraded service: is this thing slow, or is it waiting? A service that is waiting will tell you who it is waiting for.
Section 1 of 4