Incidents
Something is wrong. Find out what.
Each of these is a running system with a fault injected into it. Read the telemetry it emitted, work out what you would change, and prove it. Nothing is highlighted for you.
One page is crawling and the database looks fine
GuidedSupport has had 11 tickets in twenty minutes about pages hanging. Most of the shop is reportedly fine.
Storefront · 5 minute window · par 5 min
A service stopped, and everything it calls is fine
RoutineOne journey went from working to failing about 3 minutes ago. The others are unaffected.
Storefront · 5 minute window · par 5 min
The provider is drowning and we are the flood
HardCheckout went from fine to almost entirely failing inside a minute. There was no gradual degradation and no warning.
Storefront · 5 minute window · par 6 min
The datastore is doing many times the work
GuidedAccount got about 4 times slower after this morning's deploy, and the datastore behind it is taking a great deal more traffic than it was.
Storefront · 5 minute window · par 4 min
Nobody touched the database and the database is on fire
RoutineA datastore went from comfortable to very busy in under a minute. There has been no deploy, no traffic spike and no schema change, and it is taking noticeably more queries than it was.
Storefront · 5 minute window · par 5 min
We doubled the instances and nothing happened
HardThroughput on the write path stopped climbing about ten minutes ago and has been flat since, at a number nobody recognises.
Storefront · 5 minute window · par 6 min
The model bill doubled and nothing is failing
GuidedInference spend is up about 3× week on week with no more users than we had. Nothing is erroring anywhere and support has not heard a word.
Copilot · 5 minute window · par 5 min
The agent stopped deciding it was finished
RoutineThe slowest 8% of conversations have gone from a few seconds to most of a minute. The median is barely changed and nothing errors.
Copilot · 5 minute window · par 5 min
Every graph improved and the answers got worse
HardLatency is down about 41%, inference spend is down with it, and the error rate has not moved. On every dashboard we have, this afternoon was an improvement.
Copilot · 5 minute window · par 7 min