Checkout latency spike after search rollout
INC-1042
Sev 1
Investigating
Opened 2026-10-08 07:12 UTC · Owner R. Okafor
Event timeline
Oldest first · 6 events- 07:12Declared Sev 1: checkout p95 above 3 sSynthetic probe checkout-eu-1 · paged the duty rota
- 07:20Rollback started: search-svc v4.18 to v4.17R. Okafor · change board approved out of hours
- 07:44p95 back under the 1 s budgetCheckout probe · confirmed from two regions
- 08:02Root cause hypothesis: cold index rebuild during rolloutSearch team · deploy 4.18 correlates with the 07:10 latency start
- 08:40Duplicate-charge reports linked to this incidentSupport queue · tracked as INC-1041
- 09:24Recovery holding for 40 min; next update by 10:00R. Okafor
Notes & activity
Newest first09:24 update p95 holding at 0.8 s; error budget burn stopped.
08:47 warn Watch eu-west: index rebuild on v4.17 still in progress.
08:12 note Support linked 3 duplicate-charge tickets to this incident.
07:21 action Rollback of search-svc v4.18 approved and queued.
07:13 alert p95 latency 3.2 s over 5 min on checkout-api (eu-west).
Details
Status
Investigating
Severity
Sev 1
Component
checkout-apiOwner
R. Okafor
Opened
Updated
Impact
p95 +2.4 s · 1.8% checkout failures
Detected by
synthetic probe checkout-eu-1
Built under the Triage design authority v0.12.1 · sample incident