Checkout latency spike after search rollout

INC-1042 Sev 1 Investigating Opened 2026-10-08 07:12 UTC · Owner R. Okafor

Event timeline

Oldest first · 6 events
  • 07:12
    Declared Sev 1: checkout p95 above 3 s
    Synthetic probe checkout-eu-1 · paged the duty rota
  • 07:20
    Rollback started: search-svc v4.18 to v4.17
    R. Okafor · change board approved out of hours
  • 07:44
    p95 back under the 1 s budget
    Checkout probe · confirmed from two regions
  • 08:02
    Root cause hypothesis: cold index rebuild during rollout
    Search team · deploy 4.18 correlates with the 07:10 latency start
  • 08:40
    Duplicate-charge reports linked to this incident
    Support queue · tracked as INC-1041
  • 09:24
    Recovery holding for 40 min; next update by 10:00
    R. Okafor

Notes & activity

Newest first
09:24 update p95 holding at 0.8 s; error budget burn stopped.
08:47 warn Watch eu-west: index rebuild on v4.17 still in progress.
08:12 note Support linked 3 duplicate-charge tickets to this incident.
07:21 action Rollback of search-svc v4.18 approved and queued.
07:13 alert p95 latency 3.2 s over 5 min on checkout-api (eu-west).

Details

Status
Investigating
Severity
Sev 1
Component
checkout-api
Owner
R. Okafor
Opened
Updated
Impact
p95 +2.4 s · 1.8% checkout failures
Detected by
synthetic probe checkout-eu-1

Built under the Triage design authority v0.12.1 · sample incident