Investigation challenge
Built from an incident this team actually had. Inspect the evidence, form a hypothesis, and explain the root cause. Hints are available and cost a little credit.
OrbitPay demo data. Connect your sources to generate this from your own repositories, Slack, and Linear.
It is 22:14 UTC. Support has three merchants reporting that checkout hangs and then fails. Your dashboard shows /checkout p99 climbing from 240ms to 9s over the last four minutes, but the error rate on the API itself is flat and no deploy went out today. Find the cause and propose a fix.
OrbitPay accepts checkout requests on checkout-api and hands authorization to payments-worker over a queue. The worker calls an external card processor. Failed authorizations are scheduled for retry in a Redis sorted set. The card processor reported a brief degradation starting at 22:10 UTC and recovering by 22:11:30.
Related incident PAY-184payments-worker · src/retry/retryQueue.ts
async function scheduleRetry(attempt: PaymentAttempt) {
const delay = backoffFor(attempt.attemptCount);
// NOTE: no dedupe on (intentId, attemptCount) yet — see PAY-247
await redis.zadd(RETRY_SET, Date.now() + delay, serialize(attempt));
}payments-worker · src/retry/backoff.ts
const BASE_MS = 250;
const MAX_MS = 30_000;
// Full jitter, added after PAY-184. Before that this returned BASE_MS * 2 ** n
// with no randomisation, so every attempt in a batch retried in lockstep.
export function backoffFor(attemptCount: number) {
const ceiling = Math.min(MAX_MS, BASE_MS * 2 ** attemptCount);
return Math.random() * ceiling;
}#incidents · PAY-184 checkout timeouts
Priya Raman: p99 on /checkout just went from 240ms to 9s. Dev Shah: Retry set depth is 40k and climbing. Every failed attempt from the 22:10 processor blip is retrying at the same instant. Priya Raman: So the backoff is deterministic? Dev Shah: Yes. Same formula, same attempt count, same millisecond. It is a thundering herd we built ourselves.
#payments · Adding full jitter
Dev Shah: Shipped full jitter on the retry backoff. Ceiling stays exponential, the actual delay is uniform in [0, ceiling). Retry set depth peaked at 900 during the replay test instead of 40k.
PAY-184 · Retry queue overload during processor degradation
Severity 1. A 90-second processor degradation at 22:10 UTC produced 12,400 failed authorizations. All of them retried on an identical deterministic schedule, saturating the worker pool and pushing /checkout p99 to 9s for 34 minutes. Resolved by adding full jitter to the backoff.
State what you think is happening and why. Ask for a hint if you are stuck, and say you give up to see the answer.