Sikau

System design · Day 6 · about 40 min

Retries and idempotency

Exponential backoff with jitter, and charging a card exactly once.

Did the payment go through?

Networks fail. A client sends "charge Rs 500", the connection drops, and it can't tell whether the server got it. Retrying is right — but only if retrying is safe. An idempotency key makes the second attempt return the first result instead of charging again. And retries should back off exponentially, with random jitter, so a crowd of clients doesn't hammer a struggling server in lockstep.

Backoff, retry, dedupe

function backoff(attempt, { base = 100, cap = 10000, jitter = Math.random } = {}) {
  return Math.round(Math.min(cap, base * 2 ** attempt) * jitter()); // 100, 200, 400… capped, randomised
}

async function retry(fn, { attempts = 3, sleep = (ms) => new Promise((r) => setTimeout(r, ms)), ...opts } = {}) {
  let lastError;
  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      return await fn(attempt);
    } catch (e) {
      lastError = e;
      if (e.retryable === false) break; // e.g. 400 Bad Request: retrying won't help
      if (attempt < attempts - 1) await sleep(backoff(attempt, opts));
    }
  }
  throw lastError;
}

function createPayments() {
  const receipts = new Map();
  let total = 0;
  return {
    charge(idempotencyKey, amount) {
      const seen = receipts.get(idempotencyKey);
      if (seen) {
        if (seen.amount !== amount) throw new Error("Idempotency key reused");
        return seen; // same request again: same answer, no second charge
      }
      total += amount;
      const receipt = { id: `rcpt-${receipts.size + 1}`, amount };
      receipts.set(idempotencyKey, receipt);
      return receipt;
    },
    total: () => total,
  };
}

The rest of this lesson

  • Which errors to retryexplain
  • Idempotency keysexplain
  • Doublingfill
  • Retry or not?match
  • Safe retrieswrite code
  • Badge quizquiz
  • What you learnedrecap

Continue this lesson free — run real code in your browser

Continue with Google

See all 8 lessons of System design