Retry with a budget

Networks blip. An endpoint that failed at 03:00:00 is often fine at 03:00:05, and one retry saves a false alarm. But retries help with temporary failures only — unbounded loops hide outages and multiply load against a service that is already struggling. A retry needs a budget: a maximum attempt count, a delay, and a final non-zero status when the budget runs out.

Three attempts, growing delay

Retry a health request three times:

for attempt in 1 2 3; do
  if curl --fail --silent --max-time 5 https://example.org/ >/dev/null; then
    exit 0
  fi
  sleep "$attempt"
done

exit 1

The loop bounds everything. Success leaves immediately with exit 0. Each failure sleeps $attempt seconds — 1, then 2, then 3 — a small growing backoff that gives a recovering service room to breathe instead of hammering it on a fixed beat. When all attempts fail, the script falls through to exit 1.

--max-time 5 caps each individual attempt, so a hanging connection can’t stall the loop. The worst case is now arithmetic: three 5-second attempts plus 1+2+3 seconds of sleep, about 21 seconds, then a clean failure.

Verify the budget

Break the URL and measure the final duration:

time ./health-retry
# real    0m21.4s
printf '%s\n' "$?"
# 1

The caller should receive failure after the documented budget — not after an afternoon of silent looping. If time reports minutes instead, one of the attempts is missing its timeout.

What you must not retry

Reads and health checks are safe to repeat. State-changing operations are a different story. If a payment request times out, the server may have processed it anyway — only the response got lost. Retrying sends the payment twice.

Do not retry state-changing operations unless they are idempotent or protected by an idempotency key. Idempotent means running it twice leaves the same result as once: rm -f /tmp/report.pdf qualifies, a POST /charge does not. When you’re not sure, fail after the first attempt and let a human decide — a missed run is cheaper than a duplicated side effect.