Balance and check upstreams

One backend is a single point of failure. Run two, and Caddy can spread traffic between them and route around the one that dies.

Two upstreams and a health check

List more than one upstream and add health checking:

:8080 {
  reverse_proxy 127.0.0.1:4001 127.0.0.1:4002 {
    lb_policy round_robin
    health_uri /health
    lb_try_duration 3s
  }
}

Three subdirectives, three jobs.

lb_policy round_robin alternates requests between the upstreams. The default picks one at random, which is fine too, but round robin makes the demo easier to follow.

health_uri turns on active health checks. Caddy requests /health on each upstream at a regular interval. An upstream that fails the check is marked unhealthy and removed from rotation until it passes again.

lb_try_duration 3s gives each incoming request a three-second budget to find a working upstream. Within that window Caddy retries on another backend. After it, Caddy gives up and returns an error.

Start two throwaway backends

caddy respond from the first module gives us two fake apps in two lines:

caddy respond --listen 127.0.0.1:4001 "backend one" &
caddy respond --listen 127.0.0.1:4002 "backend two" &

Send a few requests and watch them alternate:

for i in 1 2 3 4; do curl -s http://127.0.0.1:8080/; echo; done

You get backend one, backend two, backend one, backend two.

The failure drill

Now kill backend one and keep sending requests. Three phases happen.

First, requests that land on the dead upstream get retried onto the survivor within the try duration. Clients see slow responses, not errors.

Then the active health check fails. The dead upstream leaves rotation entirely, and responses are fast again.

Restart it, and after its health check passes it returns to rotation. That full cycle, degrade, eject, recover, is what three lines of config bought you.

Be careful with retries

Retrying a failed GET is safe. Retrying a POST that charges a credit card is not. The upstream may have processed the request before it died, and a retry runs it twice.

Only allow retries of state-changing requests when the application handles them idempotently, meaning running the same request twice has the same effect as running it once. A missing response never proves the upstream did nothing.

When the check itself breaks

One failure mode to recognize. If the /health endpoint breaks, say a deploy removed it, Caddy marks every upstream down and serves 502s while the apps are fine.

So when the whole fleet goes unhealthy at the same moment, suspect the check before the backends. Request /health directly on one upstream. If it 404s, you found your outage.