Design restart policy

Restart=on-failure recovers a service from crashes. It cannot repair bad configuration. A service that dies because its config file is invalid dies again on every restart, forever. And the restart policy hides that from anyone taking a quick look.

So a restart policy has two jobs. Bring the service back after a real crash. And stop trying when the failure is permanent.

The knobs

[Unit]
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=2

Restart=on-failure restarts on crashes, unclean signals, and non-zero exit codes. It does not restart after a clean stop. Restart=always restarts clean exits too, which is right for daemons that should never exit on their own.

RestartSec= adds a delay between attempts. The default is 100 milliseconds. That turns a broken service into a tight loop that fills the journal. Two seconds is a saner floor.

StartLimitIntervalSec= and StartLimitBurst= live in the [Unit] section, not [Service]. They rate-limit starts. With the values above, five failures within sixty seconds put the unit in a permanent failed state instead of looping. Now a human has to look.

How Restart masks a crash loop

Here is the trap. You check the service, it says active (running), and you move on. But it has been crashing every few seconds and systemd keeps bringing it back.

One clue is right in systemctl status. The uptime resets on each restart. A service that is always “active since 4 seconds ago” is a loop, not a healthy daemon.

The restart counter removes all doubt:

systemctl show demo-api.service -p NRestarts

NRestarts=47

A climbing NRestarts means the policy is papering over a real failure. The journal tells the same story with repeated lines like Scheduled restart job, restart counter is at 47.

My advice: alert on the counter or on the failed state, not just on “is it running”. A service that restarts fifty times a day is not healthy, even if is-active always says yes.

Watch the rate limit trigger

Make a disposable service that always fails. ExecStart=/bin/false exits with status 1 every time:

[Unit]
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
ExecStart=/bin/false
Restart=on-failure
RestartSec=1

Reload, start it, and watch the journal. After five attempts you see the limit hit:

demo-fail.service: Start request repeated too quickly.
demo-fail.service: Failed with result 'start-limit-hit'.

Now systemctl status demo-fail.service shows failed, and systemctl start refuses to run it again. That refusal is intentional. systemd remembers the rate limit until you clear it:

sudo systemctl reset-failed demo-fail.service

Fix the cause first, then reset-failed, then start. If you skip the fix, you get five more failures and the same result.

Try this on a test machine before you need it. Seeing start-limit-hit once in a calm moment makes it much less confusing at 3 AM.