Services and processes

Diagnose a failed unit

Read systemd result, exit status, effective configuration, and journal before repeatedly restarting a service.

systemctl status gives a useful summary, but the full story usually needs more evidence. Restarting a failed unit without reading the journal is how you turn a five-minute config fix into an hour of guessing.

Read the manager result first

Start with the unit’s current state and exit code:

systemctl status app.service --no-pager
● app.service - Shop API
     Active: failed (Result: exit-code) since Mon 2026-08-03 14:02:11 UTC
    Process: 3721 ExecStart=/usr/bin/node /srv/app/server.js (code=exited, status=1/FAILURE)

Result: exit-code and status=1/FAILURE tell you the process started and quit with a non-zero exit. That is different from a timeout, a missing binary, or a dependency that never became ready. Each result type points at a different next step.

Inspect the journal and effective config

The status line is a headline. The journal is the article:

journalctl -u app.service -b -n 50 --no-pager
Aug 03 14:02:11 web1 node[3721]: Error: ENOENT: no such file or directory, open '/srv/app/config/production.json'
Aug 03 14:02:11 web1 systemd[1]: app.service: Main process exited, code=exited, status=1/FAILURE

There is the cause: a missing config file. No restart will fix that until the file exists or the path changes.

Check what systemd actually runs, including drop-in overrides:

systemctl cat app.service

Look for the user, working directory, environment files, and ExecStart path. A typo in a drop-in file is a common reason the service works on one host and fails on another.

Validate application config with the app’s own check command when one exists:

node /srv/app/server.js --check-config
# Error: DB_POOL_SIZE must be a number, got "ten"

That catches bad values before you spend time on systemd settings.

Capture state before restarting the service:

systemctl --failed
systemctl status app.service --no-pager
journalctl -u app.service -b -n 100 --no-pager

Find when the failure began and what changed just before it. Check the process exit status, user, paths, ports, and dependencies. Restart only after you have enough evidence to test one explanation, then verify the user-facing endpoint.

Break a disposable service through one invalid path on a test machine. Capture the manager result and journal, repair it, and prove the service stays healthy after restart. That dry run is worth doing once so the real incident feels familiar.

Lesson completed