Diagnose a failed service

A service shows failed. The temptation is to edit the unit, restart, and see what happens. Resist it. Every blind restart destroys evidence and adds noise to the journal.

Start with systemctl status, but do not stop at its short log excerpt. It shows the last ten lines, and the real error is often above them.

Gather everything first

Collect the service state and the current boot’s logs before restarting:

systemctl status hello.service --no-pager
journalctl -u hello.service -b --no-pager -n 100
systemctl show hello.service -p Result -p ExecMainStatus

Read the first relevant error, not only the final “failed” line. Check the executable, user, working directory, environment, and permissions. If the command works in your shell but fails as a service, run it as the service user with the same working directory and configuration. Change one cause, restart once, then verify the application from outside systemd.

What each source tells you

Result is systemd’s one-word verdict: exit-code, signal, timeout, start-limit-hit, or oom-kill. ExecMainCode says how the main process ended. ExecMainStatus is the exit code or signal number.

Then systemctl cat hello.service shows the effective unit, drop-ins included. Debug the unit that actually loaded, not the one you remember writing.

Break one on purpose

Let’s produce a failure we can read. Take the hello.service from earlier and point ExecStart= at a path that does not exist, like /usr/bin/nodejs. Reload and start:

sudo systemctl daemon-reload
sudo systemctl start hello.service
systemctl show hello.service -p Result -p ExecMainStatus

Result=exit-code
ExecMainStatus=203

The journal spells it out:

hello.service: Failed to execute /usr/bin/nodejs: No such file or directory
hello.service: Failed at step EXEC spawning /usr/bin/nodejs: No such file or directory
hello.service: Main process exited, code=exited, status=203/EXEC

203/EXEC is systemd saying “I could not even run your program”. The bug is in the unit, not in the application. The same code appears when the file exists but is not executable, so check ls -l on the path too.

Test as the service user

When the executable is fine but the app still dies, run it exactly the way systemd does. Same user, same directory, same environment:

sudo -u hello env -i PORT=3000 /usr/bin/node /opt/hello/server.js

env -i starts from an empty environment, which is what the service gets. Nine times out of ten this reproduces the failure in your terminal, where the error message is right in front of you.

Repair and confirm

Fix the path, reload, and start again:

sudo systemctl daemon-reload
sudo systemctl start hello.service
systemctl is-active hello.service
curl -s http://localhost:3000/

is-active should print active and curl should return the app’s response. That last step is the one people skip. systemd being happy is not the same as the application working.

If the unit hit its start rate limit while you were experimenting, systemctl start refuses with start-limit-hit. Run sudo systemctl reset-failed hello.service first, then start.

Try this whole loop on a test machine. Break the path, record the exact evidence, repair it, and watch the failed state clear. Doing it once in calm conditions makes the real incident shorter.