Model startup order and readiness

Ordering a service after the network target does not prove that a remote database is ready. network-online.target means the local network stack came up. Your database on another machine can still be rebooting, slow, or down.

That gap is where sleep hacks come from:

[Service]
ExecStartPre=/bin/sleep 15
ExecStart=/usr/bin/node /opt/demo-api/server.js

This works until the database takes sixteen seconds. Then it fails, someone bumps the sleep to thirty, and every deploy gets slower. The sleep encodes a guess, not a dependency. I have never seen one of these get smaller over time.

Model what systemd can see

systemd can order units on the same machine. Use that for local dependencies:

[Unit]
Wants=network-online.target
After=network-online.target postgresql.service

If PostgreSQL runs on this host, After=postgresql.service is a real, checkable ordering. Use Requires= instead of Wants= only when your service is useless without the dependency and should fail with it.

For anything remote, the dependency is invisible to systemd. There is no unit for a database on another server. So the application has to handle it.

Let the app retry temporary failures with a bounded backoff. Try the connection, wait a second, try again, wait two seconds, and give up with a clear error after a budget. In Node.js that is a small loop:

for (let attempt = 1; attempt <= 5; attempt++) {
  try {
    await db.connect()
    break
  } catch (err) {
    if (attempt === 5) throw err
    await new Promise(r => setTimeout(r, attempt * 1000))
  }
}

If the app gives up, it exits non-zero. The restart policy from the previous lesson brings it back with a delay. Together, the two recover from a slow dependency without a single sleep.

Readiness is a separate claim

“Started” and “ready to serve” are different events. A Node.js process can be running for two seconds before it has finished connecting to the database and opened its port.

Type=notify closes that gap. The application calls sd_notify() with READY=1 only when it can handle traffic. Units ordered After= it wait for that signal. Use it only when the application can report readiness. Check its documentation before switching the type, because a notify service that never sends the message hangs until the start timeout.

You can see how long each service took to reach ready:

systemd-analyze blame | head -5

15.212s demo-api.service
 2.104s postgresql.service
 1.318s nginx.service
   612ms ssh.service
   201ms cron.service

A fifteen second entry for a small API is a sleep hack, and it stands out immediately in that list.

Try this on a design you know. Take one sleep-based startup workaround and replace it with either a real local dependency (After= a unit on the same host) or an application-level retry with backoff. Then write one sentence stating the boundary: which part systemd guarantees, and which part the application handles.