Deploy a service safely

A deployment changes one known artifact and keeps a path back to the previous version. Everything else here follows from that sentence.

Install atomically, validate before restarting

Copying files over a running application is risky. For a moment, half the files are new and half are old. If the process restarts during that moment, you get a broken mix.

A versioned directory plus a symlink switch avoids this. You copy the new release next to the old one, then flip one symlink:

sudo cp -r build/ /opt/demo-api/releases/v42
sudo ln -sfn /opt/demo-api/releases/v42 /opt/demo-api/current

The service’s ExecStart= points at /opt/demo-api/current/server.js. The symlink flip is atomic. And flipping it back to v41 is the entire rollback.

Validate configuration before you restart, using whatever check the application offers. nginx -t for nginx. node --check server.js for a Node.js file. A config linter if you have one. A restart is the wrong moment to discover a typo.

Reload the right thing

daemon-reload is only for unit changes. New application code does not need it. A changed .service file or drop-in does.

People copy daemon-reload into every deploy script. That is harmless, but it hides the real rule. Then one day someone skips it after actually editing the unit, and the old definition keeps running while they debug the new one.

So restart or reload, then check three things: systemd’s view, the logs, and an application-level health endpoint:

sudo systemctl restart demo-api.service
systemctl is-active demo-api.service
journalctl -u demo-api.service --since "2 min ago"
curl -fsS http://localhost:3000/health

active
{"status":"ok","version":"v42"}

is-active proves systemd’s view. The health endpoint proves the application’s view, including that the new version actually loaded. Notice the version field in the response. Without it, you cannot tell a successful deploy from a restart of the old code.

You want both checks. A service can be active while serving errors.

Decide the rollback before you need it

Write down the exact trigger that means “roll back now”. For example: the health check fails for 60 seconds, or the error rate doubles. Vague triggers turn into long incidents while people debate whether it is bad enough yet.

The rollback itself is two commands:

sudo ln -sfn /opt/demo-api/releases/v41 /opt/demo-api/current
sudo systemctl restart demo-api.service

Then run the same health check and confirm "version":"v41".

Try this on one of your services. Write a deployment checklist: artifact install, config validation, whether daemon-reload applies, restart, the exact health checks, the rollback trigger, and the command that restores the previous known-good version. Keep it to one screen. A checklist that fits on one screen gets used under pressure. A three-page runbook does not.