Pass configuration and secrets

A service does not inherit your shell environment. The PATH tweaks and the export lines in your .bashrc do not exist when systemd starts the process. Every value the program needs has to be defined in the unit, on purpose.

This surprises people the first time. The app works when you run it from the terminal and fails as a service. Nine times out of ten, a missing variable is why.

Environment and EnvironmentFile

Environment= is for small, non-secret values written directly in the unit:

[Service]
Environment=NODE_ENV=production
EnvironmentFile=/etc/demo-api/app.env
ExecStart=/usr/bin/node /opt/demo-api/server.js

EnvironmentFile= loads KEY=VALUE lines from a separate file. That keeps configuration out of the unit and out of your Git repository:

# /etc/demo-api/app.env
PORT=3000
DATABASE_URL=postgres://app@10.0.0.5/app

Make that file root-owned and unreadable to everyone else:

sudo chown root:root /etc/demo-api/app.env
sudo chmod 600 /etc/demo-api/app.env

If the file is missing, the service refuses to start. The journal says so plainly:

demo-api.service: Failed to load environment files: No such file or directory
demo-api.service: Failed to run 'start' task: No such file or directory

When the file is optional, prefix the path with a dash: EnvironmentFile=-/etc/demo-api/local.env. systemd then skips it quietly if it is not there.

Why environment variables are weak for secrets

Even with a root-owned file, the values end up in the process environment. And the environment leaks in several ways.

systemctl show demo-api -p Environment prints them. Child processes inherit them. Crash reports and debugging tools dump them. A DATABASE_URL with a password in it can end up in a bug report.

For sensitive data, systemd has a mechanism built for exactly this:

[Service]
LoadCredential=db-password:/etc/demo-api/db-password

The service reads the secret from $CREDENTIALS_DIRECTORY/db-password. That is a private, per-service path that only this process can see. The value never enters the environment and never appears in systemctl show.

In Node.js, reading it is one line:

const password = readFileSync(`${process.env.CREDENTIALS_DIRECTORY}/db-password`, 'utf8').trim()

The application has to support reading a secret from a file. Most database drivers and libraries do, and when they don’t, a wrapper like the line above is all you need.

Do the inventory

List every value your application needs to run. Sort each one into two piles: ordinary configuration or secret.

Ordinary configuration goes in Environment= or an EnvironmentFile=. Secrets go in a credential file with tight ownership and mode, loaded through LoadCredential= or the application’s own secret-file option.

Try this on one of your services. Doing the classification once, on paper, takes five minutes. Finding a leaked DATABASE_URL in a process listing later takes much longer.