Prevent overlapping runs

A backup that takes 40 minutes, scheduled every 30, will eventually run alongside itself. Backups, deployments, and cleanup jobs can corrupt state or overload systems when two copies overlap — two tar processes writing the same archive, two deployments moving the same symlink. The fix is a lock: the first run takes it, and later runs see it’s held and stop.

Use flock on Linux

exec 9>/tmp/practical-job.lock
if ! flock -n 9; then
  printf '%s\n' 'job already running' >&2
  exit 75
fi

exec 9> opens file descriptor 9 writing to the lock file and keeps it open for the rest of the script. flock -n 9 asks the kernel for an exclusive lock on that descriptor; -n means don’t wait — fail immediately if another process holds it.

The kernel releases the lock when the process exits, however it exits. Crash, Ctrl-C, kill — the lock disappears with the process. That is what makes flock better than the homemade “write a pidfile and check it” approach: a stale pidfile from a crashed run blocks every future job until a human deletes it.

Exit status 75 is the conventional EX_TEMPFAIL — temporary failure, try again later. It tells a scheduler this wasn’t an error; another copy was doing the work.

Verify the lock holds

Hold the first process open and start a second:

./nightly-backup &      # first run, holds the lock
./nightly-backup
# job already running
printf '%s\n' "$?"
# 75

The second should fail quickly with a distinct status. Then wait for the first run to finish and start again — it must acquire the lock and proceed normally. Both checks matter: a lock that never blocks is useless, and a lock that never releases is worse.

Know the lock’s limits

A lock needs stable scope and cleanup behavior. The scope of this one is a single machine: the path identifies the job, so two different jobs need two different lock files, and on a shared machine consider a per-user directory instead of /tmp so another user can’t squat on the name.

For distributed jobs — the same task scheduled on several servers — a local file lock is not enough. Those need coordination all the machines share, such as a database lock.