Find deleted open files and set quotas

Deleting a file removes its directory entry, but a process with the file open can keep its blocks allocated.

The kernel only frees the blocks when the last open file descriptor closes. This is the classic mystery: df says the disk is full, du finds far less, and nothing you delete makes df move. Someone removed a huge log file, but the service that writes it is still running and still holding it open.

Find the holder

Use lsof +L1 to find deleted open files. It lists open files with a link count below one, meaning no directory entry points at them anymore:

sudo lsof +L1

COMMAND  PID USER  FD  TYPE DEVICE   SIZE/OFF NLINK NODE    NAME
nginx   1214 root   5w  REG  253,2 8241733632     0 5311 /var/log/nginx/access.log (deleted)

Read the columns: COMMAND and PID identify the process, FD 5w means file descriptor 5 open for writing, SIZE/OFF shows almost 8 GB still allocated, NLINK is 0 because the file is deleted, and NAME confirms which file it was.

Release the space

Restart or signal the owning service through its normal lifecycle:

sudo systemctl restart nginx
df -h /var/log    # space is back

For log files specifically, the service’s reload or log-rotation signal is even gentler than a restart. What you should not do is delete active log files with rm in the first place. Truncate them instead (truncate -s 0 access.log), which frees the blocks without breaking the writer.

Try it on a test system: run tail -f /tmp/big.log in one shell, delete the file from another, and watch it appear in lsof +L1. Stop tail and the space returns. No reboot needed.

Quotas for shared storage

On multi-user or multi-tenant storage, cleanup after the fact is the wrong tool. Filesystem quotas can bound users or groups: on ext4 you enable the usrquota or grpquota mount options, then set per-user block and inode limits with setquota and review usage with repquota.

Quotas need monitoring and clear failure handling. A user hitting their quota sees the same “no space” error as a full disk, so document who owns which limit and what happens when it is reached.

Try this on a test system: open a file, delete it, and observe the open descriptor. Release it normally and verify space returns without rebooting the server.