Filter the journal

The journal is where systemd stores every log line, from the kernel, from services, and from systemd itself. Each entry carries structured fields alongside the message: the unit, the boot, the priority, the PID.

Filters turn that structure into focused evidence. That beats scrolling through a mixed stream of everything the server logged since Tuesday.

Narrow by unit, boot, and time

Start with the two filters I use in almost every query:

journalctl -u demo-api.service -b

-u selects one unit’s messages, both what the application printed and what systemd said about it. -b restricts to the current boot, which removes last week’s noise from today’s investigation.

Time windows stack on top:

journalctl -u demo-api.service --since "14:00" --until "14:30"
journalctl -u demo-api.service --since "10 min ago"

--since and --until accept timestamps and human phrases like yesterday or 2 hours ago. Combine them with -u and you can isolate one incident on one service in seconds.

Narrow by priority, then follow

-p filters by syslog priority. -p err shows errors and worse. -p warning includes warnings too:

journalctl -u demo-api.service -p err -b

-- No entries --

An empty result here is meaningful. The service logged no errors this boot, so the problem is somewhere else. Read that as a finding, not a failed search.

-f follows new entries as they arrive, like tail -f for one unit:

journalctl -u demo-api.service -f

Keep that running in one terminal while you restart the service or send a test request in another. Watching cause and effect side by side is the fastest way to connect them.

Previous boots need persistent storage

Logs from the previous boot are one flag away: -b -1. But that only works if the journal is stored on disk. If you see this instead, the journal lives in memory and died with the reboot:

Specifying boot ID or boot offset has no effect, no persistent journal was found.

On Ubuntu the fix is creating the on-disk directory and restarting journald:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

After that, journalctl --list-boots shows every boot the journal has kept. Do this before you need it. The most interesting boot is usually the one that just crashed, and by then it is too late to turn persistence on.

Watch the disk

Persistent logs grow. Check how much space they take:

journalctl --disk-usage

Archived and active journals take up 1.2G in the file system.

journalctl --vacuum-size=500M trims old files down to a size. The default limits in /etc/systemd/journald.conf are usually sane, but I check this once on every new server.

Try this with a recent service event: a restart, an error, a deploy. Write the smallest journal query that includes the event and enough surrounding context to explain it. If your query returns more than a screen or two, add another filter.