Basic Log Management With Journalctl

On servers, logging is extremely important in determining the cause when an issue occurs. On newer systems that integrate systemd, systemd-journald was introduced to bring logs together in one place. Before log messages were written to separate files for each application typically located somewhere with /var/log. Journald changes that as it allows logs to be stored within journald and accessed with the journalctl command, freeing administrators from those moments of recollection trying to remember where that one log file is located. In this blog post, we will explore the journalctl command from syntax, to understanding, and more.

Introduction To Journalctl

On Linux distributions using systemd, the journalctl command is used to interact with the systemd journal. Systemd-journal is a centralized logging system that allows services to log to a single place. Unlike before this means you no longer need to go searching for log files for each service.

Journal Configuration

A few notes on journald configuration worth noting:

• Log data stored is stored in /var/log/journal in separate journal files but are stored in a non human-readable format hence the need for a tool to read, filter, and sort.
• configuration file is located at /etc/systemd/journald.conf

Notes Before We Get Started

• Be aware the provided by journalctl will be in the system-configured date and time unless otherwise specified.
• In some cases, you may find that journalctl is not keeping logs from before a reboot. This is due to configuration and is something that can be changed.
• To access the system journal you will either need to be part of a group or user that is granted access. Besides the root user that would normally include sudo users and users that are part of the systemd-journal, adm, or wheel groups (this is distro-dependent).
• Users who attempt to access journalctl without permission will still get information just not from the system journal, rather they will only get results related to their user.

Basic Syntax

The basic syntax of the command looks something like this:

journalctl [OPTIONS]

These options are optional though as you can run the command without any options to display entries using the default output format.

Understanding Log Entries

The default layout for log entries is as follows:

May 28 14:20:35 fedora systemd[1]: Starting sysstat-collect.service – system activity accounting tool…

1. The Timestamp
2. The hostname
3. The Service and PID
4. The log entry/log message

It should be noted that more fields are part of an entry and this will even vary between distributions and even entries. You can see a full list by running the command below, though if you do be warned you may want to pipe it to less.

journalctl --fields

Flags For Sorting And Filtering

Below are the flags I think will be most useful when sorting and filtering through journal entries.

–utc – Converts date and time to UTC, useful when working with machines across time zones.

-f, –follow – Tells journalctl you want to follow the live log entries.

-r, –reverse – Will sort from newest to oldest.

-n, –lines=[lines] – Display a specific number of lines from newest to oldest.

-k, –dmesg – Show kernel messages from the current boot only.

-u, –unit=UNIT|PATTERN – Will search for logs from system services by name or pattern.

–list-boots – Lists the boot ID’s currently available in the logs.

-b, –boot=[ID] – Shows only entries from the current boot or the specified boot ID.

-g, –grep=[pattern] – Without needing to pipe to grep, this flag allows you to search for keywords.

_PID=[ID] – Allows you to search for logs from a specific process ID

_UID=[ID] – Allows you to search for a specific user, but the catch is you need to know the user’s UID first. To retrieve this use id -u username.

_GID=[ID] – Allows you to search by group ID rather than user ID.

-S –since=[DATE] – Show entries not older than the specified date. The date can be specified in relative terms like yesterday, today, -1h, -15m, on top of a timestamp.

-U –until=[DATE] – Show entries not newer than the specified date. The date can be specified in relative terms like yesterday, today, -1h, -15m, on top of a timestamp.

-p, –priority= – Lets you filter journal records by the priority of the event. You use the number below for the type of event you want to see.

0 Emergency

1 Alert

2 Critical

3 Error

4 Warning

5 Notice

6 Informational

7 Debug

Disk Usage

In this section, we will go over how to determine how much disk space is being used by the systemd journal as well as how to clear up some of that usage

Finding Current Disk Usage

To determine how much space journald is using run the following command:

journalctl --disk-usage

This will result in output similar to the following.

Archived and active journals take up 885.2M in the file system.

Deleting Old Logs

There are two options on how to clean up old logs. The first is by size and the second is by date below are examples of both.

journalctl --vacuum-size=200M

In this example, we delete all but the newest 200M of logs.

journalctl --vacuum-time=2weeks

In this example, we delete logs older than two weeks.

Conclusion

In this blog post, we started by introducing journalctl as the command used to interact with logs stored within journald. From there touched on basic syntax and what log entries typically look like. Next, we went over the most useful flags for filtering and sorting. Lastly, we touched on disk usage both how to see current usage and how to delete old entries.

Additional Links

Are you done reading and looking for more to read why not check these out?

• Systemd Homepage
• Systemd Wikipedia
• Arch Wiki Systemd journal