Linux is an open-source operating system originating from the Unix kernel. It’s now one of the most used operating systems across devices. You may be familiar with the various flavors of Linux, including Ubuntu, Centos, and Red Hat Enterprise Linux (RHEL). Their common Linux core means all these operating systems have a logging framework installed to monitor the system and its services.
The logging framework for Linux includes a set of directories, files, services, and commands that administrators can use. As a Linux system administrator, knowing your way around the Linux log locations, commands, and configuration will be essential for troubleshooting issues on the systems or applications you administer.
This article is part one of a two-part series on Linux logging. In part one, we will go through the basics of Linux logs: the common Linux logging framework, the locations of these log files, and the different types of logging daemons and protocols (such as syslog and rsyslog). Finally, we’ll review some common Linux log commands to read and search through the logs on a system.
Common Linux Logs and Their Locations
With the Linux logs pattern, you will find logs located under the
/var/log directory, with files and directories for each service or stream of log messages on the system. Common log files include:
/var/log/messages(RHEL): This is a consolidated stream of general system messages and metrics. In this log file, you can find messages from services like mail, kernel, authentication, and cron.
/var/log/secure(RHEL): This file contains authentication logs for both successful and failed login attempts on the system.
/var/log/wtmp: This file contains a history of all user login and logout activity for auditing users’ activity on the system.
/var/log/lastlog: Similar to the
wtmpaudit file, this log file tracks users’ last logins. This is a binary file you can read via the
/var/log/kern.log: This file contains log messages produced by the kernel before being passed to the system logging service (such as rsyslog) for further processing.
/var/log/cron: The cron service runs as an orchestrator to start tasks at scheduled intervals. Messages from this service—such as when a cron job has started and if any errors occurred during its execution—can be found in this log file.
When running services on your systems such as PostgreSQL or Apache, your application-specific logs are made available under subdirectories in
/var/log. For example, if you are running an Apache web server on your Debian-based Linux system, you will find log files under the
/var/log/apache2 directory. You would find separate files for each log stream in this directory, such as the
access.log or the
Introduction to Syslog
Syslog is a network-based logging protocol that monitors your systems and applications. This protocol provides a standard way for services and applications to report their logs. That way, they can be processed and redirected as needed.
Standardized message format
The syslog protocol provides a message format defined by the RFC 5424 standard. In this format, common event information is defined, such as the timestamp, hostname, and the name of the application that produced the message. To further support the structuring of this message, syslog facilities are available to denote which part of the system the log comes from. This is done by attaching a number to the message. Below is a list of all available facilities, numbered from 0 to 23:
|5||syslog||Messages generated internally by syslogd|
|6||lpr||Line printer subsystem|
|7||news||Network news subsystem|
|16-23||local0 - local7||Locally used facilities|
Similarly, priority can be attached to a message using a number between 0 and 7.
|0||emergency||System is unusable|
|1||alert||Action must be taken immediately|
|5||notice||Normal but significant condition|
By using both the facilities and priorities in the syslog message, tools that access the syslog data can now filter messages based on the originating facility and the severity of messages. We’ll see an example of this in the next section.
Syslog Protocol Implementations
The syslog process runs as a daemon on the system to receive, store, and interpret syslog messages from other services or applications. That service typically listens on port
514 for TCP and
601 for UDP connections. Many applications allow you to configure their event logging to push messages to a running syslog service.
The syslog protocol is also implemented by different services like rsyslog and syslog-ng, allowing you to choose a service based on the feature set you need. Because these services have aligned to the syslog protocol, they are interchangeable for system and application logging, making them very scalable.
The Rsyslog Daemon
Rsyslog is a modern, open-source implementation of the syslog daemon, offering a high-performance, security-focused, modular design for any environment. The rsyslog daemon runs as a service on your host, listening for log messages sent to it and routing those messages based on defined actions.
In a typical installation of rsyslog, the daemon is configured through a file located at
/etc/rsyslog.conf. In this config file, using selectors for the facilities and priority of the log message allows you to define what action should be carried out for the message. In the following example, any messages with the facility of
notice or higher will be written to a log file located at
# <facility>.<severity> <action> mail.notice /var/log/mail_errors
These selectors are structured by facility (origin of the message) and priority (severity of the message), separated by a dot. The example below shows some possibilities of using this simple configuration to perform actions on incoming logs.
# Log a message to file mail.notice /var/log/mail_errors # Log a message to a user Kern.debug bob # Emergency messages from any facility should go to all users *.emerg * # Log a message to another host over UDP *.* @remote-host # Log a message to another host over TCP *.* @@remote-host:514
Basic Commands for Linux Logging
As an administrator of Linux servers, you will often connect to these servers to read log messages for troubleshooting systems or the services running on them. Several utility commands are available on Linux systems, simplifying how you navigate stored log messages. The following section outlines some basic log commands available:
cat: Short for concatenate, which allows you to view the contents of one or more files in the terminal.
more: Similar to
catutility, this command reads the content of files in the terminal. However, this utility will interactively display it one page at a time to the user for an easier manual reading experience.
less: Much like the
moreutility, this command displays a single terminal screen of content at a time, allowing for easier navigation of large text files.
tail: By default,
taildisplays the last ten lines written to a file. Using the follow option (
--follow) allows you to monitor the file continuously. As new lines are written, they are printed to the user’s terminal.
head: This utility is the opposite of the
tailcommand, fetching the beginning lines of a file. By default,
headwill display the first ten lines of a file.
grep: This command allows you to parse input text using filters and regex to find specific patterns in the text. It is useful for searching and manipulating text in scripts or automation.
With these basic commands, you can easily access and navigate the log messages on your system. By using pipes (
|) in your commands, you can chain multiple commands together, filtering their outputs even further. For example, the following chain of commands will read the contents of the
/var/log/cron file and check if any message contains the string
cat /var/log/cron | grep "foo"
Advanced logging operations can also be done with other commands like
cut, and advanced
grep filters, allowing you to gain more insight into what happens on your system.
Log Everything, Answer Anything – For Free
Falcon LogScale Community Edition (previously Humio) offers a free modern log management platform for the cloud. Leverage streaming data ingestion to achieve instant visibility across distributed systems and prevent and resolve incidents.
Falcon LogScale Community Edition, available instantly at no cost, includes the following:
- Ingest up to 16GB per day
- 7-day retention
- No credit card required
- Ongoing access with no trial period
- Index-free logging, real-time alerts and live dashboards
- Access our marketplace and packages, including guides to build new packages
- Learn and collaborate with an active community