In this series, we’ve introduced you to the basics of syslog, how it works and its key components. We looked at the configuration of syslog using the rsyslog.conf file, how to set up log rotation and the best practices to follow for collecting and analyzing logs.

Now, we’re introducing you to syslog-ng, one of the most feature-rich, well-documented versions of syslog. It can make your logging environment faster, more secure and more reliable. It also integrates with security information and event management (SIEM) and log management solutions.

In this third part, we cover the background of syslog-ng, consider its benefits over traditional versions of syslog and show you how to install and configure it.

What Is Syslog-ng?

Syslog-ng (“syslog new-generation”) facilitates the transmission of source logs to a remote destination using predefined filters. It goes beyond basic syslog functionality by supporting TCP, TLS encryption, advanced filtering and logging to a database. Syslog-ng also allows customization and can facilitate almost any logging need.

Syslog-ng was developed in 1998 by Balázs Scheidler and became the default logging daemon for operating systems like Debian, Gentoo and SUSE. It is now maintained by One identity (after their acquisition of Balabit), which develops both open-source and commercial products under the syslog-ng brand.

Syslog-ng and rsyslog (which we discussed in part two of this series) are both syslog daemons that enhanced the original syslog to provide faster data transmission with wider support across operating systems and programs. While rsyslog was forked from the standard syslogd package, syslog-ng is a totally separate daemon with a new configuration and syntax.

Benefits of Using Syslog-ng

Syslog-ng built upon the traditional versions of syslogd and rsyslog to enhance its features and improve the security, ease of use and documentation for its users. Let’s consider some of the specific enhancements.

Support for TCP and TLS encryption

Syslog-ng overcame one of the biggest drawbacks of traditional syslog by introducing support for TCP and TLS encryption. TCP gives you a more reliable log transfer than UDP and ensures log messages reach their destination. Some logs contain highly sensitive information, so TLS encryption ensures your logs are safe during transmission.

Simpler Configuration

Syslog-ng developed a new configuration format that most people find easier to understand than syslogd or rsyslog. The configuration file is clean, well structured and gives you the ability to reuse sources, destinations or filters throughout the configuration file, making it simpler to maintain.

Ability to Classify, Tag and Correlate Log Messages in Real Time

Syslog-ng can scan the content and parameters of incoming logs in real time. This makes it easy to apply complex filters that direct logs to different destinations or drop logs that are unimportant to reduce noise in the logging system.

Syslog-ng can also enrich logs by adding data from an external lookup file or by correlating incoming logs with a common field such as hostname or program that generated the log. This correlation can make analysis easier and enhance business insights.

Wide Support

Syslog-ng supports a wide range of message formats, including RFC3164, RFC5424, JSON and Journald. It can also run on multiple operating systems and architecture, including Linux, Unix, BSD and Solaris. The high level of adoption across the Linux community gives users access to more forums, articles and documentation with use cases to improve their usage. This also improves the likelihood that syslog-ng will continue to be used well into the future.

Message Flow Control Across the Network

From a networking level, syslog-ng can send and receive messages in IPv4 and IPv6 environments.

Network connectivity issues or performance degradation can prevent logs from being accepted by the destination server. With syslogd, these messages would likely have been lost. Syslog-ng flow control overcomes this, as it will pause the reading of messages until it has determined that it can send them to the destination successfully, acting as a buffer to maintain high log delivery rates.

Configuring Syslog-ng

In this section, we show you how to install syslog-ng and configure the syslog-ng conf file.

Installing Syslog-ng

Depending on your Linux distribution, you can use a package manager to install syslog-ng. For Ubuntu, Mint or other Debian operating systems, you can use apt.

Configuring Syslog-ng.conf

The syslog-ng.conf file contains configuration for the source, destination, filter and log statements to determine how to deal with incoming and outgoing log messages. You’ll find the configuration file at /etc/syslog-ng/syslog-ng.conf. Open a text editor to edit this file.

Source

The source configures how syslog-ng receives log messages. Each source contains one or more source drivers which define how and where we receive messages.

source <identifier> {
  source-driver(parameter); source-driver(parameter); ...
};

We define the source once and give it an <identifier>, and then we can reference this <identifier> throughout the rest of the configuration file.

The source-driver defines how and where syslog-ng receives messages. The table below is from the syslog-ng documentation and shows the available source drivers.

Source Driver	Description
file()	Opens the specified file and reads messages.
internal()	Internal syslog-ng logs
network()	Receives messages from remote hosts using the BSD-syslog protocol over IPv4 and IPv6. Supports the TCP, UDP, and TLS network protocols.
pipe()	Opens the specified named pipe and reads messages.
program()	Opens the specified application and reads messages from its standard output.
sun-stream(), sun-streams()	Opens the specified STREAMS device on Solaris systems and reads incoming messages.
syslog()	Listens for incoming messages using the new IETF-standard syslog protocol.
system()	Automatically detects which platform is running on, and collects the native log messages of that platform.
systemd-journal()	Collects messages directly from the journal of platforms that use systemd.
systemd-syslog()	Collects messages from the journal using a socket on platforms that use systemd.
unix-dgram()	Opens the specified unix socket in SOCK_DGRAM mode and listens for incoming messages.
unix-stream()	Opens the specified unix socket in SOCK_STREAM mode and listens for incoming messages.

Example

This example source statement (s_tcp) tells the syslog server to listen on TCP port 9999.

source s_tcp {
  network (
    transport("tcp")
    port(9999)
  );
};

Destination

The destination is where syslog-ng directs log messages. It follows the same syntax as the source block.

destination <identifier> {
  destination-driver(params); destination-driver(params); ...
};

Example 1

destination d_router_logs {
    file("/var/log/network/router/routerlogs.log");
};

You can also use variables to create dynamic file names, splitting them out logically by date, time, host or other factors. In the example above, all logs, regardless of their source host or log time, are sent to the file routerlogs.log. In the following example, we use variables to send logs to a dynamic file location depending on the source host and date.

Example 2

destination d_router_logs {
   file("/var/log/network/router/$HOST/$YEAR-$MONTH-$DAY.log");
};

The table below shows the available syslog-ng destination drivers. You can find out more information about each destination driver in the syslog-ng documentation.

Destination Driver	Description
elasticsearch2	Sends messages to an Elasticsearch server. The elasticsearch2 driver supports Elasticsearch version 2 and newer.
file()	Writes messages to the specified file.
hdfs()	Sends messages into a file on a Hadoop Distributed File System (HDFS)[3] node.
kafka()	Publishes log messages to the Apache Kafka[4] message bus, where subscribers can access them.
loggly()	Sends log messages to the Loggly[5] Logging-as-a-Service provider.
logmatic()	Sends log messages to the Logmatic.io[6] Logging-as-a-Service provider.
mongodb()	Sends messages to a MongoDB[7] database.
usertty()	Sends messages to the terminal of the specified user, if the user is logged in.
network()	Sends messages to a remote host using the BSD-syslog protocol over IPv4 and IPv6. Supports the TCP, UDP, and TLS network protocols.
pipe()	Writes messages to the specified named pipe.
program()	Forks and launches the specified program, and sends messages to its standard input.
sql()	Sends messages into an SQL database. In addition to the standard syslog-ng packages, the sql() destination requires database-specific packages to be installed.
syslog()	Sends messages to the specified remote host using the IETF-syslog protocol. The IETF standard supports message transport using UDP, TCP, and TLS networking protocols.
unix-dgram()	Sends messages to the specified unix socket in SOCK_DGRAM style (BSD).
unix-stream()	Sends messages to the specified unix socket in SOCK_STREAM style (Linux).

Filter

Filters are optional and define criteria for how syslog-ng will route log messages to different destinations. You can create simple filters for things like host or severity, or you could add more complex logic with Boolean expressions.

filter <identifier> {
  <filter_type>("<filter_expression>");
};

Example 1

The example below declares a filter, f_router, and matches any logs with the text “allow” from host 10.1.2.3.

filter f_router {
  host("10.1.2.3") and match("allow" value("MESSAGE"));
};

The table below shows the available syslog-ng filters.

Filter	Description
facility()	Filter messages based on the sending facility.
filter()	Call another filter function.
host()	Filter messages based on the sending host.
inlist()	File-based whitelisting and blacklisting.
level() or priority()	Filter messages based on their priority.
match()	Use a regular expression to filter messages based on a specific header or content field.
message()	Use a regular expression to filter messages based on their content.
netmask()	Filter messages based on the IP address of the sending host.
program()	Filter messages based on the sending application.
source()	Select messages of the specified source statement.
tags()	Select messages having the specified tag.

Log Path

Log paths are found at the end of the configuration file, defining the connections between the source, destination, and any filters. To define a log path, we can add a log statement based on the syntax shown below. Log statements are processed in order, so you must position them correctly to ensure your logs end up at the right destination.

log {
  source(s1); source(s2); ...
  optional_element(filter1|parser1|rewrite1);
  optional_element(filter2|parser2|rewrite2);
  ...
  destination(d1); destination(d2); ...
  flags(flag1[, flag2...]);
};

Example 1

The following log statement sends messages arriving to localhost on port 514 to a remote server on port 9997. Only messages containing error are included.

source s_localhost {
  network(
    ip(127.0.0.1)
    port(514)
  );
};

destination d_tcp {
  network("10.1.2.3"
    port(9997)
  );
};

filter f_error { 
  host("example") and match("error" value("MESSAGE")); 
};

log {
  source(s_localhost);
  filter(f_error);
  destination(d_tcp);
};

When declaring the source, destination and filters, the common convention is to prefix the name with s_, d_, or f_. This makes it easy to distinguish references later in the log statement.

Log your data with CrowdStrike Falcon Next-Gen SIEM

Elevate your cybersecurity with the CrowdStrike Falcon^® platform, the premier AI-native platform for SIEM and log management. Experience security logging at a petabyte scale, choosing between cloud-native or self-hosted deployment options. Log your data with a powerful, index-free architecture, without bottlenecks, allowing threat hunting with over 1 PB of data ingestion per day. Ensure real-time search capabilities to outpace adversaries, achieving sub-second latency for complex queries. Benefit from 360-degree visibility, consolidating data to break down silos and enabling security, IT, and DevOps teams to hunt threats, monitor performance, and ensure compliance seamlessly across 3 billion events in less than 1 second.

Schedule Falcon Next-Gen SIEM Demo