In our series so far, we’ve introduced the basics of syslog, how it works and its key components. We’ve demonstrated how to install and configure rsyslog and syslog-ng, how to manage log rotation and the best practices to follow when managing your syslog environment.

Now, we’ll consider a log management solution that can help you transform your syslog data into business insights. Through centralization, visualizations and automation, log management solutions can empower you to take action using the data you collect.

In the fourth post, we look at the benefits of using a log management system and how to integrate its usage with syslog.

Centralizing Syslog With a Log Management Solution

Log management solutions are software products that help you organize large volumes of data for analysis. They have different components so you can collect, parse, store, analyze and visualize data from your devices and applications.

In most systems, you can collect data directly from the source, using an agent or API or collecting it from your syslog servers. You can also parse common data formats (such as syslog or JSON) to extract key-value pairs for analysis.

Log management solutions allow your security team to investigate attacks, alert the infrastructure team of an outage and even help developers refine their code. Cloud-based solutions offer flexible storage options so you can ingest as much data as you want and store it for as long as you need.

Many IT environments use syslog together with a log management solution. Devices with a high log volume, such as firewalls or routers, often send to a syslog server first before sending to the log management system. As syslog servers are usually in the same region as the device, this practice reduces latency and prevents data loss. Syslog servers can also filter out unnecessary logs, reducing what’s sent to the log management system.

Log management solutions enable you to centralize the logs from all your systems, regardless of where they’re hosted. While it’s technically possible to do this with your own custom scripts or programs, log management solutions are easier to configure and maintain. Lots of use case solutions come out-of-the-box or are easy to add or customize.

Better Management of Syslog

Log management solutions can bring your attention to problems with your syslog servers. Use built-in alerts or your own custom queries to identify if a server has stopped sending logs, or if it’s sending fewer logs than usual. Forward internal syslog logs to spot critical errors so you can take remedial action quickly.

Use alerts to trigger automatic actions like sending emails to key team members, creating tickets, unlocking accounts or extending disk space. Without centralization, it would be difficult to manage this type of automation across all syslog servers. You would need to perform manual tasks like deploying identical scripts across environments, managing credentials and opening network access. At scale, this becomes unmanageable.

On the other, cloud-based log management solutions simplify log centralization. They also let you store syslog logs for a longer time, facilitating better historical analysis. These solutions make it easier and cheaper to purchase additional storage and often give you the flexibility to add and remove as your needs fluctuate.

Best Practices When Using Log Management Solutions

Log management solutions can be incredibly useful. But it’s important to remember that you must run and maintain the system properly in order to get the most from it. By ignoring best practices, you risk paying for more data than you need, developing dashboards that no one uses and choosing a platform that doesn’t deliver the features you need. Although there are many best practices, ensuring that your data is actionable and leveraging the cloud are two key points.

Actionable Data

Even though log management solutions can make it cheaper to store more data for longer, be careful not to ingest data simply for the sake of it. Establish a data onboarding process where you can review logs with key stakeholders to understand how to use the data to take action in the business.

On your syslog servers, consider implementing filters to drop events you won’t need.

Leverage the Cloud

Log management solutions that leverage the cloud provide you with extra flexibility, better pricing, increased storage and access to more locations around the world. Investing in on-premises hardware to complete the same logging features can be a financial burden and may take longer to set up and maintain long term.

Management of an on-premises system can take time away from analysis as you often need to troubleshoot hardware issues, patch security vulnerabilities, perform upgrades, etc. Cloud vendors usually take care of all of this for you and guarantee nearly 100% uptime and improved visibility of critical alerts.

Integrating CrowdStrike Falcon LogScale With Syslog

When working with syslog, you can leverage rsyslog to ship your logs to CrowdStrike Falcon® LogScale, taking advantage of pre-built integrations between rsyslog, the Elasticsearch format and Falcon LogScale.

Rsyslog includes the Elasticsearch Output Module plugin, which provides support for logging to Elasticsearch. By using this plugin, rsyslog formats the logs into the Elasticsearch format which Falcon LogScale supports for ingestion.

After installing rsyslog and rsyslog-elasticsearch on your local machine, create a configuration file for rsyslog. CrowdStrike recommends naming this file /etc/rsyslog.d/33-logscale.conf. The configuration should look like this:

module(load="omelasticsearch")

template(name="logscaletemplate" type="list" option.json="on") {
  constant(value="{")
  constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
  constant(value="\",\"message\":\"") property(name="msg")
  constant(value="\",\"host\":\"") property(name="hostname")
  constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
  constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
  constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
  constant(value="\",\"name\":\"") property(name="programname")
  constant(value="\",\"pid\":\"") property(name="procid")
  constant(value="\"}")
}

*.* action(type="omelasticsearch"
        server="$LOGSCALE_URL/$LOGSCALE_INGEST_PORT"
        template="logscaletemplate"
        uid="logscale-tutorial"
        pwd="$INGEST_TOKEN"
        bulkmode="on"
        usehttps="on"

Details about the specific configurations can be found in the documentation for the rsyslog log shipper or this walkthrough post.

For $LOGSCALE_URL, use the endpoint for your Falcon LogScale instance. Users of Falcon LogScale’s Community Edition should use https://cloud.community.humio.com and $LOGSCALE_PORT should be set to 443. For $INGEST_TOKEN, use the ingest token generated for your log repository at Falcon LogScale.

After you have configured rsyslog, restart it with the following command:

$ sudo systemctl restart rsyslog.service

You will begin seeing log entries from your system appear in your Falcon LogScale repository.

As you can see, integrating Falcon LogScale with your syslog setup is simple and straightforward.

Log your data with CrowdStrike Falcon Next-Gen SIEM

Elevate your cybersecurity with the CrowdStrike Falcon^® platform, the premier AI-native platform for SIEM and log management. Experience security logging at a petabyte scale, choosing between cloud-native or self-hosted deployment options. Log your data with a powerful, index-free architecture, without bottlenecks, allowing threat hunting with over 1 PB of data ingestion per day. Ensure real-time search capabilities to outpace adversaries, achieving sub-second latency for complex queries. Benefit from 360-degree visibility, consolidating data to break down silos and enabling security, IT, and DevOps teams to hunt threats, monitor performance, and ensure compliance seamlessly across 3 billion events in less than 1 second.

Schedule Falcon Next-Gen SIEM Demo