In part one of this guide, we introduced the basics of Apache web server logging. We looked at the Apache access and error logs, their locations, log formats, and different log levels. We showed how to set logging parameters, bother server-wide and for each VirtualHost. We also covered how to use the Linux  logrotateutility to rotate the logs.

Now, we’ll take it to the next level by covering advanced concepts like conditional logging, logging modules, log file integrity monitoring, and centralized log management.

Conditional Logging

Conditional logging allows you to skip certain client requests from being recorded based on specific conditions. This is helpful in keeping the log size manageable and eliminating unnecessary noise. You can use the  SetEnvIfdirective with  CustomLogin the Apache configuration file to set up conditional logging.

For example, let’s say you don’t want to log the requests sent from your IP address in the access log. The code snippet below shows how you can set this in the Apache web server’s configuration file:

SetEnvIf Remote_Addr "16.45.41.41" nolog
CustomLog "/var/log/httpd/access_log" common env=!nolog

This assumes your local IP address is 16.45.41.41. The setting will take effect once you restart the Apache service.

Similarly, you can exclude requests sent to a particular URL. For example, the configuration below specifies Apache should not log requests for the 404.html page.

SetEnvIf Request_URI "^/404.html$" nolog
CustomLog "/var/log/httpd2/access_log" common env=!nolog

Logging Modules

Apache uses modules to extend the functionality of the base web server application. Modules can have different statuses like MPM, Base, Experimental, Extension, or External. Logging modules are used to enhance the logging capabilities of Apache. Below are some common logging modules.

mod_log_config

mod_log_config is Apache’s base logging module, enabled by default in RHEL and Debian-based systems. It’s identified as  log_config_module, and you can verify it using the following commands:

#On Ubuntu
apache2ctl -M | grep log_config_module 

#On CentOS
httpd -M | grep log_config_module

 mod_log_configprovides functionalities like  CustomLogand conditional logging.

mod_log_debug

mod_log_debug is an experimental module used to capture debug information. For example, you can use it to log a message whenever a URL request is processed, a request times out, or the value of an environment variable in the client request needs to be persisted.

Your Apache web server might not have this module enabled by default. You can use the configuration setting below to enable it.

#On Ubuntu
LoadModule log_debug_module modules/mod_log_debug.so x

#On Ubuntu
LoadModule log_debug_module /usr/lib/apache2/modules/mod_log_debug.so

mod_log_forensic

mod_log_forensic provides forensic logging of client requests. This includes logging all events before and after processing a particular client request. The requests are all assigned the same ID, which helps triage any issues and correlate events.

By default, this module is not enabled, but you can do so by adding the following directives in the Apache configuration file:

# On CentOS
LoadModule log_forensic_module modules/mod_log_forensic.so
ForensicLog logs/forensic_log

#On Ubuntu
LoadModule log_forensic_module /usr/lib/apache2/modules/mod_log_forensic.so
ForensicLog ${APACHE_LOG_DIR}/forensic_log

Here is a sample forensic log entry after enabling this module:

+YoyI7ZlV8b6Dt73pCgvzMgAAABA|GET / HTTP/1.1|Host:34.227.9.153|Connection:keep-alive|Cache-Control:max-age=0|Upgrade-Insecure-Requests:1|User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36|Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9|Accept-Encoding:gzip, deflate|Accept-Language:en-GB,en-US;q=0.9,en;q=0.8|If-None-Match:"11-5df853ad7a394"|If-Modified-Since:Sat, 21 May 2022 13%3a03%3a14 GMT
-YoyI7ZlV8b6Dt73pCgvzMgAAABA

Log File Integrity Monitoring

File integrity monitoring is an important part of log file management. Without a hardened server configuration, malicious users can modify the log files or delete them—as hackers often do. File integrity monitoring enables you to assess which users are accessing a file or directory and what system calls they are making against those.

Monitoring with auditd

You can use the Linux auditd utility to monitor file integrity. The command snippet below adds an auditd rule to watch for any changes made to the /var/log/httpd directory and append a key called apache_watch to the event. This makes it easier to search for rule violations from the auditd log.

auditctl -w /var/log/httpd/ -k apache_watch

To demonstrate this, we would start by making some changes to the directory (for example, accessing, deleting, or creating files). Then, we would run the following command to check auditd log:

ausearch -k apache_watch | aureport -f -i

We see entries like the following:

File Report
===============================================
# date time file syscall success exe auid event
===============================================
1. 05/24/2022 08:20:46 /var/log/httpd/ openat yes /usr/bin/bash centos 7479
2. 05/24/2022 08:20:47 /var/log/httpd/a openat yes /usr/bin/touch centos 7480
3. 05/24/2022 08:21:14 /var/log/httpd/ openat yes /usr/bin/bash centos 7482
4. 05/24/2022 08:21:15 /var/log/httpd/ openat yes /usr/bin/bash centos 7483
5. 05/24/2022 08:21:17 /var/log/httpd/access_log openat yes /usr/bin/vi centos 7484
6. 05/24/2022 08:21:17 /var/log/httpd/access_log readlink no /usr/bin/vi centos 7485
7. 05/24/2022 08:21:17 /var/log/httpd/.access_log.swp openat yes /usr/bin/vi centos 7486
8. 05/24/2022 08:21:17 /var/log/httpd/.access_log.swpx openat yes /usr/bin/vi centos 7487
9. 05/24/2022 08:21:17 /var/log/httpd/.access_log.swpx unlink yes /usr/bin/vi centos 7488
…
… 
14. 05/24/2022 08:21:17 /var/log/httpd/.access_log.swp chmod yes /usr/bin/vi centos 7493
15. 05/24/2022 08:21:17 /var/log/httpd/access_log openat yes /usr/bin/vi centos 7494
16. 05/24/2022 08:21:21 /var/log/httpd/.access_log.swp unlink yes /usr/bin/vi centos 7495
17. 05/24/2022 08:21:43 /var/log/httpd/access_log openat yes /usr/bin/vi centos 7496
18. 05/24/2022 08:21:43 /var/log/httpd/access_log readlink no /usr/bin/vi centos 7497
19. 05/24/2022 08:21:43 /var/log/httpd/.access_log.swp openat yes /usr/bin/vi centos 7498
20. 05/24/2022 08:21:43 /var/log/httpd/.access_log.swpx openat yes /usr/bin/vi centos 7499
21. 05/24/2022 08:21:43 /var/log/httpd/.access_log.swpx unlink yes /usr/bin/vi centos 7500
22. 05/24/2022 08:21:43 /var/log/httpd/.access_log.swp unlink yes /usr/bin/vi centos 7501
…
…
30. 05/24/2022 08:21:46 (null) fchown yes /usr/bin/vi centos 7509
31. 05/24/2022 08:21:46 /var/log/httpd/4913 unlink yes /usr/bin/vi centos 7510
32. 05/24/2022 08:21:46 /var/log/httpd/ unlink no /usr/bin/vi centos 7511
33. 05/24/2022 08:21:46 /var/log/httpd/access_log~ rename yes /usr/bin/vi centos 7512
34. 05/24/2022 08:21:46 /var/log/httpd/access_log openat yes /usr/bin/vi centos 7513
35. 05/24/2022 08:21:46 /var/log/httpd/access_log~ getxattr yes /usr/bin/vi centos 7514
36. 05/24/2022 08:21:46 /var/log/httpd/access_log getxattr yes /usr/bin/vi centos 7515
37. 05/24/2022 08:21:46 /var/log/httpd/access_log setxattr yes /usr/bin/vi centos 7516
…
…

You can use the report to find out who’s accessing your log files. From there, if needed, you can restrict access to the log files and their directories.

Piping Logs to Syslog

You can also send your Apache web server logs directly to Syslog. To enable this, replace all ErrorLog and CustomLog settings in the Apache configuration file, as shown here:

ErrorLog  "| /usr/bin/logger -thttpd -plocal6.err"
CustomLog "| /usr/bin/logger -thttpd -plocal6.notice" extended_ncsa

With this setting, once you restart the Apache web server service, it will no longer write events to the native access log or error log files. Instead, you will find all Apache log entries in /var/log/messages (on RHEL-based systems) or /var/log/syslog (on Debian-based systems).

Centralized Log Management

It’s neither practical nor ideal to store Apache logs on the web server machine itself, particularly in a production environment. Logs can grow very large, consuming all the disk space on the server, resulting in service interruption. It takes longer to open and write to large log files, and this impacts performance. Finally, any issues with the underlying storage (such as disk block corruption) can render the log file inaccessible.

Additionally, you may have multiple web servers running behind a load balancer in a cloud environment. Some web servers may be ephemeral, spun up or down spending on the current load. Storing logs on individual servers can be ineffective or disruptive, especially if you’re losing important log files when deprovisioning a server. This makes it difficult to search, analyze, and correlate events consistently.

To address these shortcomings, consider setting up a centralized log management solution and pushing all Apache web server logs to that solution. This enables efficient log storage and enhances the ability to search, analyze, and correlate events.

Log Analysis with Visualization

Having all the logs in one place doesn’t help if you can’t analyze them and make sense of the data. Log analysis helps you find application performance bottlenecks or security issues, while log visualization helps you see key metric values and trends on a graphical interface.

Such analysis and visualization are important for insights into your web application’s availability, performance, and security. You can create KPI reports on these analyses and set up alerts. Log analysis is also critical for security and compliance. Correlating web server log events against other application logs can reveal potential threats and ongoing breaches.

Log your data with CrowdStrike Falcon Next-Gen SIEM

Elevate your cybersecurity with the CrowdStrike Falcon^® platform, the premier AI-native platform for SIEM and log management. Experience security logging at a petabyte scale, choosing between cloud-native or self-hosted deployment options. Log your data with a powerful, index-free architecture, without bottlenecks, allowing threat hunting with over 1 PB of data ingestion per day. Ensure real-time search capabilities to outpace adversaries, achieving sub-second latency for complex queries. Benefit from 360-degree visibility, consolidating data to break down silos and enabling security, IT, and DevOps teams to hunt threats, monitor performance, and ensure compliance seamlessly across 3 billion events in less than 1 second.

Schedule Falcon Next-Gen SIEM Demo