What is a Web Server Log?

September 30, 2022

A web server log is a text document that contains a record of all activity related to a specific web server over a defined period of time. The web server gathers data automatically and constantly to provide administrators with insight into how and when a server is used, as well as the users that correspond with that activity.

While server data is available immediately within the server log itself, in most cases the log file is also stored in a database and can be used to produce customized reports on demand. This information can be used by administrators to better understand and accommodate web traffic patterns, better allocate IT resources, and adapt sales and marketing activity.

Standard log file format

Most web servers generate a log file in the Common Log Format (CLF) for every HTTP request. Each log entry contains detailed information about the request, including: the IP address of the device; the date and time of the request; the name and location of the requested file; and the file size.

Since CLF files are raw log files, they tend to be of limited use to administrators or webmasters in their existing form. Further, given the volume of data captured, most organizations automatically delete CLF files after a given time.

Server Log Content and Values

Each line within the server log file contains significant information, including:

  • The device’s IP address
  • Request method
  • Date and time of the request
  • Status of the request
  • Referrer method
  • User-Agent
  • Requested file information, including file name, size and network location

The server log data can also be used to produce the following insights:

  • Number of users who accessed the server
  • Number of unique or authenticated visitors
  • Location of visitors
  • Peak traffic periods
  • Visit duration
  • Page views per visit
  • Top-viewed pages or content
  • HTTP referrers
  • HTTP status
  • HTTP errors
  • Search terms or phrases used to find the site
  • Whether or not they are using a mobile device

Why do you need server logs?

Web Server logs provide an overview of all activity associated with the web server. For most organizations these logs are  the only way to understand how and when the server is used and by whom.

What can you do with a server log

Information within the web server log can be leveraged throughout the business to enhance performance and optimize operations. For example, this information can be used to:

  • Optimize limited IT resources, including staff
  • Establish dedicated logging levels and prioritize activity based on impact to the business or severity of the issue
  • Address and debug HTTP errors
  • Identify and fix broken links from external sources
  • Streamline the user journey based on typical navigation patterns
  • Adapt other business activity, such as sales, marketing or partner outreach
  • Identify security risks and issues, including the presence of bots, malicious code or spam

Additional web server log functions

Most organizations use web server logs to produce other file logs that serve a distinct purpose. These may include:

  1. Error logs
  2. Access logs
  3. Referrer logs

Error log: One of the most common server logs is an error log. As the name implies, an error log tracks all failed requests for the server. Web administrators can use this information to review standard error information and determine what aspects of their website need to be updated or changed.

Access log: An access log gathers data related to the files requested from the server. This log will reveal the number of users who accessed the server, how they were directed to the site, and their activity on the site. An access log helps organizations understand how the site is being used and the most popular or useful aspects of the site, which can in turn be used to improve or evolve the user journey, site navigation, or content.

Referrer log: A referrer log collects information about the URLs that direct users to your site. A referrer log is a vital aspect of most modern sales and marketing initiatives, in that most businesses want to enhance natural traffic. The referrer log is one way to determine which affiliate links or partners are responsible for drawing people into the site.

How can you monitor your web server logs?

While there are seemingly infinite insights to be gained from server logs, there are a few core challenges that prevent organizations from unlocking the value within log data.

Challenge #1: Volume

Server logs produce a significant amount of data that must be collected, stored and analyzed in order to produce timely and relevant insights.

Challenge #2: Integration

Web Server log data can be very useful to the organization, but additional value can be gleaned when integrated with other file logs, such as event logs, application logs, system logs, availability logs, and resource logs.

Challenge #3: Standardization

Unfortunately, not all log files follow a uniform format. Depending on the type of log, the data may be structured, semi-structured or unstructured. In order to perform effective log file analysis, the data requires a level of normalization to make it easily parsable.

Challenge #4: High IT Burden

When done manually, log management is incredibly time consuming and expensive. Digital log management tools help to automate some of these activities and alleviate the strain on IT professionals.

Log Everything, Answer Anything – For Free

Falcon LogScale Community Edition (previously Humio) offers a free modern log management platform for the cloud. Leverage streaming data ingestion to achieve instant visibility across distributed systems and prevent and resolve incidents.

Falcon LogScale Community Edition, available instantly at no cost, includes the following:

  • Ingest up to 16GB per day
  • 7-day retention
  • No credit card required
  • Ongoing access with no trial period
  • Index-free logging, real-time alerts and live dashboards
  • Access our marketplace and packages, including guides to build new packages
  • Learn and collaborate with an active community

Get Started Free