Most modern software development projects involve multiple developers, operations engineers, QA testers, and managers — and they all require access to the project infrastructure. However, enabling access to modify infrastructure leads to a higher risk of security breaches and legal disputes. Because of this, organizations rely on audit logs to address possible security concerns by recording precise actions in, and changes to, their infrastructure.
In this article, we will define audit logs and explore their use cases in infrastructure. We’ll look specifically at cloud and database activity. We’ll also consider legal compliance, and common challenges with data volumes, log retention periods, and correlations across different systems that require careful evaluation.
What are audit logs?
Audit logs are a collection of records of internal activity relating to an information system. To be useful, these records must contain the following information:
- Which component was affected by an action?
- Who performed this action?
- When was this action executed?
- What kind of action was executed?
Audit logs differ from application logs and system logs. Application logs record activity performed by external users, and system logs record activity performed by software, such as operating systems. However, audit logs are exclusively concerned with activities performed by internal users and services on system infrastructure.
Immutability is an important aspect of audit logs. No one should be allowed to modify audit log records, as that would diminish the integrity of the logs and render them useless.
Besides being convenient and useful, audit logs serve an important purpose for security and legal compliance, particularly when it comes to cloud infrastructure and databases.
Tracking changes to cloud infrastructure
Cloud-native projects often comprise numerous cloud services. The attractiveness of the public cloud hinges on its infrastructure elasticity, meaning the scale and configurations of these cloud services are very dynamic. Because of this, organizations that use cloud providers to deploy and manage their software need the capability to track each change made within their infrastructure, no matter how insignificant the change may appear. In addition, malicious activity on cloud infrastructure must be detectable to establish the exact actor, time, and nature of malicious actions.
Many cloud providers provide built-in audit logging services. For example:
Tracking database administration and system activity
Backend infrastructure depends on databases. Although developers may consider databases deployed in the cloud as a part of cloud infrastructure, database audit logs focus more deeply on actions performed on the database system. For example, database audit logs report on when clients connect and disconnect and the reasons for those actions. This information is important for establishing potential misuse and distortion of data.
Audit logs are also essential for tracking who makes alterations to a database schema, along with changes to schema components that affect the format, data structure, and record updates. Some databases are schema-less, so audit logs may be less significant in those cases. However, for relational databases, audit logs are crucial.
The most commonly used database systems provide audit logging plugins and extensions. For example:
Compliance and legal defense
Certain industries rigorously regulate audit logging levels. For example, a company in the Fintech or Medtech space must be prepared to implement audit logging at a comprehensive level, often requiring regular log analysis to confirm no malicious activity over a specified period. In addition to industry regulations, specific certifications impose high-level audit logging mandates. For example:
- HIPAA compliance certification for organizations in the healthcare industry.
- ISO 27001 certification for organizations in the IT industry.
Some companies must present information security compliance certifications to their partners and clients. As a result, audit logs provide proof of meeting regulation and certification requirements.
Audit logs are also beneficial when resolving legal disputes. Disputes arise more often than expected, especially in sensitive industries like finance and medicine, so audit logs help companies by serving as crucial evidence.
The challenges of audit logging
Although audit logs play a critical role in security and compliance, implementing them can be challenging. Let’s consider some potential issues related to storing audit logs.
Software projects are often sophisticated systems consisting of dynamic components. The sheer amount of data in audit logs can be enormous and very costly to store. For this reason, organizations must carefully consider which logs are most important to store, based on compliance requirements, security posture, and overall system functionality.
Log retention period
While some regulations and certifications specifically require audit logging, companies must retain these logs for prolonged periods in order to satisfy compliance demands. For example, one requirement for HIPAA compliance certification is that a company retain its audit logs for six years. For ISO 27001 certification, companies must store their audit logs for at least three years.
Retaining logs for long periods of time incurs financial costs and also requires resources for maintenance and management.
Correlating audit logs across different systems
Correlating, comparing, and analyzing audit logs across cloud and database vendors for different log formats and protocols can be strenuous. Dealing with large volumes of logs requires smoothing out differences, which requires significant time and effort. Additionally, discrepancies are common at the level of the details, with the possibility of causing compliance complications.
Organizations must be diligent to prevent security and legal incidents in their databases and their cloud infrastructure. Audit logs are critical for providing vital records and context, and they are often necessary to meet compliance requirements.
Managing audit logs may prove challenging, as large data volumes and lengthy retention periods can drive up costs and drain resources. However, neglecting audit logs can lead to non-compliance or bring significant operational and financial ramifications. In this sense, audit logs serve as a reliable insurance policy, as the true value of committing resources to implement audit logging is realized when unforeseen circumstances arise.