Azure is Microsoft’s cloud computing platform, offering various features like storage, computing, networking, Internet of Things (IoT), analytics, and more. As a comprehensive platform, Azure is home to thousands of applications.
Logging is a crucial component of all applications—both in the cloud and on-premise—helping with troubleshooting and implementing security of compliance standards. With today’s organizations slowly migrating applications to cloud platforms, the need for logging has increased further. Now, you have to monitor both application logs and cloud platform logs to ensure optimal application performance.
In this first part, we will cover platform logs in Azure—their types, importance, and possible use cases. We will also discuss Azure Monitor Logs and understand the importance and use cases of logs in that context.
Shared Responsibility and the Cruciality of Logging
Azure has a shared responsibility model. This means that even while you are using Azure’s resources, you still have a responsibility to watch Azure resource usage as well as maintain your application’s performance and security.
Although Azure maintains responsibility for its resources’ basic health, it’s still possible for Azure resources to go offline. You also need to consider scenarios where human error could impact application health and security. Developers and cloud architects can misconfigure a resource, making an application vulnerable to attacks. A new teammate could accidentally delete an important resource.
Without logs, engineering teams are hamstrung as they attempt to analyze and trace problems to fix issues.
Azure Monitor
The Azure Monitor is Azure’s native service for all logging and monitoring needs. It is a powerful tool that helps you set up self-healing and scalable application architectures. The tool provides a lot of flexibility and opportunities to automate many Azure processes.
Azure Monitor consists of two sub-services:
- Azure Monitor Metrics: used to analyze resource metrics and automate actions if the metrics exceed certain thresholds.
- Azure Monitor Logs: a comprehensive platform that allows you to aggregate and analyze logs.
We will focus on Azure Monitor logs in this article.
Within Microsoft Azure, there are two types of logs:
- Platform logs: the native logs of the Azure platform, made up of activity logs and resource logs.
- Application logs: the logs developers send from applications to Azure Monitor Logs.
Let’s cover each of these in detail.
Platform Logs in Microsoft Azure
With cloud adoption comes the challenge of managing multiple resources, controlling costs, and ensuring that best practices and security controls are implemented. This is where platform logs come in handy. Platform logs provide detailed auditing information for the Azure platform and its resources.
Activity logs
Activity logs contain information on all the management operations of Azure resources. These logs are essential to track all user activity in the Azure platform and can help you troubleshoot or identify changes in the Azure platform. Activity logs contain information about when resources are modified, launched, or terminated. For example, you can use Activity Logs to answer questions like:
- Who created this Virtual Machine (VM)?
- Who deleted the MySQL database?
All activity logs are retained for 90 days, after which time they are deleted. During the 90-day period, there is no charge for these logs. Azure users can also use Activity Logs Insights to view all resource management operations in a subscription. The dashboard also provides data about which users or services performed activities in the subscription.
Resource logs
Resource logs contain information about all operations performed within an Azure resource. These logs track all activity in the data plane of Azure resources, such as requests to a VM or access attempts to secrets from a key vault.
Resource logs are not collected by default. To collect resource logs, you must configure diagnostic settings for each Azure resource to send its logs to different destinations.
By using diagnostic settings, you can send both activity and resource Logs to different destinations:
- Log Analytics Workspace: Runs Kusto query language (KQL) queries against the logs, correlates logs from multiple resources, and performs analytics. You can use the query results to set up alerts.
- Azure Event Hubs: Real-time data ingestion service that can be used to pipe the logs out of the Azure Platform to third-party services such as Splunk or Falcon LogScale.
- Azure Blob Storage: Used for inexpensive archival storage of logs.
Azure Active Directory (AD) logs
Azure Active Directory (AD) is the identity provider for the Microsoft Azure platform. Azure AD logs contain sign-in logs and audit logs.
Sign-in logs contain details about logins to applications using Azure AD. This includes information for the Azure portal logins and sign-in activities to other services using Azure AD. Audit Logs contain logs related to managing identity-related services. You can use audit logs to determine which user performed exactly what actions in Azure AD. For example, you can determine who gave admin access to a user, or who deleted a user from AD.
The Azure Monitor Logs Platform
Azure Monitor Logs is a platform to aggregate, organize, analyze, and use logs for all kinds of alerts, visualization, analytics, and more. All logs in the Azure platform, like the custom application logs or platform logs, can be moved to Azure Monitor Logs.
Before we dive into the services of Azure Monitor Logs, it is essential to understand the following types of logs in the Azure Monitor Logs platform.
Analytics logs
All logs in the Azure Monitor Logs platform are stored as analytics logs by default. The default retention period of these logs is 30 days (or 90 days for certain logs), but this can be extended up to two years.
For analytics logs, you have the full capabilities of the KQL to perform comprehensive analytics operations. There is also no limit or cost for KQL queries. You only pay for data ingestion costs and retention costs (if you extend the retention period beyond 30 days).
Basic Logs
While analytics logs are ideal for most use cases, they come with a high cost. There can be scenarios where you do not need the full capabilities of the KQL and simply use the logs for troubleshooting and testing applications. You don’t need to perform complex analytics or generate alerts for these requirements. This is where the basic logs will be a better option.
Basic logs are a less expensive category of logs in Azure Monitor Logs. However, you need to explicitly convert analytics logs to basic logs. The default (and only) retention period of basic logs is eight days, with an archival period of 22 days (in case you change this log back to analytics logs).
As mentioned earlier, these logs do not have the full set of KQL queries, and you can only perform limited operations. You only pay a small ingestion cost as compared to analytics logs. However, unlike analytics logs, you must pay for KQL queries that are run on basic logs.
Archive Logs
Both analytics and basic logs are interactive, so you can run queries against those logs. However, what if you only store logs for compliance and audit reasons and don’t need to run queries against the data? While keeping logs in blob storage or other archival options is possible, importing logs from archival storage back to analytics logs (to perform useful operations or queries) is very complex.
Archive logs provide a cost-effective archival option for logs in Azure Monitor Logs. You can configure the retention period of these logs for up to seven years, and you only pay for the retention. However, there is no way to perform any analytics operations against these logs.
If you want to perform analytics on archive logs, you need to move these logs back to analytics logs by using SEARCHand RESTORE operations. For both of these operations, you must pay ingestion costs and data scanning costs during the operation. These operations create entirely new tables in analytics logs with suffixes _SRCHand _RSTR.
It’s important to note that the data in these tables is retained in analytics logs for as long as the archive logs exist. Once the necessary analytics are complete, you should delete these tables to avoid paying any unnecessary costs.
Conclusion
Logging is an essential component of all applications. It can help in application development and ensure application health and availability. Apart from application logs, platform logs are essential for all organizations that use the Microsoft Azure platform. These logs can help assess all platform and resource management operations.
The Azure Monitor Logs platform is a one-stop shop for all logging needs in the Azure Platform. It provides cost-effective and efficient log storage options and can help organizations set up efficient architectures in the Azure platform to self-heal applications and automate application management.
Log your data with CrowdStrike Falcon Next-Gen SIEM
Elevate your cybersecurity with the CrowdStrike Falcon® platform, the premier AI-native platform for SIEM and log management. Experience security logging at a petabyte scale, choosing between cloud-native or self-hosted deployment options. Log your data with a powerful, index-free architecture, without bottlenecks, allowing threat hunting with over 1 PB of data ingestion per day. Ensure real-time search capabilities to outpace adversaries, achieving sub-second latency for complex queries. Benefit from 360-degree visibility, consolidating data to break down silos and enabling security, IT, and DevOps teams to hunt threats, monitor performance, and ensure compliance seamlessly across 3 billion events in less than 1 second.
