In part two of this series, we’ll dive deeper into Azure Monitor Logs and see its tools and services. We will then create a logs analytics workspace to store logs, run Kusto Query Language (KQL) queries, and look at how we can create alerts and Azure Monitor dashboards with query results.
Azure Monitor Logs
Azure Monitor Logs is one half of the Azure Monitor Platform. You can use Azure Monitor Logs to ingest and analyze logs from several resources and applications. Azure Monitor Logs include several tools.
Log Analytics Workspace
The Log Analytics Workspace (LAW) is a critical component for data aggregation. A LAW stores all logs in the Azure monitor logs platform, and you must create at least one LAW to use Azure Monitor Logs.
Once you create a LAW, you must configure log sources to send logs to the new workspace. There are no charges associated with creating a new workspace. However, the data is ingested as Analytics Logs, and you are charged for ingestion into the workspace. The pay-as-you-go pricing plan is the default option.
If you plan on ingesting over 100GB of data per day, then commitment-tier pricing might help to reduce costs. Organizations or users that ingest over 500GB of data daily into Azure Monitor Logs can use dedicated clusters to save even more on data ingestion costs.
Log Analytics Tool
The Azure log analytics tool allows users to write, edit, and run KQL queries on data stored in Log Analytics Workspaces. You can use the log analytics tool to write all kinds of KQL queries for data analytics. It also offers hundreds of pre-made queries for common use cases.
Log Analytics Agent
The Log Analytics Agent is software you install on an Azure virtual machine (VM) to integrate application logs with Azure Monitor Logs. When you deploy an application to Azure, you will likely deploy it to an Azure VM. It is possible to write application code that writes log files locally on an Azure VM. However, it is crucial to export logs from the VMs to use analytics and prevent accidental deletions. This is where the Log Analytics Agent comes into play.
You can use the Log Analytics Agent to send any logs generated by your application to a LAW. Keep in mind that the Log Analytics Agent cannot send metrics to Azure monitor metrics. The Log Analytics Agent is quite challenging to implement at scale as it requires a unique configuration for each VM.
Important Logs to Check
When using Azure, you should keep track of the following Platform Logs:
- Activity Logs: These are the most important types of logs in Azure. Monitoring these logs can help you analyze what or who made changes to your Azure services.
- Resource Logs: These logs can help you determine the health and availability of Azure Resources.
- Azure AD Logs: Monitoring these logs can help you analyze all changes in Azure Active Directory.
Operations in Azure Monitor Logs
In this section, we’ll provide a step-by-step guide to performing key operations in Azure Monitor Logs.
Data Aggregation and Storage
To store logs in Azure Monitor, first create a LAW. Navigate to the workspace. You’ll find the Log Analytics Workspaces option under Services. Alternatively, you can use the Azure portal search and type “log analytics workspace”.
In the screenshot below, you can see the default analytics workspace for this account. To create a new workspace for the demo, click on the + Create button.
Choose a subscription and resource group for this workspace and enter a workspace name. Click on the Review and + Create buttons to create the workspace.
Once you’ve created and deployed the workspace, click on the workspace and head to the Activity log page. All activity logs for the resource group
Azure_Monitoring are collected in this workspace. You can see the event type, name, and user who initiated this event. This information is essential to keep track of all user activity in the Azure portal.
You need to configure this workspace to connect to VMs in this resource group. Scroll down to find the Virtual machines page and click on the VM in the list to connect the VM to this workspace.
You have now successfully created a new Log Analytics Workspace and connected a VM to this workspace.
You can also see the Azure Activity logs for all management operations performed in this resource group. Before running any KQL queries on logs, head to the Tables section. This section displays the log tables that Azure has created for the account. You can run queries against these tables and perform complex operations (like JOIN) across tables.
To access the Log Analytics Tool, navigate to the Logs page in your workspace. Azure has a pre-made glossary of commands and queries that you can use to run common analytics commands.
However, when your default workspace is new and does not have much initial data, it is possible that some queries in this section do not work as expected. For this, we’ll check the availability of a VM using a KQL query.
Each VM connected to a workspace sends a metric (called a Heartbeat) once per minute. By using this metric, we can check the VM’s availability over the last day.
Heartbeat | where TimeGenerated > ago(1d) | summarize heartbeat_count = count() by bin(TimeGenerated, 30m), Computer, _ResourceId | extend alive=iff(heartbeat_count > 0, true, false) | sort by TimeGenerated asc // sort the results by time (ascending order)
This query shows the availability of the VM over the past day.
So far, we’ve seen how to analyze logs in Azure Monitor Logs. However, we also need to be able to visualize log analytics and present the analytics results. When running a query, you can choose to “pin” the query results to Azure Monitor, Grafana, or an Azure Workbook.
You can create alerts on log query results. In the example below, we’ll see how to create an alert that is triggered when our VM becomes unavailable.
First, navigate to the Alerts page and click the Create alert rule button.
Choose the Custom log search option.
Next, enter the following query:
Perf | where CounterName == "% Processor Time" and ObjectName == "Processor" and InstanceName == "_Total" | summarize AggregatedValue = avg(CounterValue) by bin(TimeGenerated, 1hr), Computer | render timechart
To configure the measurement and alerting logic, set the Measure field to
Table rows. In the drop-down menu for the Aggregation type, select
Count. Set the alert logic Operator to
Equal to and the Threshold value to
1. These settings allow this event to fire whenever the heartbeat command results in an output of 1.
You can set any granularity and evaluation frequency depending on your specific needs and requirements.
You can use the Data Export feature to configure data rules in the log analytics workspaces for exporting log tables to a storage account or event hubs.
Log Everything, Answer Anything – For Free
Falcon LogScale Community Edition (previously Humio) offers a free modern log management platform for the cloud. Leverage streaming data ingestion to achieve instant visibility across distributed systems and prevent and resolve incidents.
Falcon LogScale Community Edition, available instantly at no cost, includes the following:
- Ingest up to 16GB per day
- 7-day retention
- No credit card required
- Ongoing access with no trial period
- Index-free logging, real-time alerts and live dashboards
- Access our marketplace and packages, including guides to build new packages
- Learn and collaborate with an active community