As part of your security strategy in the Azure cloud, gathering your log activity in a secure place and using it efficiently should be the goal of any organization/security department. We already started planning our organization toward that goal in the first article of this series, where we configured the log analytics workspaces and additional infrastructure to support it using the Azure Portal. This article, the second in our series, will configure the Azure resources to send their log information to our newly created log analytics. We will start with a simple Kusto Query Language (KQL) query to understand how the logs and query editor can be used.
Configuring Azure resources diagnostic settings
We have built a log analytics infrastructure to support our environment, but our next step is to configure subscriptions and resources to send the diagnostic settings to it.
One of the first things that we need to look at from the security perspective is to ensure that all activities for all resources in any given subscription are forward to our log analytics workspaces.
The activity log contains all write operations for all resources in any given subscription. The records are kept for 90 days, and that information is stored in the backplane for all customers. The data can be retrieved using PowerShell, Azure CLI, REST API, and Azure Portal. We want to add that same information to log analytics for further analysis.
Logged in the Azure Portal, click on Monitor, and then Activity Log (Item 1). All activity in the subscription will be listed. Click on Diagnostic settings (Item 2).
All diagnostic settings at the subscription level will be listed (keep in mind that we can have more than one). Click on Add Diagnostic Settings on the new page, define a name (Item 1), select which categories we will be collecting and where we will send (Item 3), which is our log analytics workspace we created in the first article in this series.
The diagnostic settings is available in most of the Azure resources. We will find them under the Monitoring section when looking at a resource (Item 1).
The configuration is simple. A list of any existing diagnostic settings will be shown (Item 2). At a single glance, we can see who is receiving the logs (storage account, event hub, log analytics workspace, or a partner solution) and edit them.
We can add a diagnostic setting (Item 3), and before adding a new one, we can check which categories are available to be logged (Item 4) — it varies depending on the resource, and they do change! Microsoft may add more categories when new telemetry is being provided in any given Azure resource.
When adding a new diagnostic setting, we need to define a name, categories, and destination, as depicted below.
Getting started with Kusto Query Language
When you see Kusto Query Language for the first time, you will be torn if it is a SQL query or a PowerShell statement. I think it is a good mix between the two.
A Kusto Query Language is a read-only request to process and return results from a log analytics workspace. Remember that you can always add data to log analytics workspaces, but you can delete records, which is vital when looking at security.
KQL is simple to use, and it is written in plain text. The query has statements, and after the process, it returns a table with the data processed (one or more columns and rows).
Where can we see the KQL in action? That is easy. In the log analytics resource, click on Logs. A whole new page with tons of options will be listed. To get more real estate, I recommend tucking in the side menu (Item 1). Under Log Management (Item 2), we will see all “tables” created by sources sending data to our log analytics.
If we double-click on any of the tables, it will start the Kusto Query Language (Item 3) by placing the table’s name in the query editor.
To see the entire content of that specific area, click on Run (Item 4), and the results of our first query will be displayed (Item 5).
You may have noticed that the previous results had several lines. As depicted in the image below, if we expand any given line/record, we will have all the records (columns) and their data.
The editor is simple but powerful, and we can click on the … located on the left side of each value and include or exclude that value from our current query.
The result of that operation would be a new KQL on the StorageBlogLogs table, where the process to list containers was performed. As we can see, the results were down to one.
Azure log activity: Basic setup complete, thanks to Kusto Query Language
We have the basic setup complete. We built the log analytics infrastructure, configured the sources to send their diagnostic information, and ran our first query to retrieve the collected data. We will need to go deeper in creating Kusto Query Language queries and log analytics workspaces designs based on a couple of scenarios to instigate discussion and help you understand and apply the best strategy to your organization. And that’s all coming up in the next article in this series.
Featured image: Shutterstock
