Analyzing Exchange Logs with Azure Log Analytics (Part 1)

If you would like to read the other parts in this article series please go to:

• Analyzing Exchange Logs with Azure Log Analytics (Part 2)
• Analyzing Exchange Logs with Azure Log Analytics (Part 3)
• Analyzing Exchange Logs with Azure Log Analytics (Part 4)

Introduction

Microsoft Operations Management Suite (OMS) is Microsoft’s new cloud-based management solution in Azure that provides Automation, VM Backup & Site Recovery, and Security & Compliance across an organization’s on-premise and public cloud environments. Log Analytics is one of the components of this OMS suite. It is a software-as-a-service (SaaS) solution that uses the power of Azure to collect, store and analyze log data generated by resources in an organization’s cloud and on-premise environments, such as Windows and/or Linux servers. It gives administrators real-time insights using integrated search and custom dashboards to readily analyze millions of records across all workloads and servers regardless of their physical location.

With Log Analytics we can gain a deeper insight into our environment by:

• Effortlessly collect, centrally store and analyze log data;
• Develop actionable insights using inbuilt intelligence;
• Investigate and fix incidents quicker;
• Gain consistent visibility across on-premises and cloud resources.

There is a free tier of Log Analytics that provides organizations with 500MB of daily upload and seven days of data retention. If more is needed, then the following is the pricing (in USD):

^*If a customer reaches its 500MB daily limit, data analyzing stops and resumes at the start of the next day (based on UTC).

Components & Architecture of Log Analytics

As the main components of Log Analytics are hosted in Azure, its deployment requirements are minimal. First we have Connected Sources, which are the computers and other resources that generate data collected by Log Analytics. This can include Windows, Linux or Azure servers for example.

Data Sources are the different types of data collected from each connected source, including events and performance data from Windows or Linux servers, IIS logs, and custom text logs. As we will see later in this article series, we configure each data source that we want to collect, and the configuration is automatically delivered to each connected source.

In order to collect data, we must install agents on Windows and Linux servers, but there is no additional agent required for computers that are already members of a connected SCOM management group. SCOM agents will continue to communicate with management servers which will forward their data to Log Analytics. Some solutions though will require agents to communicate directly with Log Analytics.

We can also import data using Solutions, which add functionality to Log Analytics. They primarily run in the cloud and provide analysis of data collected in the OMS repository. We can use Solutions to, for example, provide a summary of ongoing user activities in Office 365 (as we will see later in this article series), assess the risk and health of Active Directory, view the status of antivirus and antimalware across servers, identify missing system updates, and much more.

At the center of Log Analytics is the OMS Repository where all this collected data is saved into. Each data source might create a different record type with their own set of properties, but this data may still be analyzed together through queries to the repository. This allows us to use the same tools and methods to work with different kinds of data collected by different sources.

In the next diagram, we can see the overall architecture of Log Analytics:

All agents are registered with an enrollment key and a secure connection is established between the agent and the Log Analytics service using certificate-based authentication and SSL over port 443. Similarly, with Operations Manager we register an account with Log Analytics and a secure HTTPS connection is established between the Operations Manager management server. If Operations Manager is unable to communicate to the service for any reason, the collected data is stored in a temporary cache and the management server tries to resend the data every 8 minutes for 2 hours.

Signing up for Log Analytics

Before signing up for Log Analytics, we need to understand the concept of an OMS workspace. You can think of the workspace as a unique OMS environment with its own data repository, data sources, and solutions. You can create multiple workspaces in your subscription to support multiple environments such as production and test environments, or even different teams in your organization for example.

Getting started to use Log Analytics is quick an easy! We have two options when choosing how to create an OMS workspace:

• Using the Microsoft Operations Management Suite website;
• Using a Microsoft Azure subscription.

As already mentioned, we can create a free OMS workspace using the OMS website, or we can use an existing Azure subscription (if we have one). Both workspaces are equivalent, with the exception that with the free one, we can only send 500 MB of data daily to the OMS service. If we use an Azure subscription, we can also use that subscription to access other Azure services.

Before looking at how to create our first OMS workspace for Log Analytics, let us have a quick look at the prerequisites and deployment considerations:

• We need a paid Azure subscription to fully use Log Analytics. If we do not have an Azure subscription, we create a free account that will let us access any Azure service. Alternatively, we can create a free OMS account. We will look at both options shortly;
• Each Windows computer that we want to gather data from must be running Windows Server 2008 SP1, or above, or Windows 7 SP1 or above;
• The agents on each server need to be able to connect with the OMS service. When using a firewall to restrict access to the Internet, make sure it allows agents to access *.ods.opinsights.azure.com, *.oms.opinsights.azure.com and *.blob.core.windows.net, all on port 443;
• An OMS Log Analytics Forwarder (Gateway) server can be used to forward traffic from servers without Internet access to OMS (this is outside the scope of this article series);
• If using Operations Manager, Log Analytics supports Operations Manager 2012 SP1 UR6 and above, as well as Operations Manager 2012 R2 UR2 and above;

When data usage and performance are a concern, it is recommended to deploy agents individually and test the performance impact and data usage before adding additional agents. It is better to start with minimal collection until data usage and performance impact has been identified.

Subscribing using the Operations Management Suite website

If you do not have an Azure subscription, or you simply want to quickly test Log Analyzer without “touching” your production Azure for example, you can trial Log Analyzer for free by following these steps:

1. Navigate to the Operations Management Suite website and click on Create a free account:

2. Next click on Get started >:

3. Sign in with your Microsoft account such as Outlook.com, or with an organizational account;
4. Provide a unique workspace name, specify an email address and the region where you want to have your data stored and click on CREATE:

5. You are now ready to get started with the Operations Management Suite portal and Log Analytics! Through the dashboard we can quickly see our data usage, how many sources we have configured, our data plan, and more:

The Get stated wizard helps us set up and configure Log Analytics, but let us first check how to sign up using an existing Azure subscription before doing so.

What’s next?

In this first part of this article series we introduced Log Analytics and looked at how to sign up using the Operations Management Suite website. In the next part, we will sign up using the Azure portal and see how to connect our Exchange server(s) to Log Analytics.

If you would like to read the other parts in this article series please go to:

• Analyzing Exchange Logs with Azure Log Analytics (Part 2)
• Analyzing Exchange Logs with Azure Log Analytics (Part 3)
• Analyzing Exchange Logs with Azure Log Analytics (Part 4)