A company’s messaging infrastructure is typically one of its most critical systems. Every administrator knows the importance of continuously monitoring Exchange. This is not only to prevent downtime and quickly fix problems, but also to be aware of the health of the infrastructure, to help identify potential problems, and spot performance degradations before they turn into actual problems and cause downtime.
Monitoring solutions such as Microsoft’s System Center Operations Manager (SCOM), SolarWinds, Nagios, and MailScape are some examples of monitoring tools for Exchange. However, not every organization has the means and capabilities to acquire and use them. On top of that, many monitoring solutions alert administrators exclusively by email, which can be problematic if the mail flow itself is affected, or through a dashboard that administrators might not have access to outside business hours.
Throughout my career, I have developed several scripts that run continuously every x minutes, and if they detect something is wrong with Exchange, they send an email alert. There are, however, two issues with this approach:
- If mail flow is affected, as already mentioned, then administrators will not receive the email alert.
- When out of the office, not everyone is constantly checking their emails, meaning it might be a while until an administrator is made aware something is wrong.
For these reasons, I researched the best methods to send notifications to my phone when something was wrong with Exchange. I tried Mobile Device Management (MDM) solution we were using at the time, but it seemed there was no way to interact with its API in order to send alerts programmatically. An alternative I also considered was WhatsApp. Although it is possible to use PowerShell to send WhatsApp notifications, it has a few drawbacks, such as the need for the recipients to register with a service like WhatsMate WA Gateway and the fact that I would need to maintain a list of users who would like to receive the notifications on my scripts. Users wouldn’t be able to opt-in or opt-out from receiving notifications themselves.
Slack to the rescue
Slack is a messaging app for teams that is even used by the NASA team behind the Mars Curiosity Rover. Slack is becoming extremely popular, and I am starting to see why. Just have a quick look at the video below and judge for yourself. I’m not going to fully explore in this article how Slack works, but from my limited experience with it, it seems really good.
[tg_youtube video_id=”9RJZMSsH7-g”]
We sign up for Slack by creating a team in the format of team_name.slack.com. Obviously, we use the company’s name as our Slack team, such as company.slack.com. This team will then have one or more “chat groups” called Channels to organize team conversations. For example, we can create a channel for a project or a particular topic. To test Slack I created a channel called Exchange:
Once the channel is set up and users either join it or we invite them, we can start a conversation:
The purpose of using Slack is mainly to send messages programmatically, so let’s see how we can do this.
To send data into Slack in real time, webhooks are used. Webhooks are a way to post messages from external sources into Slack. They use normal HTTP requests with a JSON payload that includes the message text, attachments, and other features. But don’t worry, we will not need to learn JSON!
The first step is to get a test OAuth Token for our testing. This token will give us, and anyone who has it, full access to our private data and that of our entire team (all channels), so keeping it a secret is extremely important! To get one, sign in into your Slack team (using a browser, not the app), and then navigate to the Tokens for Testing and Development page.
Once we have our token, we can always reissue it, for example if someone manages to get hold of it:
We will not be building any JSON payloads ourselves. Instead, we will rely on PowerShell modules that other people have already developed. After all, why reinvent the wheel? From the few that I have tested, the one from Warren Frame, aka RamblingCookieMonster, seems to be the best. So let’s try using it.
Navigate to his PSSlack page on GitHub, download the PowerShell module and install/import it into your test workstation. Once this is done, you should have access to the PSSlack’s cmdlets:
Get-Command -Module PSSlack
Get-Help Send-SlackMessage -Full
…and so on.
Send Slack messages using PowerShell
Now that we have our token and PowerShell module, we are set to go, so let’s start testing! We start by saving our token into a variable. This is not really necessary, but I’m doing it to keep the code easier to read. Next, we use the Send-SlackMessage cmdlet to send a message into our exchange channel:
$token = “xoxp-50080309206-(...)”
Send-SlackMessage -Token $token -Channel “#exchange” -Parse Full -Text “High mail queues on EXCH1\MDB01: 317 emails”
Once we run the command, we get a confirmation that the message was sent (ok: True). If we go back to our channel in our PC, we can see our first Slack message using PowerShell!
Obviously we are not actually checking the mail queues on server EXCH1 as of yet, but we will get to that shortly.
Exchange is down alert message
The next example shows an alert we could send from a script that detected that Exchange was down. It includes an attachment to produce a rich-formatted message.
Add-Type –Assembly “System.Drawing”
$slackAttach = New-SlackMessageAttachment -Color $([System.Drawing.Color]::Red) -Title "Exchange might be down!" -TitleLink https://dashboard.company.com -Text "Please action immediately!" -Pretext "System Unresponsive" -FallBack "Exchange might be down!"
$slackAttach | New-SlackMessage -Channel "#exchange" -IconEmoji :warning: -AsUser -Username "Exchange Admin" | Send-SlackMessage -Token $Token
OK, it’s starting to look better! You can pipe New-SlackMessageAttachment to New-SlackMessage, but I separated it to make it easier to read.
Notice that the “Exchange might be down!” text is also a link to an in-house dashboard. This could be a SCOM or Splunk dashboard, for example. To make it stand out we used the color red and a warning icon. And yes, there are loads of different emojis we can use 🙂
Get live data from Exchange
In our next example, we start to get data live from Exchange and report on it. This simple example just sends a message containing certain details about a mailbox database named MDB01:
$exchDB = Get-MailboxDatabase “MDB01” -Status | Select Name, LastFullBackup, DatabaseSize, Mounted, ServerName
$slackFields = [PSCustomObject] @{
Name = $exchDB.Name
BCK = $exchDB.LastFullBackup
Size = $exchDB.DatabaseSize.Split("(")[0]
Mounted = $exchDB.Mounted
Server = $exchDB.ServerName
Users = (Get-Mailbox -Database $exchDB.Name -ResultSize Unlimited).Count
} | New-SlackField -Short
Add-Type -Assembly "System.Drawing"
$slackAttach = New-SlackMessageAttachment -Color $([System.Drawing.Color]::Green) -Fields $slackFields -Fallback "DB Info"
$slackAttach | New-SlackMessage -Channel "#General" -IconEmoji :information_source: -AsUser -Username "Exchange Admin" | Send-SlackMessage -Token $Token
The end result looks as follows:
Each message can contain up to 100 attachments, so we could send a separate message with each database’s details or, preferably, create one attachment for each database, and then send a single message with all those attachments. Here’s an example of one message with two attachments:
Exchange transport queue monitoring
Let’s look at how we could monitor Exchange’s transport queues and issue an alert if the total number of queued emails goes beyond a certain limit. To achieve this, we get the queues of all the servers, exclude any Shadow Redundancy emails, and count the total number of emails across all queues. Then, if that number is above our limit, we send an alert. Obviously, this is a basic script just for demonstration purposes. In a production environment a few tweaks would likely be required, such as the threshold limit, any queues or servers to include/exclude, and so on.
While ($True) {
[Int] $msgCount = 0
Get-TransportService | Get-Queue | Where {$_.MessageCount -gt 0 -and $_.DeliveryType -notlike "Shadow*"} | ForEach {$msgCount += $_.MessageCount}
If ($msgCount -gt 20) {
$slackAttach = New-SlackMessageAttachment -Color $([System.Drawing.Color]::Red) -Title "High Mail Queues!" -Text "Total queued emails: $msgCount." -FallBack "High Mail Queues!"
$slackAttach | New-SlackMessage -Channel "#exchange" -IconEmoji :warning: -AsUser -Username "Exchange Admin" | Send-SlackMessage -Token $Token
}
Start-Sleep –Seconds 1800
}
Here is the result when the script triggers an alert:
Going mobile with Slack messages
So far, we have only seen what these alerts look like on the browser version of Slack, but the main purpose of using Slack in the first place is so that administrators can receive these alerts on their phones. As such, let’s see how this last alert looks on the Slack Mobile App.
When the message is first received, and assuming the phone is locked, a notification is displayed in the lock screen with the title of our alert:
If we open the app and go to our exchange channel, we can see this last alert plus all the previous ones we generated:
If we send a simpler message, or simply put the alert details in the Title field, we can see what is going on from the notification itself without having to open the app:
Search and Exchange remote monitoring
Using this PowerShell module, we can also search our channel for specific messages using the Find-SlackMessage cmdlet:
Find-SlackMessage –Token “our_token” –Query “what_to_search”
In the next example, I am searching for any messages that contain the word transaction. I then sort the output by date and select the first message returned, so we get the latest message containing the word transaction sent to any channel:
Find-SlackMessage –Token “our_token” –Query “transaction” -SortBy TimeStamp | Select -First 1 | Format-List
Please note that the timestamp returned is in UTC time while the time displayed on my phone or in the browser is not.
Using this search method, we can even develop a script that every x minutes scans a particular channel (or all channels for that matter) and reads what users post. If it detects an Exchange cmdlet, the script will run it and post the result in the same channel. This is useful for quick checks/troubleshooting when administrators are out and about and don’t have access to their Exchange environment. Obviously, it would be crucial to limit what cmdlets run (no Set-* cmdlets) or limit the script to only execute cmdlets posted by a particular user, for example.
In the following screenshot we can see user Nuno posting a cmdlet to check his mailbox size and the script replying to Nuno a minute later with the output of that cmdlet:
Using the search method we discussed earlier, we can see who posted the cmdlet (username), the cmdlet itself (text), and even any previous or future messages (previous and next). Here we can see that the next message after this one is actually the response from the script:
Slack’s versatility is making it a very popular messaging app for many organizations. In this article, we saw how we can use it to monitor Exchange and send alerts to administrators’ mobile devices. With all the features available in Slack, however, we can do even greater things. Just use your imagination!
