If you are using Splunk as your log management tool, creating Splunk alerts is a great way to monitor your apps and services for errors. The alerts can trigger different kind of actions. For instance sending a POST to a webhook or be sent as an e-mail. An easy way to monitor the alerts is to create a Microsoft Teams channel for your alerts. Each Teams channel has an e-mail address that can receive data that will be displayed in the channel.
1. Setup Microsoft Teams
Create a new channel in the Team you want to receive your alerts. From the channel properties, select Get Email Address, and copy the address. In advanced settings for the e-mail, select that anyone can send to the address.
The email feature needs to be turned on by your IT admin if it’s not available. More information on sending emails to Teams here
2. Create a Splunk Alert
Run a new search in Splunk with the desired log level and index. Example:
index=“prod” source=“MyIntegration” LogLevel=Error
Click Save As, and select Alert.
For trigger action, select “Send Email”.
Use the e-mail for your Teams channel as e-mail in the To field.
Enter a title and an optional description.
Select Scheduled as alert type and select run every hour
Use Shared in app as permission
For trigger conditions use Number of results is greater than 0.
Select Once for trigger.
Priority: Normal
Subject: Splunk Alert: $name$
Message: The alert condition for ‘$name$’ was triggered.
For the includes, select the following: Link to alert, Link to Results, Inline (Table), Allow empty attachment. Type: HTML & Plain text.
Finally, save the new alert.
From now on when an error is logged for your service, an alert should be created in Splunk and displayed in the Teams channel (for the scheduled alert time).