Alert Middleware
Alert Middleware is a product built at Doctor Droid that significantly improved alert consumption by engineering teams by allowing them to setup rules for routing, enrichment and noise management. This document highlights how to get started and set it up in a few steps.
To get started with setting up Alert middleware, sign up on Doctor Droid platform.
Use the Universal Webhook URL -> https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest
Authentication
The webhook call to send your alerts is authenticated. Find your auth Key in the API keys section in the bottom left in your Doctor Droid Portal.
Below are some guidelines on how to set it up in some of the popular alerting tools:
1. Cloudwatch
- Go to the SNS topic where your cloudwatch alarms is sending messages
- Create a new subscription for that topic with protocol -> HTTPS and endpoint -> https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest?token=<api_key>
2. Datadog
-
Go to webhooks integration plugin in your datadog account and add the webhook.
-
Add a variable called $SOURCE and give it a static value 'DATADOG'
-
Add the webhook with the following body that includes drd_alert_source field that passes the $SOURCE as value.
-
Attach the webhook you just created as a recipient in your monitor by tagging it in the body
3. Grafana
-
Create a new contact point in your Grafana portal
-
Select the webhook contact point in the Notification section in your alert rule.
4. Sentry
-
Go to Settings -> Custom Integrations to create a new integration for the webhook URL.
-
Append the API Key as a query parameter in the URL so that your Webhook URL looks like https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest?token=<Api_Key>
-
Give Read permissions to Issue & Event. Save Changes.
-
In your alert rules, add an action to send a notification via an integration and select the webhook integration that you just created.
5. Custom Script
Add code within your alert generating script to pass it on to Doctor Droid. Find below code some sample code snippets:
import requests
import json
url = "https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest"
payload = json.dumps({
"drd_alert_source": "Grafana"
})
headers = {
'Authorization': 'Bearer <token>',
'Content-Type': 'application/json'
}
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
const axios = require('axios');
let data = JSON.stringify({
"drd_alert_source": "Grafana"
});
let config = {
method: 'post',
maxBodyLength: Infinity,
url: 'https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest',
headers: {
'Authorization': 'Bearer <token>',
'Content-Type': 'application/json'
},
data : data
};
axios.request(config)
.then((response) => {
console.log(JSON.stringify(response.data));
})
.catch((error) => {
console.log(error);
});
curl --location 'https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest' \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '{
"drd_alert_source": "Grafana"
}'
Looking for integration guide for another tool? Let us know!
Updated 18 days ago