Alert Middleware

Alert Middleware is a product built at Doctor Droid that significantly improved alert consumption by engineering teams by allowing them to setup rules for routing, enrichment and noise management. This document highlights how to get started and set it up in a few steps.

To get started with setting up Alert middleware, sign up on Doctor Droid platform.

Use the Universal Webhook URL -> https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest

📘

Authentication

The webhook call to send your alerts is authenticated. Find your auth Key in the API keys section in the bottom left in your Doctor Droid Portal.

Below are some guidelines on how to set it up in some of the popular alerting tools:


1. Cloudwatch


2. Datadog

  • Go to webhooks integration plugin in your datadog account and add the webhook.

  • Add a variable called $SOURCE and give it a static value 'DATADOG'

  • Add the webhook with the following body that includes drd_alert_source field that passes the $SOURCE as value.

  • Attach the webhook you just created as a recipient in your monitor by tagging it in the body


3. Grafana

  • Create a new contact point in your Grafana portal

  • Select the webhook contact point in the Notification section in your alert rule.


4. Sentry

  • Go to Settings -> Custom Integrations to create a new integration for the webhook URL.

  • Append the API Key as a query parameter in the URL so that your Webhook URL looks like https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest?token=<Api_Key>

  • Give Read permissions to Issue & Event. Save Changes.

  • In your alert rules, add an action to send a notification via an integration and select the webhook integration that you just created.


5. Custom Script

Add code within your alert generating script to pass it on to Doctor Droid. Find below code some sample code snippets:

import requests
import json

url = "https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest"

payload = json.dumps({
  "drd_alert_source": "Grafana"
})
headers = {
  'Authorization': 'Bearer <token>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

const axios = require('axios');
let data = JSON.stringify({
  "drd_alert_source": "Grafana"
});

let config = {
  method: 'post',
  maxBodyLength: Infinity,
  url: 'https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest',
  headers: { 
    'Authorization': 'Bearer <token>', 
    'Content-Type': 'application/json'
  },
  data : data
};

axios.request(config)
.then((response) => {
  console.log(JSON.stringify(response.data));
})
.catch((error) => {
  console.log(error);
});

curl --location 'https://playbooks.drdroid.io/alert-middleware/webhooks/alert/ingest' \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '{
    "drd_alert_source": "Grafana"
}'

Looking for integration guide for another tool? Let us know!