Post-Deployment Monitoring Playbook
Setup continuous monitoring for a few hours post each deployment for critical metrics, log queries or DB queries which will indicate if a rollback is needed
Go through this document first to understand how to setup slack app, integrate it within your setup and the basic structure of how the workflows are structured. While that document helps in enriching a Slack alert with your executed playbook's summary, this document will help in setting up a continuous monitor post your deployments.
How to create a continuous monitoring workflow?
-
Select 'API' as the trigger type.
-
Generate cURL for the API signature you'll be needed to call to trigger the workflow. It has the specific workflow ID in the request body.
-
Choose your schedule as 'Run continuously' and select for how long and how frequent you want the playbook to be executed.
-
Choose 'Send a Slack Message' as your notification and select the channel where you want to publish the executed monitor's summary.
Integrate the API request in your deployment tool
This section will explain how this API can be called from some popular deployment tools.
In case you want to call this from your code, just copy-paste the cURL here and get working code in a language of your choice.
Jenkins
The best way to call the workflow trigger API from your Jenkins build is to create a separate job which executes the curl command for the API. You can then add execution of this job as a post-build action from your job when deployment completes.
Github Actions
You can add the trigger API as the last step in your Github Deployment action. You can use HTTP Request Action to achieve this.
ArgoCD
You can create a job with a postSync hook to execute a cURL command. Here it shows in ArgoCD Docs on how it can be used to trigger a Slack notification.
CircleCI
Use webhooks in CircleCI to setup this workflow trigger API call post a deployment is done. Read more in their documentation.
GitLab
Use webhook in GitLab to setup this workflow trigger API call post a deployment is done. Read more in their documentation.
Updated 4 months ago