added
🔒 Rate Limiting for Investigations [AI Agent]
7 days ago by Siddarth Jain
To improve system performance and avoid overload during alert spikes, we’ve implemented rate limiting for investigations triggered by our AI agent.
What Happened
A client experienced a surge of 846 alerts within 30 minutes in a single Slack channel. Our AI agent attempted to analyze every alert, causing unnecessary system stress.
What’s New
We’ve introduced a rate-limiting rule:
➡️ The agent will now perform a maximum of 3 investigations per 15 minutes for identical alerts in a given channel.
Impact
- Reduces redundant investigations.
- Prevents performance degradation during alert storms.
- Ensures the AI agent focuses on meaningful anomalies without overwhelming the system.