Creating an Alert
Statful Alerts are notifications sent in response to a user-defined set of conditions being met.
To set up an Alert, go to Alerts menu >> ADD NEW ALERT.
This page implements the following steps:
Firstly, designate the name of the Alert.
As an option, you can also add a description, and decide if the Alert should default to Enabled or Disabled.Apply one or multiple conditions.
Statful adapts to having multiple triggers for the same metric, by applying different thresholds, and even other sources. Please follow through the analysis of the structure of an alert.Specify whom to notify.
Each Alert is associated with one or more notification profiles. You can set up an email, Slack, PagerDuty, and even customised webhooks here.
Structure of an Alert
It’s up to the user to define the conditions that will trigger a notification:
- Source: metric and tags (optional);
- Any/No data triggers
- Transformation triggers
- Transformation: mean, sum, count, max or min;
- Comparison: below or above data received;
- Value: the numeric value of the threshold;
- Duration: time window.
The Alert can be read as:
The
<transformation>value of the metric<source:metric>with tag/values<source:tags>has gone<condition><threshold>for<duration>.
Example: “The mean value of the metric system.gauge.memory with tag/values type:used, app:DB and environment:production has gone below 100 for 30 seconds“.
Considerations
- You may choose whether you want Any or All conditions to be verified.
- Conditions are evaluated as a group of one-dimensional logical operations.
- Conditions are based on a simplified query - group by or baseline queries are not supported.
Processing Metrics for Alerts
The conditions that trigger an Alert (activate or deactivate it) are based on the analysis of a rolling temporal window of raw metric data coming into Statful.
We provide below a detailed explanation of how the trigger process goes for Transformation and Any/No Data conditions.
Transformation triggers
- Received data points are stored in buckets per Alert.
- Data points are pulled from buckets and processed on a rolling window of real-time. At every X seconds, as defined by the user, the received data points go through a transformation. The result is then compared with the specified threshold.
- On each iteration, data points that fall outside of the defined time-window are removed from the bucket and not considered for processing.
- An iteration is only processed if there are at least two data points that represent an interval higher than the defined duration. This measure guarantees that a minimum data sample is received before processing.
Any/No Data triggers
- When receiving data points, each timestamp is stored in a bucket per Alert. The data point itself gets discarded.
- It follows the same approach of a rolling window of time like the transformation conditions, where the timestamp of the last data point is checked against the current timestamp. The comparison validates if any or no data is received as per the defined condition.
- An iteration starts processing as soon as the Alert is configured.