How e-mail/SMS alert internals work

One of the key features of our server monitoring service, Server Density, is the e-mail and SMS alerting. This has a number of components to provide quite a flexible alerting system.

Alerts

Free users can add 2 alerts but paid users can add as many as they wish. These are configured on a check level e.g. CPU load. Thresholds are set so that the alert will be triggered at a certain value for that check, e.g. CPU load over 2.0. There are many options for alerts, which can also be created multiple times for as many users as there are in the system:

• Trigger threshold comparison so you can be alerted depending on the value e.g. whether it is below x, equal to or above y, etc.
• Notification delay so the alert condition must exist for a period of time before any notification will be sent.
• Frequency so notifications can be sent multiple times at defined intervals e.g. in case you don’t get woken up by the first SMS!
• And a further, final notification when the alert condition stops existing i.e. the alert is fixed.

The alert threshold is compared to the real-time data received for each server. Once the threshold is met, then a triggered alert is created.

Triggered alerts

There will only ever be 1 active (i.e. non-fixed) triggered alert for each alert, however there many be many historical triggered alerts that have been previously triggered and subsequently fixed. Once triggered, it is this triggered alert that handles the notifications, either immediately or delayed based on the alert configuration explained above. A notification is generated at the frequency specified in the parent alert configuration, which can be no more frequent than every minute.

The triggered alert is constantly being compared with the trigger threshold and real time data to make it fixed if the data shows the alert condition has been resolved.

It is these triggered alerts that appear in the “Last 5 triggered alerts” section on the dashboard.

Notifications

These are generated based on the existence of an active (i.e. non-fixed) triggered alert and can be e-mail or SMS. Where the alert is configured for both then both will be generated separately. If the alert is configured to send out multiple notifications then notifications will exist on a one to many relationship with the triggered alert.

Pausing

When you pause an alert, it pauses the very top level so that no further triggered alerts and therefore no further notifications will be generated.

Although this system seems a little complicated, the implementation is quite straightforward and means there is a lot of flexibility so that alerts can be set up exactly to your requirements.