Hi friends, today I want to share with you my guide on how to integrate the alerts from Grafana into many platforms. That way, if something goes wrong with your validator, you can always be the first to know about it.
You need Grafana installed on the server (it’s a good service for monitoring the health of a node). If there are some questions about it, I will prepare guideline later.
We will use Pager duty and e-mail to accept the Alert. You should know that you can also use telegram bots or other programs to receieve messages. How to sign up in pager duty.
The guide consists of several parts.
Part 1. Creating integration key in pager duty
Part 2. Preparing the Alert itself in Grafana.
1.1 In your Pager duty app go to - services
1.2. Push “New service”
1.3 Name it
1.4 Push “Generate new escalation policy”
1.5 Select Events API v2
1.6 Copy your Integration Key
We created the integration key that we will need to connect our monitoring system with our messenger.
2.1 Go to your Grafana. Alerting - Contact Points.
2.2 Push New contact point
2.2.1 Write any name
2.2.2 Select the service you want to be notified to from the list (Pager Duty or E-mail as you wish)
2.2.3 Insert the Integration Key you created in section 1.
2.2.4 You can test it by clicking on the “Test” button. A test message will come.
2.2.5 Save the contact point.
2.3 Remember you can just add your e-mail and you won’t need pager duty or telegram.
2.4 Next, we need to add our Contact point to the default Notification policies
2.5 Now we have to go to our dashboard
2.6 Choose the metric which defines the validator work (e.g. CPU load in % or fails in your proposals)
At the same time you can add alerts for other parameters yourself, if necessary. I will show you some metrics I’ve added for my validator.
It’s simple to find metric in your dashboard (1) and then push arrow (2) and edit
2.7 Then switch to alert at the bottom and create a new rule
2.8 Add Expression (button +Add Expression)
Set the rule: When CPU load is above 70%, then an alert will be triggered and I will receive an email notification (or Pager Duty, what you decide) Next, save.
2.9 Then we select the folder where the rule will be stored (GrafanaCloud)
The name of the group can be any. You can also specify the text that will be in the message. Edit field Description. Be sure to save.
I will also give you some screens how you can add alert on:
2.10 Failed proposal (will alert when there is at least one failed proposal)
2.11 Proposal count
Hope you found something useful for you! Feel free to ask any question