Notify Slack of process state changes using Supervisor's EventListener

Introduction

Do you run a supervisor and rarely have a process down? I think there is a way to monitor it externally, but I tried to notify Slack of the state change of the process using the EventListener of Supervisor, which I had been interested in for a long time.

What i did

Creating a configuration file

Create a configuration file as shown below.

/etc/supervisord.d/process_state_event_listener.conf

[eventlistener:process_state_event_listener]
command=python /path/to/process_state_event_listener.py
events=PROCESS_STATE
redirect_stderr=false
stdout_logfile=/var/log/supervisor/event_listener_stdout.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=5
stderr_logfile=/var/log/supervisor/event_listener_stderr.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=5
environment=SLACK_WEB_HOOK_URL="https://hooks.slack.com/services/xxxxxxx/xxxxxxx/xxxxxxxxxx"

events We want to be notified of all process state changes, so specify PROCESS_STATE.

environment Set the URL for Incoming WebHook in your environment variable to notify Slack.

Reference URL

• http://supervisord.org/events.html#configuring-an-event-listener

Creating an event listener

The final code looks like this:

/path/to/process_state_event_listener.py

import sys
import os
from supervisor import childutils
from datetime import datetime
import slackweb


def main():
    while True:
        run_forever()


def run_forever():

    headers, payload = childutils.listener.wait(sys.stdin, sys.stdout)
    eventname = headers['eventname']

    payload_header, data = childutils.eventdata(payload+'\n')

    childutils.listener.ok(sys.stdout)

    color = "good"
    if eventname in ['PROCESS_STATE_BACKOFF', 'PROCESS_STATE_EXITED', 'PROCESS_STATE_FATAL', 'PROCESS_STATE_UNKNOWN']:
        color = "danger"

    attachment = {
        "title": "event",
        "text": eventname,
        "color": color,
        "fields": [{
            "title": "datetime",
            "value": datetime.now().strftime("%Y/%m/%d %H:%M:%S")
        }, {
            "title": "hostname",
            "value": os.uname()[1]
        }]
    }

    if 'processname' in payload_header:
        attachment["fields"].append({
            "title": "process",
            "value": payload_header['processname']
        })

    if 'expected' in payload_header:
        attachment["fields"].append({
            "title": "expected",
            "value": payload_header['expected']
        })

    if 'SLACK_WEB_HOOK_URL' in os.environ:
        slack_url = os.environ['SLACK_WEB_HOOK_URL']
        slack = slackweb.Slack(url=slack_url)
        slack.notify(attachments=[attachment])


if __name__ == '__main__':
    main()

Notification image

Reference URL

--POST message from python to Slack via incoming webhook --Qiita

in conclusion

There is a plugin called superlance if you want to use email notification, but OP25B I didn't receive the email on gmail due to a problem, so I notified Slack. Since the use of Supervisor has increased, I would like to continue to apply it in various ways.