The point of no return

Centralized Supervisor Interface: Cesi

28th December 2014 by Ali Erdinç Köroğlu

I already mentioned about supervisor in here. But how about to manage all supervisors from one web interface which has authorization and advanced process management filtering? As you know Supervisord provides a basic web interface to monitor and restart your processes where supervisor installed but XML-RPC interface and API allows you to control remote supervisor and the programs it runs. So it’s possible.. One UI to rule them all..

Here is Cesi (Centralized supervisor interface): it’s a web interface to manage supervisors from single UI, developed by Gülşah Köse and Kaan Özdinçer.

Application Dependencies

  1. Python : a programming language :)
  2. Flask : a microframework for Python
  3. SQlite : a self-contained, serverless, zero-configuration, transactional SQL database engine

I’ll cover everything step by step for CentOS7 minimal installation..
Since Flask is not in CentOS7 or EPEL repositories, I’ll install it via pip. So what is pip? Pip is a tool for installing and managing Python packages from Python Package Index repository. We also need Git (for cesi) and Nginx (you can get nginx from EPEL or from official Nginx repository)

Required CentOS packages

[root@supervisord ~]# yum install python-pip sqlite git nginx

Flask installation

[root@supervisord ~]# pip install flask

Getting a clone from github

[root@supervisord ~]# cd /opt/
[root@supervisord opt]# git clone https://github.com/Gamegos/cesi

Initial database operations

[root@supervisord cesi]# sqlite3 /opt/cesi/cesi/userinfo.db < userinfo.sql

Conf file should be in /etc

[root@supervisord cesi]# mv /opt/cesi/cesi/cesi.conf /etc

Let us define some remote supervisors

/etc/cesi.conf
[node:srv4]
username = superv
password = superv1q2w3e
host = 192.168.9.4
port = 9001
 
[node:srv10]
username = superv
password = superv
host = 192.168.9.10
port = 9001
 
[environment:glassfish]
members = srv4, srv10
 
[cesi]
database = /root/cesi/cesi/userinfo.db
activity_log = /root/cesi/cesi/cesi_activity.log

Since cesi will run by nginx user /opt/cesi should be accessible for it

[root@supervisord opt]# chown -R nginx:nginx /opt/cesi

We’ll run cesi via supervisord :)

/etc/supervisor.d/cesi.ini
[program:cesi]
command=/bin/python /opt/cesi/cesi/web.py
process_name=%(program_name)s
user=nginx
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/cesi-stdout.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
stderr_logfile=/var/log/cesi-stderr.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false

Flask will initialise twice the application in debug mode and when you try to stop via supervisord it will not stop.

root     17614  0.0  0.3 227168 12632 ?        Ss   21:05   0:00 /usr/bin/python /usr/bin/supervisord
nginx    17723  6.5  0.4 235768 16684 ?        S    21:36   0:00 /bin/python /opt/cesi/cesi/web.py
nginx    17728  6.5  0.4 309592 16936 ?        Sl   21:36   0:00 /bin/python /opt/cesi/cesi/web.py
[root@supervisord cesi]# supervisorctl stop cesi
cesi: stopped
[root@supervisord cesi]# ps aux
.
.
root     17614  0.0  0.3 227320 12664 ?        Ss   21:05   0:00 /usr/bin/python /usr/bin/supervisord
nginx    17728  0.5  0.4 309592 16936 ?        Sl   21:36   0:01 /bin/python /opt/cesi/cesi/web.py
root     17738  0.0  0.0 123356  1380 pts/0    R+   21:40   0:00 ps aux
[root@supervisord cesi]# supervisorctl stop cesi
cesi: ERROR (not running)

As you see it’s not stopped, so simplest thing would be to change use_reloader in web.py

/opt/cesi/cesi/web.py
--- web.py.org	2014-12-27 21:23:37.625143414 +0200
+++ web.py	2014-12-27 21:23:48.222118215 +0200
@@ -531,7 +531,7 @@
 
 try:
     if __name__ == '__main__':
-        app.run(debug=True, use_reloader=True)
+        app.run(debug=True, use_reloader=False)
 except xmlrpclib.Fault as err:
     print "A fault occurred"
     print "Fault code: %d" % err.faultCode

Working like a charm..

[root@supervisord cesi]# supervisorctl start cesi
cesi: started
[root@supervisord cesi]# ps aux| grep cesi
nginx    17704  0.2  0.4 237048 18232 ?        S    21:30   0:00 /bin/python /opt/cesi/cesi/web.py

System side is ok and let us configure the nginx..

/etc/nginx/conf.d/default.conf
server {
        server_name localhost 192.168.9.240 212.213.214.215;
 
        location / {
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://127.0.0.1:5000;
        }
 
        location /static {
                root /opt/cesi/cesi/;
        }
}

Let’s check TCP sockets

[root@supervisord cesi]# netstat -anp| grep LISTEN       
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN      17761/python        
tcp        0      0 192.168.9.240:9001      0.0.0.0:*               LISTEN      17614/python        
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      17842/nginx: master 
tcp        0      0 192.168.9.240:22        0.0.0.0:*               LISTEN      852/sshd

It’s time to login..


Login screen


Dashboard


All supervisors in one screen


A node


A process log

Kudos goes to Gülşah and Kaan and that’s all folks..

A New Supervisord Plugin for Nagios

23rd December 2014 by Ali Erdinç Köroğlu

In my last entry I mentioned about Supervisord and how beautiful it is. While I’m looking for how to integrate Supervisord to Nagios I couldn’t find what I’m looking for so I decided to write a new supervisord plugin for nagios. What I was looking for is a plugin works on Nagios server (so no need to install remote servers) to get information about processes from remote supervisords. So here it is.. check_supervisor

Usage: check_supervisord.py -H 192.168.1.1 -P 9001 -u superv -p superv -a glassfish
 
Options:
  -h, --help     
  -H HOSTNAME, --hostname=HOSTNAME (Supervisord hostname)
  -P PORT, --port=PORT (Supervisord port)
  -u USERNAME, --username=USERNAME (Supervisord username)
  -p PASSWORD, --password=PASSWORD (Supervisord password)
  -a PROCNAME, --process-name=PROCNAME (Process name defined in /etc/supervisor.d/*.ini or supervisorctl status)

Console Output:

[root@nagios ~]# /usr/lib64/nagios/plugins/check_supervisord.py -H 192.168.1.1 -P 9001 -u superv -p superv -a glassfish
glassfish OK: 12 day(s) 17 hour(s) 29 minute(s)

You should add the new command definition for check_supervisord into nagios’s commands.cfg

/etc/nagios/objects/commands.cfg
1
2
3
4
define command{
    command_name    check_supervisord
    command_line    $USER1$/check_supervisord.py -H $HOSTADDRESS$ -P $ARG1$ -u $ARG2$ -p $ARG3$ -a $ARG4$
}

And you can use in a definition like this :)

/etc/nagios/conf.d/services/glassfish.cfg
1
2
3
4
5
6
define service {
    use generic-service
    host_name   srv1
    service_description Glassfish
    check_command   check_supervisord!9001!superv!superv!glassfish
}


A screenshot from nagios web interface

Best Way to Daemonize Applications on Linux

22nd December 2014 by Ali Erdinç Köroğlu

I tried to explain how to daemonize applications before but how about monitor and even start/stop/restart processes locally or remotely? Well, here is Supervisor. Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.

As you know we need to write rc.d or systemd scripts for every single process instance. It’s hard to write and maintain also those scripts are not able to automatically restart a crashed process. So supervisord is the solution, it’s simple, efficient, centralized, extensible etc.. etc..

Supervisor has two component, supervisord and supervisorctl. Supervisord is the server side of supervisor and is responsible for starting child programs, controlling, logging and handling events. It’s also providing a web interface to view and control process status and an XML-RPC interface to control supervisor and the programs it runs. Supervisorctl on the otherhand, providing a shell-like interface for connecting to supervisord. But 1st let us install into our CentOS7 server. Supervisor is available on EPEL7 repository, if you don’t know how to add EPEL repository, please read this.

Installation is easy

[root@Neverland ~]# yum install supervisor

Please don’t forget to enable supervisor for systemd

[root@Neverland ~]# systemctl enable supervisord

It begins with a config file :)

/etc/supervisor.conf
[unix_http_server]
file=/var/tmp/supervisor.sock; (the path to the socket file)
 
[inet_http_server]      ; inet (TCP) server disabled by default 
port=192.168.1.1:9001   ; (ip_address:port specifier, *:port for all iface)
username=superv         ; (default is no username (open server))
password=superv         ; (default is no password (open server))
 
[supervisord]
logfile=/var/log/supervisor/supervisord.log  ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          ; (num of main logfile rotation backups;default 10)
loglevel=info               ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false              ; (start in foreground if true;default false)
minfds=1024                 ; (min. avail startup file descriptors;default 1024)
minprocs=200                ; (min. avail process descriptors;default 200)
 
[rpcinterface:supervisor]
supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface
 
[supervisorctl]
serverurl=unix:///var/tmp/supervisor.sock ; use a unix:// URL  for a unix socket
 
[include]
files = supervisord.d/*.ini

This process config file includes details such as directory, command, process name, process owner, logging etc. If you wanna know more, please read this.

/etc/supervisor.d/fixtures.ini
[program:fixtures]
directory=/opt/pronet/fixtures
command=/usr/java/jdk1.7.0_71/bin/java -Dfile.encoding=UTF-8 -Dproject.properties=/opt/pronet/fixtures/fixtures.properties -Dlog4j..
process_name=%(program_name)s
user=pronet
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/pronet/fixtures-stdout.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
stderr_logfile=/var/log/pronet/fixtures-stderr.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false

So when you start or restart supervisord fixtures process will start or restart too.. (depends on your process config)

[root@Neverland ~]# systemctl start supervisord
[root@Neverland ~]# supervisorctl status fixtures
fixtures                         RUNNING    pid 5786, uptime 0:00:03

And you can monitor or control remotely..

[root@nagios ~]# supervisorctl -s http://192.168.1.1:9001 -u superv -p superv status fixtures
fixtures                         RUNNING    pid 5786, uptime 0:04:20
[root@nagios ~]# supervisorctl -s http://192.168.1.1:9001 -u superv -p superv restart fixtures
fixtures: stopped
fixtures: started

There are some centralized supervisord web interfaces but I’ll cover them later :)