I typically have a number of smallish programs I want launched as daemons.
Getting those set up nicely typically requires root privileges, startup-scripts, etc.
which I find cumbersome. Instead of /etc/init.d and friends, I use a lovely
program called Supervisord. I think of Supervisord as my personal
init: it launches programs, monitors them for failure, logs their output, and restarts them.
Furthermore, it is trivial to set up new “daemons”, either manually
or via, say Ansible (which even has a module specifically for Supervisord).
The Supervisord package has Web and XML-RPC interfaces which can be disabled if you don’t need them.
After installing Supervisord (as root), I can choose to configure it in such a way as that it can be controlled by a non-privileged user (the configuration below does that: Supervisord will make my user owner of its socket, which allows me to control it without using sudo. (As you know, using sudo can be hazardous to my health … ;-)
[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
chmod=0700 ; socket file mode (default 0700)
chown=jpm:jpm ; socket file uid:gid owner
[inet_http_server]
port=127.0.0.1:9003
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
user=jpm ; (default is current user, required if root)
directory=/tmp ; (default is not to cd during start)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = /etc/supervisord.d/*.iniOnce Supervisord is running, I can create a small configuration file (ini -style) for a new program I want to run under its control. Here’s an example for mqttwarn:
[program:mqttwarn]
directory = /home/jpm/services/mqttwarn
command = /home/jpm/services/mqttwarn/mqttwarn.py
user = jpm
environment= MQTTWARNINI="/home/jpm/Auto/services/mqttwarn/mqttwarn.ini"If supervisord is running as root (which it typically does, but need not), I can configure the user under which my program is launched, set up specific environment variables, define expected exit status codes, whether or not to restart the program on exit, stdout redirection, logfile rotation, and a slew of other things.
I then use supervisorctl to “talk” to the supervisord process, and tell it to re-read its configuration and launch or stop processes depending on the state of the configuration:
$ cp xx.ini /etc/supervisord.d/mqttwarn.ini
$ supervisorctl reread
mqttwarn: available
$ supervisorctl update
mqttwarn: added process group
$ supervisorctl status
WSS RUNNING pid 7330, uptime 0:08:40
mqttwarn RUNNING pid 8331, uptime 0:00:12
yamlwatch RUNNING pid 7329, uptime 0:08:40
[...]
$Programs can be stopped or restarted. reread reloads the configuration without stopping or starting processes, and update will relaunch processes with changed configurations. So, if you want to pick up changes to config files with minimal disruptions, use reread followed by update.
If Supervisord’s XML-RPC interface is enabled, I can use the interface to check on Supervisord with, say, Nagios or Icinga. For a very simple example, this Python script connects to Supervisord and obtains information on whether WSS is running or not:
#!/usr/bin/env python
import xmlrpclib
import sys
# Simple Nagios/Icinga check for a process under Supervisord.
# Requires XML-RPC interface (inet) enabled
__author__ = 'Jan-Piet Mens <jpmens()gmail.com>'
supervisorduri = 'http://localhost:9003/RPC2'
procname = 'WSS'
OK = 0
WARNING = 1
CRITICAL = 2
stati = {
0 : 'OK',
1 : 'WARNING',
2 : 'CRITICAL',
3 : 'UNKNOWN' }
status = OK
statustext = None
server = xmlrpclib.Server(supervisorduri)
try:
info = server.supervisor.getProcessInfo(procname)
except xmlrpclib.Fault:
status = CRITICAL
statustext = "supervisor: process %s not found" % procname
except Exception, e:
status = CRITICAL
statustext = "supervisor: process %s: %s" % (procname, str(e))
state = info['statename']
if state != 'RUNNING':
status = CRITICAL
if statustext is None:
statustext = '%s supervisor for %s: %s' % (stati[status], procname, info['description'])
print statustext
sys.exit(status)The above program produces output like this:
OK supervisor for WSS: pid 10088, uptime 0:07:17Supervisord is highly recommended; I rely on it on my workstation and on servers.
Related: