Monday, May 26, 2014

Supervisor

Supervisor is a great tool to start and manage long-running processes and the log files associated with them. It offers a lot of helpful features and is easy to get up and running.

Install & Start Supervisor

yum install python-setuptools
easy_install supervisor

 Configuring Supervisor


If no major errors ocurred, right now, we should have all we need installed. Now it’s time to configure it… Let’s start by putting in place the default supervisord configuration:
Once that is done, let’s go into the file supervisord.conf and add the include area in the bottom:

After this, we need to set Supervisor to run automatically every time you restart your server. In CentOS we need to create /rc.d/init.d/supervisord with the following:

 curl https://gist.githubusercontent.com/olsgreen/8122622/raw/c13dd4a2140d3471a0fbb460081ddfe6d51a8c54/supervisord.sh >/etc/init.d/supervisord

 Change :

prog_bin="${exec_prefix}/bin/supervisord -c /etc/supervisord.conf"

After finish that, we need to make our server knows about it:
Now that we have this all set, we need to inform Supervisor of which processes we would like it to monitor and keep running.
Did you get an error saying the process is already running? Did it says it can’t find the supervisord config?
Try the following…

Our First Program

Supervisor looks for configuration files in /etc/supervisor/conf.d/ and it is a good idea to keep a single conf file per process. To start, let's set up a simple long-running python script. Don't pay too much attention to the contents of the conf file, we'll go over that later.
Create /etc/supervisor/conf.d/test_python.conf containing:
[program:test_python]
command=python -u test.py
directory=/home/ubuntu
stdout_logfile=/home/ubuntu/test_python_output.txt
redirect_stderr=true
Create ~/test.py containing:
import time
while True:
 print(time.ctime())
 time.sleep(1)
When creating the conf file, be sure to replace /home/ubuntu both times with the path to your home directory (~ isn't allowed). If you run test.py manually you'll see it prints out the time once per second.

$ sudo supervisorctl
> reread # Tell supervisord to check for new items in /etc/supervisor/conf.d/
> add test_python # Add this process to Supervisord> start test_python# May say "already started"

You can check that it is indeed running that process with this command:
$ ps aux | grep php

# You should see some output like this:
php artisan queue:listen --env=your_environment
sh -c php artisan queue:work  --queue="default" --delay=0 --memory=128 --sleep --env=your_environment
php artisan queue:work --queue=default --delay=0 --memory=128 --sleep --env=your_environment

Using SupervisorCTL

SupervisorCTL is the tool you will use to manage everything in Supervisor. Start by running sudo supervisorctl and then typing reread. The reread command goes through the conf.d directory and loads any new program conf files. Once you see the new test_python program available, you can use the add command to load and start it (add test_python). The status command shows you all running processes, their PID and how long they've been running. Here is a short list of my commonly used supervisorctl commands and what they do.
  • reread - Reload all program conf files from the conf.d directory.
  • add [program_name] - Add a newly created conf file to Supervisor and start the process.
  • status - Check the status of all programs currently managed by Supervisor.
  • start [program_name] - Start the given program. Used often with one-time scripts.
  • restart [program_name] - Restart the given program.
  • tail -f [program_name] - Watch the log file in real-time (same as UNIX tail -f [filename]).
  • help - List all available commands.
Exit supervisorctl and check your log file (~/test_python_output.txt) to see that it is still running. If it isn't, you may have missed a step.

Program Configuration Files

There is a wide array of options you can use while setting up your program configuration file. I've listed my favorite options below. For a full list of commands and their parameters, check out the settings documentation.
  • command is the command to run.
  • directory is the directory to run the command in.
  • numprocs will automatically start N instances of that process.
  • autostart is a boolean which tells whether to run automatically or not when Supervisor starts.
  • autorestart will automatically restart a program that dies, under specific conditions (see docs).
  • stdout_logfile is where to log the output of your program.
  • redirect_stderr tells supervisor whether to include stderr in your standard output file.
  • stdout_logfile_maxbytes is the size a log file is allowed to grow to until it is auto rotated.
  • stdout_logfile_backups is the number of rotated log files to keep.

Web Interface

Supervisor provides a really nifty web interface to monitor and restart your processes. Just update the following file, restart supervisor, and you'll see it on port 9001. Feel free to change the port and protect it using HTTP Auth or whatever you see fit.
Add the following to /etc/supervisor/supervisord.conf
[inet_http_server] 
port=*:9001

One-Time Scripts

I've found that Supervisor also works great for running commonly used scripts that have multiple parameters or large log files. Be sure to set autostart andautorestart in the conf file to false, so your script doesn't end up running repeatedly.

Examples

MongoDB
[program:mongod_member]
command=/home/ubuntu/mongodb-linux-x86_64-2.0.6/bin/mongod --dbpath /mnt/mongo/ --replSet mongo5 --nojournal
directory=/home/ubuntu
stdout_logfile=/mnt/log/mongod.log
redirect_stderr=true
http://qiita.com/tooooooooomy/items/83970fac8aebe200a6b2