So this weekend we were having some power outages where our onsite servers are located.  When the servers booted back up, Unicorn server was not running.  This was quick and easy to fix using the capistrano-unicorn gem:

If you get a generic looking 500 page then it may be the Unicorn server. You can check if Unicorn is running using ps aux as well. There should be one master and 5 workers running. If not you can use:

cap unicorn:reload # Reload Unicorn
cap unicorn:restart # Restart Unicorn
cap unicorn:show_vars # Debug Unicorn variables
cap unicorn:shutdown # Immediately shutdown Unicorn
cap unicorn:start # Start Unicorn master process
cap unicorn:stop # Stop Unicorn

Remember to use the development/staging/production in between the cap and unicorn (e.g. cap development unicorn:stop) In general, I would just use stop and start. It gives pretty good errors when you run it.

But that is a reactive fix and we want to be proactive and not think about these things on the weekend.  So, I tried a few other things like Monit but I could not get it to play nice with Unicorn and even then I realized that there was nothing setting Monit to start on server boot.  I read a couple of things about Bluepill gem and I decided to give it a try.

First off I added the Bluepill gem to my Gemfile but that did not seem to work so I installed on the server:

sudo gem install bluepill

Following the Github README, I created /var/run/bluepill directory for bluepill to store its pid and sock files.

Next I created a Bluepill configuration file in path/to/app/config and named it production.pill.  Bluepill can use different pill files based on the environment.

[sourcecode language="ruby"]

Bluepill.application("my_app", log_file: "/var/www/my_app/shared/log/bluepill.log") do |app|
app.process("unicorn") do |process|
process.working_dir = "/var/www/my_app/current"

process.start_grace_time = 20.seconds
process.stop_grace_time = 20.second
process.restart_grace_time = 20.seconds

process.pid_file = "/var/www/my_app/shared/pids/unicorn.pid"

process.start_command = "bundle exec unicorn -c /var/www/my_app/current/config/unicorn.rb -E production -D"
process.stop_command = "kill -QUIT "
process.restart_command = "kill -USR2 "

process.uid = "deploy"
process.gid = "deploy"
end
end

[/sourcecode]

We will want to test to see if Bluepill is working.  First we need to load the Bluepill configuration file and start it:

$  sudo bluepill load /var/www/my_app/current/config/production.pill

$  sudo bluepill status

Now we should have Bluepill monitoring our unicorn.  I tested this by stopping the unicorn server and checking the application and sure enough, the Unicorn master was restarted by Bluepill.  You can also watch the Bluepill logfile with sudo bluepill log unicorn.

So now that is all up and running, we need to make sure that Bluepill is started when the server boots.  Create a bluepill.cong in the /var/init/ directory and add this:

[sourcecode language="ruby"]

description "My App Bluepill"

start on runlevel [2]
stop on runlevel [016]

expect daemon
exec bluepill load /var/www/my_app/current/config/production.pill

respawn

[/sourcecode]