Archive for the ‘Hosting’ Category

Fast deployment using Capistrano, RVM and more

March 20th, 2012 5 comments

There is so much confusion around deploying applications using RVM, I went down and verified. The process was quite straight forward, but I found few points that could be improved.

So all my experiments were done using fresh rails app with one model, you can find sources here:

My goal was to be able to install rails app on server without complicated recipes and doing much on server.

So the boring part first, setup server:

  1. I’m assuming your domain (DNS) is configured and points to the server
  2. I’m assuming you have nginx up and running, most distributions have own instructions for it, but the process should be rather easy like apt-get install nginx
  3. To keep good security levels we assume one user is created per application, I used this command useradd --comment "AD - Example Rails App" -s /bin/bash -m -U ad but you might need other command depending on the system, do not forget to generate and exchange SSH keys
  4. Then I had to create nginx configuration for my site /etc/nginx/sites-available/`` the code bellow
  5. And finally link the configuration ln -s /etc/nginx/sites-available/ /etc/nginx/sites-enabled/, do not forget to restart nginx
server {
  listen 80;
  access_log /var/log/nginx/;
  error_log /var/log/nginx/;
  location / {
    proxy_pass http://unix:/home/ad/app/shared/pids/unicorn.socket;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;  # this is required for HTTPS (another server section):  # proxy_set_header X-Forwarded-Proto https;  }

Server is at this point ready and trying to proxy incoming connections to our application. As mentioned above I have minimal rails application, so locally I have installed capistrano and generated basic setup with capify . and replaced config/deploy.rb with this

We need also few gems to be added to Gemfile and then bundle installed:

group :development do
  gem 'capistrano'
  gem 'capistrano-unicorn'
  gem 'capistrano-file_db'

but feel free to check the whole file:

On the list is one new gem I have created to simplify deployments capistrano-file_db. It’s minimalistic gem to support deployment of file based databases (defaults to sqlite3).

I have reused also existing gem capistrano-unicorn – this one was not so straight and needed minimal hacking, but I was able to get it running in few minutes, I will be fixing the problems I found shortly, hopefully my changes get merged to master soon.

Lets go through the config/deploy.rb, the first section is pretty standard and reflects most of the configuration we did for server:

set :application, "ad"

set :deploy_to, "/home/#{application}/app"
set :user, "#{application}"
set :use_sudo, false
set :scm, :git
set :repository, "git://"

The fun starts later, first we set ruby version to our current local selected ruby.

set :rvm_ruby_string, ENV['GEM_HOME'].gsub(/.*\//,"")

Next we tell rvm to check for remote installation in user home – as by default it checks system installation.

set :rvm_type, :user

Here we instruct rvm to install head version remotely, this line can get removed after 1.11.0 gets out, it will be defaulting back to stable.

set :rvm_install_type, :head # before rvm 1.11.0 gets released

We tell bundler to omit development dependencies, which of course are our capistrano libraries.

set :bundle_without, [:development]

And finally we instruct unicorn plugin about the proper pid path, unfortunately it was not the only place we needed to setup it, check for more details:

set :unicorn_pid, "#{shared_path}/pids/"

That’s almost it, now lets tell capistrano where the deploy, and to always do migrations:

server "", :app, :web, :db, :primary => true

before 'deploy:restart', 'deploy:migrate'

This two steps are important, they tell capistrano to install rvm and ruby on server, it will also create application gemset if used, it will use rvm_ruby_string, so make sure to set it to something useful like `rvm current`:

before 'deploy:setup', 'rvm:install_rvm'
before 'deploy:setup', 'rvm:install_ruby'

And on the end all the capistrano plugins we want to use:

require "rvm/capistrano"
require "bundler/capistrano"
require "capistrano-unicorn"
require "capistrano-file_db"
load 'deploy/assets'

For the deploy/assets to work we have gem 'therubyracer' in Gemfile.

That’s all, let deploy our application, first prepare server

cap deploy:setup

cap deploy:cold

this will take some time so grab a tee/coffee.

After server is prepared we can deploy application after every commit to repository:

cap deploy

And it’s the end of the journey, it takes a lot longer to write this tutorial then to just do the deployment, doing it second time try to fit in 10 minutes !

You can get more details on RVM / Capistrano integration in the freshly updated documentation page: also worth reading is the new project files not requiring trusting:

don’t be afraid of nginx

April 6th, 2011 1 comment

I was just setting SSL on nginx for one of customers, that was extremely easy, everything done in few minutes, the nginx documentation guided me step by step, even with custom certificates.

If you ever had been considering switch then just have a look on the documentation, there is nothing to add:

Categories: Hosting, Linux Tags: , , ,

passenger standalone with god on rvm

March 21st, 2011 3 comments

Some time ago I have migrated my server to nginx. Now I got first chance to setup rails application on it.

Searching for new experiences I have tried a new setup – passenger standalone on rvm monitored by god and cron (see Q&A on the end).

Prerequisite to that guide is a working deployment script (putting files on server and db:migration working also rvm set up on server). Important is also to replace all occurrences of “/path/to/application” with your real path to application on server.

Nginx is also required, in case you do not have it already you can install it with following command (ubuntu server on linode):

sudo apt-get install nginx

Assuming you have an deploy script up and running you need to add few changes before deployment.

First add god configuration, in your RAILS_ROOT create file config/local.god:

passenger_config = { :instances => (1..3), :path => '/path/to/your/application/current' } do |w| = 'Passenger'
  w.start = "ps -U $USER u | grep -v awk | awk '/nginx/ {print $2}' | xargs kill 2>/dev/null ; passenger start #{passenger_config[:path]} -d -e production -S #{passenger_config[:path]}/tmp/pids/passenger.socket --pid-file #{passenger_config[:path]}/tmp/pids/ --min-instances #{passenger_config[:instances].min} --max-pool-size #{passenger_config[:instances].max}"
  w.stop = "passenger stop #{passenger_config[:path]} --pid-file #{passenger_config[:path]}/tmp/pids/"
  w.pid_file = "#{passenger_config[:path]}/tmp/pids/"
  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false

Also new gems have to be added in Gemfile:

group :production do 
  gem 'passenger', '~>3.0.5'
  gem 'god', '~>0.11.0'

Now deploy both files to server, they will be needed later on, and do not forget to run

bundle install --without=production

before deployment. Second add an request to restart passenger after deploy:

touch /path/to/application/tmp/restart.txt

If your deployment script is capistrano you should use the following snippet:

namespace :deploy do
  [:start, :stop].each do |method|
    task method, :roles => :app, :except => { :no_release => true } do
      run "local_god #{method} Passenger"
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "touch #{File.join(current_path,'tmp','restart.txt')}"

So now lets go to server to finish configuration.

We need to install god and passenger (which we added on the beginning):

bundle install --without=development test cucumber

please note that cucumber is only required when you use it, skip this param if cucumber is only an tasty vegetable for you.

One of most important steps is to create god wrapper that will be used for starting application (replace my_ruby_1.9.2@my_application with your rvm identifier):

rvm wrapper my_ruby_1.9.2@my_application local god

Next we need to make test run of passenger, it will also compile nginx and notify of any problems if it finds any:

cd /path/to/application/current ; passenger start

Passenger will be started on port 3000 if there is no firewall on server (you missed to install it) then you can visit your browser and enter your server.address:3000. adding -p 8080 would server application on the port 8080 (welcome java users). To stop passenger just hit CTRL+C :)

When that works we can set up proxy in the main nginx configuration. On ubuntu (linode) the steps look as follows:

sudo vim /etc/nginx/available-sites/your-application.conf

and put the following content (do not forget to replace my_domain with your dns name of the server):

server {
  listen 80;
  server_name my_domain;
  access_log /var/log/nginx/my_domain.access.log;
  error_log /var/log/nginx/my_domain.error.log;
  location / {
    proxy_pass http://unix:/path/to/application/current/tmp/pids/passenger.socket;
    proxy_set_header X-Real-IP $remote_addr;

after that (ESC :wq) you need to link configuration and restart server

sudo ln -s /etc/nginx/sites-available/your-application.conf /etc/nginx/sites-enabled/your-application.conf
sudo /etc/init.d/nginx restart

last step is to put local god into autorecovery mode with a simple cron job run crontab -e and enter the following content:

* * * * * ps aux | grep -v grep | grep -q "local\.god" || /home/user_name/.rvm/bin/books_god -c /path/to/application/current/config/local.god

That’s all, in one minute your server should be ready, do not forget that first call to passenger application might be sluggish, but it is only the first one when rails need to be initialized, rest of them is really fast.

To check the whole setup you could put small change in some view, deploy application, refresh your browsers (wait few seconds for rails startup) and check your results.

Q & A

Q: Why passenger, why not unicorn or thin ?

A: It is extremely easy to set up.

A: It is one of the fastest ruby servers (some say it is fastest one).

A: It takes far less memory then glassfish or TorqueBox.

Q: Why passenger standalone, not integrated into apache or nginx?

A: It is integrated with nginx, but installation is automated – almost no manual steps needed.

A: It runs from user context, so it can be fully controled by user and does not affect other users.

A: It does not need to recompile main server – no dependencies for serving other pages, multiple version can be run at the same time.

Q: Why rvm?

A: It allows easy maintenance of ruby version for user installations.

A: It allows to separate gem repositories for multiple applications.

Q: Why God?

A: It allows easy monitoring and management of processes – in our case manage passenger standalone.

Q: Why Capistrano?

A: Really?

Q: Why do you kill nginx when starting passanger?

A: It’s a hack, it might sometimes happen that passenger process is already stopped but nginx did not finished it’s operation yet, so to allow new process to start we kill the old one.

Q: Can I ask a new question, or provide more answers

A: I encourage you to ask questions and give answers, of course I will post only those that give any value.

fastest way to get git server v2

March 19th, 2011 1 comment

Over a year ago I have posted instruction how to setup simple git server, today I had to do it again but already having code so here is new instruction.

Create remote repository:

ssh -C "git init --bare repos/library3.git"

Create local repository:

git init
git add .
git commit -m "initial commit"

Tell local repository to synchronize with remote repository:

git remote add origin
git config branch.master.remote origin
git config branch.master.merge refs/heads/master
git push

You might be interested in the original post: fastest way to get git server

Categories: Development, Hosting, Linux Tags:

linux server memory management

February 27th, 2011 Comments off

It has been long time since last post, but finally something useful shown up, today I want to present simple script for monitoring memory in Linux.

But first happy news: our server is now sucessfully migrated to nginx, no more apache, now maximal memory usage is 20MB per process.

So here is script, it is almost simple, find some processes, and kill them if to much RAM used:


memory=$(free -m | awk '{if (FNR==2) print $2}')
count=$(ps aux | awk -v threshold=$threshold '{if (FNR==1 || $4>=threshold) print}' | tee $log | wc -l)
[ $count -gt 1 ] && echo "Showing processes over ${threshold_mb}MB, killing processes over ${kill_at_mb}MB" && cat $log

cat $log | awk -v kill_at=$kill_at '{if ($4>kill_at) print $2" "$4 }' | while read pid mem;
used_mb=$(($(echo $mem | sed 's/\.//')*memory/1000))
# fix rounding problem in awk
if [ $used_mb -gt $kill_at_mb ]; then
kill -9 $pid
echo "pid $pid used ${used_mb}MB - killed"
rm -f $log

Most important in this script are first two variables, threshold_mb which is a limit for showing processes (it is rough comparison) and kill_at_mb which is limit for killing processes, by default this limit is 5 times bigger then showing threshold. Cause killing is more important operation additional check for used memory is more strict to be sure, only processes over the limit are killed.

Save it to disk and in crontab:

sudo crontab -e

add the following lines:

MAILTO=<your @email>
* * * * * /root/bin/

That’s all, now add some filter in your mailbox to ignore mails form server – this is all automatic, but you might want to have this for later check if some important processes are missing.