No more “bundle exec …”

I want to present the testing release of my gem rubygems-bundler 1.0.0 – integration for rubygems and bundler. Recently I was motivated to get back to it, actually I got back to it twice.

The first time thanks to Joshua Hull and his great effort on noexec gem, this gem was merged in to rubygems-bundler in version 0.9.0 – this was a great improvement, providing automatic detection if the gem should be loaded using Bundler.setup or not.

The second time was thanks to discussions with Evan Phoenix and Yehuda Katz. Evan motivated me to remove the installation of the wrapper via extensions, which was a hack. Yehuda gave me good ideas on handling the wrapper and automating things, so modification of ~/.gemrc is not needed anymore.

So, rubygems-bundler is now at version 1.0.0.beta5, and should be available, officially, as 1.0.0 very shortly.

Features

• Automatically uses custom_shebang when not specified by user in ~/.gemrc.
• Automatically installs wrapper when not available during gem install ... or gem regenerate_binstubs
• Backports custom_shebang from rubygems 2.0(master) to any rubygems down to 1.3.7.
• Adds the command gem regenerate_binstubs which allows regeneration of binstubs for existing gems. It does not, however, regenerate extensions.
• Adds an uninstaller rubygems-bundler-uninstaller which will restore binstubs for gems to use the safer #!/usr/bin/env ruby, and remove the existing wrapper.

Installation / Testing

gem install rubygems-bundler --pre

You might get warnings to uninstall, this is needed to prevent collisions with the rubygems plugin in older versions.

What’s the future of this gem?

Yehuda and I discussed it, and we have agreed to merge it into bundler, as it’s the most user friendly solution that could be done. We will be working hard to integrate it into bundler 1.3 (1.2 is in testing now).

Some parts of this gem will be made obsolete with the release of rubygems 2.0: custom_shebang – already implemented and gem regenerate_binstubs – will be part of gem pristine (#326).

Unfortunately it is to late to merge custom shebang back to rubygems 1.8 (#325).

My dream is that we will no longer have to create third-party tools (like my gem) and the integration will be fully done between rubygems 2.0 and bundler 1.3.

But, even then, you can still use my gem if you require lower versions of rubygems or bundler.

There is so much confusion around deploying applications using RVM, I went down and verified. The process was quite straight forward, but I found few points that could be improved.

So all my experiments were done using fresh rails app with one model, you can find sources here: https://github.com/mpapis/ad

My goal was to be able to install rails app on server without complicated recipes and doing much on server.

So the boring part first, setup server:

1. I’m assuming your domain (DNS) is configured and points to the server
2. I’m assuming you have nginx up and running, most distributions have own instructions for it, but the process should be rather easy like apt-get install nginx
3. To keep good security levels we assume one user is created per application, I used this command useradd --comment "AD - Example Rails App" -s /bin/bash -m -U ad but you might need other command depending on the system, do not forget to generate and exchange SSH keys
4. Then I had to create nginx configuration for my site /etc/nginx/sites-available/ad.niczsoft.com the code bellow
5. And finally link the configuration>ln -s /etc/nginx/sites-available/ad.niczsoft.com /etc/nginx/sites-enabled/ad.niczsoft.com, do not forget to restart nginx

server {
  listen 80;
  server_name ad.niczsoft.com;
  access_log /var/log/nginx/ad.niczsoft.com.access.log;
  error_log /var/log/nginx/ad.niczsoft.com.error.log;
  location / {
    proxy_pass http://unix:/home/ad/app/shared/pids/unicorn.socket;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;
    # this is required for HTTPS (another server section):
    # proxy_set_header X-Forwarded-Proto https;
  }
}

Server is at this point ready and trying to proxy incoming connections to our application. As mentioned above I have minimal rails application, so locally I have installed capistrano and generated basic setup with capify . and replaced config/deploy.rb with this https://github.com/mpapis/ad/blob/master/config/deploy.rb

We need also few gems to be added to Gemfile and then bundle installed:

group :development do
  gem 'capistrano'
  gem 'capistrano-unicorn'
  gem 'capistrano-file_db'
end

but feel free to check the whole file: https://github.com/mpapis/ad/blob/master/Gemfile

On the list is one new gem I have created to simplify deployments capistrano-file_db. It’s minimalistic gem to support deployment of file based databases (defaults to sqlite3).

I have reused also existing gem capistrano-unicorn – this one was not so straight and needed minimal hacking, but I was able to get it running in few minutes, I will be fixing the problems I found shortly, hopefully my changes get merged to master soon.

Lets go through the config/deploy.rb, the first section is pretty standard and reflects most of the configuration we did for server:

set :application, "ad"

set :deploy_to, "/home/#{application}/app"
set :user, "#{application}"
set :use_sudo, false
set :scm, :git
set :repository, "git://github.com/mpapis/ad.git"

The fun starts later, first we set ruby version to our current local selected ruby.

set :rvm_ruby_string, ENV['GEM_HOME'].gsub(/.*\//,"")

Next we tell rvm to check for remote installation in user home – as by default it checks system installation.

set :rvm_type, :user

Here we instruct rvm to install head version remotely, this line can get removed after 1.11.0 gets out, it will be defaulting back to stable.

set :rvm_install_type, :head # before rvm 1.11.0 gets released

We tell bundler to omit development dependencies, which of course are our capistrano libraries.

set :bundle_without, [:development]

And finally we instruct unicorn plugin about the proper pid path, unfortunately it was not the only place we needed to setup it, check https://github.com/mpapis/ad/blob/master/config/unicorn/production.rb for more details:

set :unicorn_pid, "#{shared_path}/pids/unicorn.pid"

That’s almost it, now lets tell capistrano where the deploy, and to always do migrations:

server "niczsoft.com", :app, :web, :db, :primary => true

before 'deploy:restart', 'deploy:migrate'

This two steps are important, they tell capistrano to install rvm and ruby on server, it will also create application gemset if used, it will use rvm_ruby_string, so make sure to set it to something useful like `rvm current`:

before 'deploy:setup', 'rvm:install_rvm'
before 'deploy:setup', 'rvm:install_ruby'

And on the end all the capistrano plugins we want to use:

$:.unshift(File.expand_path('./lib', ENV['rvm_path']))
require "rvm/capistrano"
require "bundler/capistrano"
require "capistrano-unicorn"
require "capistrano-file_db"
load 'deploy/assets'

For the deploy/assets to work we have gem 'therubyracer' in Gemfile.

That’s all, let deploy our application, first prepare server

cap deploy:setup && cap deploy:cold

this will take some time so grab a tee/coffee.

After server is prepared we can deploy application after every commit to repository:

cap deploy

And it’s thue end of the journey, it takes a lot longer to write this tuttorial then to just do the deployment, doing it second time try to fit in 10 minutes !

You can get more details on RVM / Capistrano integration in the freshly updated documentation page: http://beginrescueend.com/integration/capistrano/ also worth reading is the new project files not requiring trusting: http://beginrescueend.com/workflow/projects/#ruby-versions

By now you have most likely heard of RVM and rbenv. (There is no mistake in writing one in uppercase and second in lowercase, I’m just following conventions given by their respective authors.) As a contributor in the RVM Project I thought I could find out what really rbenv does. Reading the sources gave me some insights into rbenv – unfortunately there was no new shell trick to learn for me from rbenv. Still I could understand how it works and find differences between the two tools. Both claim, and do provide, a way to switch the currently active ruby.

The main difference between them is when the ruby is switched.

• For RVM, it’s during changing directories – the process is done once, and from now on the shell already has all ruby and gem binaries available. This can also be triggered manually.
• For rbenv, all ruby and gem binaries are always available in shell but will load proper code only when in proper context – so switching is done each time any binary is executed.

Going into more details: RVM will take about 60ms on every called cd, which means you would need to type over 16 cd commands to take a whole second – try to do that. During the directory change, RVM will set up environment variables to reflect your ruby and gemset selection allowing it to be separated from other rubies and gems, or binaries. You will not be able to run any other binary or gem available in another project, and any auto-loading mechanisms using plugins will not be able to load them from other project environment.

For rbenv, switching directories with cd takes no time (0ms) so it is faster by default for changing directories. After installing some gems, you need to rehash and it takes 60ms. Using ruby or switching to a project dir does not change the user’s environment almost at all, except for the RBENV_VERSION variable for current – but it does not mean the env is not set at all – it will be set internally every time ruby and any other ruby or gem binary is run – with a overhead of about 50ms for each invocation.

To explain in depth what rbenv does here is a quote I was originally going to put in here but I already posted the message on reddit:

There is one big hole in shims approach – it makes all the gem and ruby binaries available in your shell … just not always functional.

Having something installed in one ruby (like haml) will make it shim available in all rubies – just not working.

Shim approach is working against the system – building another abstraction layer not respecting UNIX mechanisms – like PATH search – just fooling your system (and you).

It is not possible to run a shim/ruby without rbenv loaded – it s required to to make the environment working as expected – in contrary RVM by default builds environment files which sourced once make your system aware of the exact combination of ruby and gems you want to use.

The environment is set and provided to the ran binary in both cases, only the burden of loading it is shifted, and it means:

• The loading place makes some difference. In RVM, you can check the state of the environment with ‘rvm info’. For rbenv, there are no changes – so there is nothing to check.
• For RVM, only the selected ruby/gemset related binaries will be available in the shell. In rbenv, all binaries of all rubies and gems all always available in shell – only they will do nothing when called, it is not possible to check if binary exists in system as the checks always will hit the shim binary and report it as existing.
• For those concerned about loading and execution time, you can not see the difference. It can be only measured. Anything that occurs at or below 300ms in a shell is imperceptible.

Sam Stephenson provided a list of points that make a difference between the tools, lets consider them:

1. Need to be loaded into your shell. Instead, rbenv’s shim approach works by adding a directory to your $PATH.
   RVM also does not have to be loaded as a function. It allows operation by loading only the environment file, which is done only once and allows single and fast initialization of the current ruby and gemset. RVM also allows operation as a binary when all it’s functionalities are available (except changing environment other than setting default, which behaves identically to rbenv’s only means of operation).
2. Override shell commands like cd. That’s dangerous and error-prone.
   Overriding of cd is optional. I searchied for almost 8 hours collectively over the last month to find a project that overrides cd – guess what, I could not find one. There are few shell tricks that allow you to override cd, but as cd override is optional, you can do it in your own function merged with the tricks. As for error prone – I see no reports of errors about the cd function. Once it is done, nothing bad can happen. Additoinally RVM provides a security feature whereby you can view the code to be executed ahead of time upon cd and choose to trust it or not.
3. Have a configuration file. There’s nothing to configure except which version of Ruby you want to use.
   rbenv, at present, allows setting over 4 shell variables which influence the running process, there is no configuration file that could collect them – they have to be set in your shell rc files – for every shell you use.
4. Install Ruby. You can build and install Ruby yourself, or use ruby-build to automate the process.
   During it’s life RVM collected lots of knowledge how to manage and install rubies – including patches that make ruby work as expected on different environments. Also supporting fancy things like golf ruby – where original ruby binary is replaced with goruby binary. Why would you not want to capitalize on other’s experience?
5. Manage gemsets. Bundler is a better way to manage application dependencies. If you have projects that are not yet using Bundler you can install the rbenv-gemset plugin.
   Using gemsets is optional but encouraged as it helps in the separation process, some gems force use of bundler but this does not solve all the separation problems. And the pure existence of rbenv-gemsets proves how useful it is. Even bundler is not impervious to plethora of installed system level gems and their oddities of interaction.
6. Require changes to Ruby libraries for compatibility. The simplicity of rbenv means as long as it’s in your $PATH, nothing else needs to know about it.
   RVM does not require changes to any gems/libraries although it allows some additions that are meant to make life easier – like per project changing PATH for bundler generated binaries, or allowing set :rvm_ruby_string to specify a ruby version instead of setting PATH like rbenv requires. All those things are optional and not required for proper work neither of RVM or the tool concerned. Finally, separation of gemsets in RVM allows skipping calls to bundle exec – which in case of rbenv will be required (unless using my gem rubygems-bundler gem or other such tricks).
7. Prompt you with warnings when you switch to a project. Instead of executing arbitrary code, rbenv reads just the version name from each project. There’s nothing to “trust.”
   All the warnings appear only when users create the .rvmrc file manually, if it is created automatically with the –rvmrc flag then a lot less warnings are printed. rbenv actually prints a warning when switching to a project which wants to use ruby that is not installed. Trusting of .rvmrc files can be easily switched to not read them at all or to trust any .rvmrc file.

Opposed to Sam Stephenson points above I would like to give my counterarguments for using rbenv:

1. Execution of any binary is 50ms slower as compared to binaries run in rvm – so if you are using many calls to ruby – this might get in your way of your performance.
2. Execution of any binary does not require proper ruby/gem to be available, it will silently fail without any warning if it is not available.
3. It is not possible to check if ruby or gem binary is available in the system, separate calls to gem, binaries or rbenv are required to find out if it is really available – which adds more work you have to do / worry about. With RVM you set environment – and LINUX conventions allow it to work as expected, with rbenv – the environment is faked with shims, you never know (by linux conventions) what will happen.

So summing up the tools are different but they allow developers to do almost exactly the same thing just with different approaches, knowing the differences could help developers chose the right tool for the right job.

Important point mentioned quite often when RVM and rbenv is compared is the size and complication of RVM. Where rbenv is a new tool, it is relatively small and its internals are less complex. rbenv uses the same approaches as RVM and is more readable if only by smaller code base which is due to the fact it is a far less mature piece of software and has far less experience accommodating different environments and platforms. It has not been extensively tested on most available operating systems. While RVM continues to press forward with new features on a solid software foundation, rbenv will continue to grow and progress but will continue to ride RVM’s coattails. It’s simple to take an existing piece of open source software and apply hindsight to create a more slim, less feature packed and less tested piece of software. We wish to see rbenv grow and expand into a great piece of software that provides a different solution to ruby environment management so both projects may learn from one another.

There is one additional thing that Is available for RVM which you will not find for rbenv – Jewelery Box – the official OS X RVM GUI. As rbenv claims to be minimalistic tool it does not seam to need any GUI – but as popularity of JB shows there is a need for such tool.

In closing, I would like to mention that we know that RVM grow big and could use some refactoring to make it easier to hack on it and to make the code base more readable and maintainable – we are working on that by planning RVM2 as an SM extension – SM is another project from Wayne E. Seguin worth checking and big enough to cover few articles. We hear our users loud and clear and have big plans for the future of RVM.

About half year ago I have written a gem to eliminate the need of using bundle exec – rubygems-bundler. For now at least 480 developers decide it is useful (downloads count for current version).

Yesterday I finally was able to close issue 3 for rubygems-bundler, but as the support for plugins was enabled only on 1.1 branch – today I have released a bundler gem fork of the latest version 1.0.21 – mpapis-bundler, including only two changes compared the original gem in version 1.0.21:

1. Support for rubygems plugins cherry picked from 7b4bd9e3d7a644dd4ad73fe95e2786e8f7411d04
2. Rebranding and dependency to rubygems-bundler 8b34ef025d57a6cef58d1d8fdf486d4c25fe7d69

So anyone tired of writing bundle exec can now easily avoid it by using mpapis-bundler instead of bundler.

Both gems rubygems-bundler and mpapis-bundler are backporting functionalities that will be available in rubygems and bundler – don’t wait for future, future is now.

Ever used grep and wanted to check that file with less? Now it is easy with wrapping functions, they parse grep outputted lines.

So lets say we search for info in rvm packages:

grep -rn '__rvm_use(' ~/.rvm/scripts
/home/mpapis/.rvm/scripts/selector:391:__rvm_use()

So now you want to see what is inside of ‘.rvm/scripts/selector’ at line 391

less -N +j391 /home/mpapis/.rvm/scripts/selector

But wouldn’t be nice to just write less, double click the found line and middle click:

less /home/mpapis/.rvm/scripts/selector:391:__rvm_use

This will work when using following function:

less()
{
  local params=() param parsed
  for param in "$@"
  do
    case $param in
    (*:*)
      parsed=( ${param//:/ } )
      params+=( "-N" "+j${parsed[1]}" "${parsed[0]}" )
      ;;
    (*)
      params+=( "$param" )
      ;;
    esac
  done
  command less "${params[@]}"
}

Put the above code into ~/.functions and source it in ~/.bashrc

test -s ~/.functions && . ~/.functions || true

Now restart your shell and enjoy you extended less function.