TF – Testing Framework

For some time I have been testing shell scripts using my own creation TF – Testing Framework.

I’m proud to present release 0.4.0 of it today. It includes:

• 2-4x speed improvement depending on used Ruby interpreter,
• add support for different shells via shebang
• improved validation of environment variables, including array variables testing and testing environment variable type,
• extended output matching allowing to separately match stdout and stderr,

Installation

Install using (ruby required):

gem install tf

Supported tests

The test can be negated by replacing = with !=:

• #status=<number> – check if command returned given status (0 is success)
• #match=/<regexp>/ – regexp match command output both stdout and stderr
• #match[stdout|stderr]=/<regexp>/ – regexp match command either stdout or stderr
• #env[<var_name>]=~/<regexp>/ – regexp match the given environment variable name
• #env[<var_name>]?=[array|string|nil] – verify type of the given environment variable name
• #env[<var_name><var_name>][]=<size> – verify size of the given environment variable name
• #env[<var_name>][]=/<regexp>/ – regexp match all of elements of the given environment variable name
• #env[<var_name>][<index>]=/<regexp>/ – regexp match given element of the given environment variable name

And here is very simple test:

true # status=0

false # status!=0

echo "Super" # match=~/^Super$/

rvm install 1.9.3 # match[stderr]!=/Error/; status=0

rvm alias create default 1.9.3 # status=0

TF is used for testing RVM, SM Framework and rubygems-bundler. Let me know if any other awesome projects use TF!

TF was earlier known as DTF – Deryl’s Testing Framework, but as original author Deryl wanted to go in a bit different direction (more validation related), I had to move code to this new repository and name.

No more “bundle exec …”

I want to present the testing release of my gem rubygems-bundler 1.0.0 – integration for rubygems and bundler. Recently I was motivated to get back to it, actually I got back to it twice.

The first time thanks to Joshua Hull and his great effort on noexec gem, this gem was merged in to rubygems-bundler in version 0.9.0 – this was a great improvement, providing automatic detection if the gem should be loaded using Bundler.setup or not.

The second time was thanks to discussions with Evan Phoenix and Yehuda Katz. Evan motivated me to remove the installation of the wrapper via extensions, which was a hack. Yehuda gave me good ideas on handling the wrapper and automating things, so modification of ~/.gemrc is not needed anymore.

So, rubygems-bundler is now at version 1.0.0.beta5, and should be available, officially, as 1.0.0 very shortly.

Features

• Automatically uses custom_shebang when not specified by user in ~/.gemrc.
• Automatically installs wrapper when not available during gem install ... or gem regenerate_binstubs
• Backports custom_shebang from rubygems 2.0(master) to any rubygems down to 1.3.7.
• Adds the command gem regenerate_binstubs which allows regeneration of binstubs for existing gems. It does not, however, regenerate extensions.
• Adds an uninstaller rubygems-bundler-uninstaller which will restore binstubs for gems to use the safer #!/usr/bin/env ruby, and remove the existing wrapper.

Installation / Testing

gem install rubygems-bundler --pre

You might get warnings to uninstall, this is needed to prevent collisions with the rubygems plugin in older versions.

What’s the future of this gem?

Yehuda and I discussed it, and we have agreed to merge it into bundler, as it’s the most user friendly solution that could be done. We will be working hard to integrate it into bundler 1.3 (1.2 is in testing now).

Some parts of this gem will be made obsolete with the release of rubygems 2.0: custom_shebang – already implemented and gem regenerate_binstubs – will be part of gem pristine (#326).

Unfortunately it is to late to merge custom shebang back to rubygems 1.8 (#325).

My dream is that we will no longer have to create third-party tools (like my gem) and the integration will be fully done between rubygems 2.0 and bundler 1.3.

But, even then, you can still use my gem if you require lower versions of rubygems or bundler.

There is so much confusion around deploying applications using RVM, I went down and verified. The process was quite straight forward, but I found few points that could be improved.

So all my experiments were done using fresh rails app with one model, you can find sources here: https://github.com/mpapis/ad

My goal was to be able to install rails app on server without complicated recipes and doing much on server.

So the boring part first, setup server:

1. I’m assuming your domain (DNS) is configured and points to the server
2. I’m assuming you have nginx up and running, most distributions have own instructions for it, but the process should be rather easy like apt-get install nginx
3. To keep good security levels we assume one user is created per application, I used this command useradd --comment "AD - Example Rails App" -s /bin/bash -m -U ad but you might need other command depending on the system, do not forget to generate and exchange SSH keys
4. Then I had to create nginx configuration for my site /etc/nginx/sites-available/ad.niczsoft.com`` the code bellow
5. And finally link the configuration ln -s /etc/nginx/sites-available/ad.niczsoft.com /etc/nginx/sites-enabled/ad.niczsoft.com, do not forget to restart nginx

server {
  listen 80;
  server_name ad.niczsoft.com;
  access_log /var/log/nginx/ad.niczsoft.com.access.log;
  error_log /var/log/nginx/ad.niczsoft.com.error.log;
  location / {
    proxy_pass http://unix:/home/ad/app/shared/pids/unicorn.socket;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;  # this is required for HTTPS (another server section):  # proxy_set_header X-Forwarded-Proto https;  }
}

Server is at this point ready and trying to proxy incoming connections to our application. As mentioned above I have minimal rails application, so locally I have installed capistrano and generated basic setup with capify . and replaced config/deploy.rb with this https://github.com/mpapis/ad/blob/master/config/deploy.rb

We need also few gems to be added to Gemfile and then bundle installed:

group :development do
  gem 'capistrano'
  gem 'capistrano-unicorn'
  gem 'capistrano-file_db'
end

but feel free to check the whole file: https://github.com/mpapis/ad/blob/master/Gemfile

On the list is one new gem I have created to simplify deployments capistrano-file_db. It’s minimalistic gem to support deployment of file based databases (defaults to sqlite3).

I have reused also existing gem capistrano-unicorn – this one was not so straight and needed minimal hacking, but I was able to get it running in few minutes, I will be fixing the problems I found shortly, hopefully my changes get merged to master soon.

Lets go through the config/deploy.rb, the first section is pretty standard and reflects most of the configuration we did for server:

set :application, "ad"

set :deploy_to, "/home/#{application}/app"
set :user, "#{application}"
set :use_sudo, false
set :scm, :git
set :repository, "git://github.com/mpapis/ad.git"

The fun starts later, first we set ruby version to our current local selected ruby.

set :rvm_ruby_string, ENV['GEM_HOME'].gsub(/.*\//,"")

Next we tell rvm to check for remote installation in user home – as by default it checks system installation.

set :rvm_type, :user

Here we instruct rvm to install head version remotely, this line can get removed after 1.11.0 gets out, it will be defaulting back to stable.

set :rvm_install_type, :head # before rvm 1.11.0 gets released

We tell bundler to omit development dependencies, which of course are our capistrano libraries.

set :bundle_without, [:development]

And finally we instruct unicorn plugin about the proper pid path, unfortunately it was not the only place we needed to setup it, check https://github.com/mpapis/ad/blob/master/config/unicorn/production.rb for more details:

set :unicorn_pid, "#{shared_path}/pids/unicorn.pid"

That’s almost it, now lets tell capistrano where the deploy, and to always do migrations:

server "niczsoft.com", :app, :web, :db, :primary => true

before 'deploy:restart', 'deploy:migrate'

This two steps are important, they tell capistrano to install rvm and ruby on server, it will also create application gemset if used, it will use rvm_ruby_string, so make sure to set it to something useful like `rvm current`:

before 'deploy:setup', 'rvm:install_rvm'
before 'deploy:setup', 'rvm:install_ruby'

And on the end all the capistrano plugins we want to use:

require "rvm/capistrano"
require "bundler/capistrano"
require "capistrano-unicorn"
require "capistrano-file_db"
load 'deploy/assets'

For the deploy/assets to work we have gem 'therubyracer' in Gemfile.

That’s all, let deploy our application, first prepare server

cap deploy:setup

cap deploy:cold

this will take some time so grab a tee/coffee.

After server is prepared we can deploy application after every commit to repository:

cap deploy

And it’s the end of the journey, it takes a lot longer to write this tutorial then to just do the deployment, doing it second time try to fit in 10 minutes !

You can get more details on RVM / Capistrano integration in the freshly updated documentation page: http://beginrescueend.com/integration/capistrano/ also worth reading is the new project files not requiring trusting: http://beginrescueend.com/workflow/projects/#ruby-versions

By now you have most likely heard of RVM and rbenv. (There is no mistake in writing one in uppercase and second in lowercase, I’m just following conventions given by their respective authors.) As a contributor in the RVM Project I thought I could find out what really rbenv does. Reading the sources gave me some insights into rbenv – unfortunately there was no new shell trick to learn for me from rbenv. Still I could understand how it works and find differences between the two tools. Both claim, and do provide, a way to switch the currently active ruby.

The main difference between them is when the ruby is switched.

• For RVM, it’s during changing directories – the process is done once, and from now on the shell already has all ruby and gem binaries available. This can also be triggered manually.
• For rbenv, all ruby and gem binaries are always available in shell but will load proper code only when in proper context – so switching is done each time any binary is executed.

Going into more details: RVM will take about 60ms on every called cd, which means you would need to type over 16 cd commands to take a whole second – try to do that. During the directory change, RVM will set up environment variables to reflect your ruby and gemset selection allowing it to be separated from other rubies and gems, or binaries. You will not be able to run any other binary or gem available in another project, and any auto-loading mechanisms using plugins will not be able to load them from other project environment.

For rbenv, switching directories with cd takes no time (0ms) so it is faster by default for changing directories. After installing some gems, you need to rehash and it takes 60ms. Using ruby or switching to a project dir does not change the user’s environment almost at all, except for the RBENV_VERSION variable for current – but it does not mean the env is not set at all – it will be set internally every time ruby and any other ruby or gem binary is run – with a overhead of about 50ms for each invocation.

To explain in depth what rbenv does here is a quote I was originally going to put in here but I already posted the message on reddit:

There is one big hole in shims approach – it makes all the gem and ruby binaries available in your shell … just not always functional.

Having something installed in one ruby (like haml) will make it shim available in all rubies – just not working.

Shim approach is working against the system – building another abstraction layer not respecting UNIX mechanisms – like PATH search – just fooling your system (and you).

It is not possible to run a shim/ruby without rbenv loaded – it s required to to make the environment working as expected – in contrary RVM by default builds environment files which sourced once make your system aware of the exact combination of ruby and gems you want to use.

The environment is set and provided to the ran binary in both cases, only the burden of loading it is shifted, and it means:

• The loading place makes some difference. In RVM, you can check the state of the environment with ‘rvm info’. For rbenv, there are no changes – so there is nothing to check.
• For RVM, only the selected ruby/gemset related binaries will be available in the shell. In rbenv, all binaries of all rubies and gems all always available in shell – only they will do nothing when called, it is not possible to check if binary exists in system as the checks always will hit the shim binary and report it as existing.
• For those concerned about loading and execution time, you can not see the difference. It can be only measured. Anything that occurs at or below 300ms in a shell is imperceptible.

Sam Stephenson provided a list of points that make a difference between the tools, lets consider them:

1. Need to be loaded into your shell. Instead, rbenv’s shim approach works by adding a directory to your $PATH.
   RVM also does not have to be loaded as a function. It allows operation by loading only the environment file, which is done only once and allows single and fast initialization of the current ruby and gemset. RVM also allows operation as a binary when all it’s functionalities are available (except changing environment other than setting default, which behaves identically to rbenv’s only means of operation).
2. Override shell commands like cd. That’s dangerous and error-prone.
   Overriding of cd is optional. I searchied for almost 8 hours collectively over the last month to find a project that overrides cd – guess what, I could not find one. There are few shell tricks that allow you to override cd, but as cd override is optional, you can do it in your own function merged with the tricks. As for error prone – I see no reports of errors about the cd function. Once it is done, nothing bad can happen. Additoinally RVM provides a security feature whereby you can view the code to be executed ahead of time upon cd and choose to trust it or not.
3. Have a configuration file. There’s nothing to configure except which version of Ruby you want to use.
   rbenv, at present, allows setting over 4 shell variables which influence the running process, there is no configuration file that could collect them – they have to be set in your shell rc files – for every shell you use.
4. Install Ruby. You can build and install Ruby yourself, or use ruby-build to automate the process.
   During it’s life RVM collected lots of knowledge how to manage and install rubies – including patches that make ruby work as expected on different environments. Also supporting fancy things like golf ruby – where original ruby binary is replaced with goruby binary. Why would you not want to capitalize on other’s experience?
5. Manage gemsets. Bundler is a better way to manage application dependencies. If you have projects that are not yet using Bundler you can install the rbenv-gemset plugin.
   Using gemsets is optional but encouraged as it helps in the separation process, some gems force use of bundler but this does not solve all the separation problems. And the pure existence of rbenv-gemsets proves how useful it is. Even bundler is not impervious to plethora of installed system level gems and their oddities of interaction.
6. Require changes to Ruby libraries for compatibility. The simplicity of rbenv means as long as it’s in your $PATH, nothing else needs to know about it.
   RVM does not require changes to any gems/libraries although it allows some additions that are meant to make life easier – like per project changing PATH for bundler generated binaries, or allowing set :rvm_ruby_string to specify a ruby version instead of setting PATH like rbenv requires. All those things are optional and not required for proper work neither of RVM or the tool concerned. Finally, separation of gemsets in RVM allows skipping calls to bundle exec – which in case of rbenv will be required (unless using my gem rubygems-bundler gem or other such tricks).
7. Prompt you with warnings when you switch to a project. Instead of executing arbitrary code, rbenv reads just the version name from each project. There’s nothing to “trust.”
   All the warnings appear only when users create the .rvmrc file manually, if it is created automatically with the –rvmrc flag then a lot less warnings are printed. rbenv actually prints a warning when switching to a project which wants to use ruby that is not installed. Trusting of .rvmrc files can be easily switched to not read them at all or to trust any .rvmrc file.

Opposed to Sam Stephenson points above I would like to give my counterarguments for using rbenv:

1. Execution of any binary is 50ms slower as compared to binaries run in rvm – so if you are using many calls to ruby – this might get in your way of your performance.
2. Execution of any binary does not require proper ruby/gem to be available, it will silently fail without any warning if it is not available.
3. It is not possible to check if ruby or gem binary is available in the system, separate calls to gem, binaries or rbenv are required to find out if it is really available – which adds more work you have to do / worry about. With RVM you set environment – and LINUX conventions allow it to work as expected, with rbenv – the environment is faked with shims, you never know (by linux conventions) what will happen.

So summing up the tools are different but they allow developers to do almost exactly the same thing just with different approaches, knowing the differences could help developers chose the right tool for the right job.

Important point mentioned quite often when RVM and rbenv is compared is the size and complication of RVM. Where rbenv is a new tool, it is relatively small and its internals are less complex. rbenv uses the same approaches as RVM and is more readable if only by smaller code base which is due to the fact it is a far less mature piece of software and has far less experience accommodating different environments and platforms. It has not been extensively tested on most available operating systems. While RVM continues to press forward with new features on a solid software foundation, rbenv will continue to grow and progress but will continue to ride RVM’s coattails. It’s simple to take an existing piece of open source software and apply hindsight to create a more slim, less feature packed and less tested piece of software. We wish to see rbenv grow and expand into a great piece of software that provides a different solution to ruby environment management so both projects may learn from one another.

There is one additional thing that Is available for RVM which you will not find for rbenv – Jewelery Box – the official OS X RVM GUI. As rbenv claims to be minimalistic tool it does not seam to need any GUI – but as popularity of JB shows there is a need for such tool.

In closing, I would like to mention that we know that RVM grow big and could use some refactoring to make it easier to hack on it and to make the code base more readable and maintainable – we are working on that by planning RVM2 as an SM extension – SM is another project from Wayne E. Seguin worth checking and big enough to cover few articles. We hear our users loud and clear and have big plans for the future of RVM.

As a proof of concept I created an super action that allows to process more than one action at once.
Probably most of you will ask a question ‘why’ if there is an posibility to use nested resources. Yes you can, but what if there are more different actions that do not have common root?

Code bellow is only proof of concept, it was tested only using integration tests, if you ever try it more please let me know.

First Step is to create middleware scaffold ‘lib/super_action.rb’:

class SuperAction
  include Rack::Utils

  def initialize(app)
    @app = app
  end

  def call(env)
    @app.call(env)
  end
end

Next we have to connect it, we can not do it as advised in environment file, we also do not want to change the whole stack, the way to go is to hook into application loading process soewhere after it initilaizes all automatic middlewares. Add the following code (module) into ‘config/environment.rb’:

...
require File.join(File.dirname(__FILE__), 'boot')

module Rails
  class Initializer
    alias :old_load_application_classes :load_application_classes
    def load_application_classes
      configuration.middleware.delete ActiveRecord::QueryCache
      configuration.middleware.delete ActiveRecord::ConnectionAdapters::ConnectionManagement
      configuration.middleware.insert_before ActionController::ParamsParser, 'SuperAction'
      configuration.middleware.insert_before 'SuperAction', ActiveRecord::ConnectionAdapters::ConnectionManagement
      configuration.middleware.insert_before 'SuperAction', ActiveRecord::QueryCache
      old_load_application_classes
    end
  end
end

Rails::Initializer.run do |config|
...

We can not do the middleware changes directly in initializer using config.middleware because then it raises exception NameError on ‘ActiveRecord::QueryCache’ – it is loaded in the initialization stack after environment. So this is the current stack looks now for me:

~/projects/super_action> rake middleware
use Rack::Lock
use ActionController::Failsafe
use ActionController::Session::CookieStore, #
use ActiveRecord::ConnectionAdapters::ConnectionManagement
use ActiveRecord::QueryCache
use SuperAction
use ActionController::ParamsParser
use Rack::MethodOverride
use Rack::Head
use ActionController::StringCoercion
run ActionController::Dispatcher.new

Important here was to be before ActionController::ParamsParser, because it is responsible for providing correct params to action. Putting ConnectionManagement and QueryCache before super action should be considered as an optimization.

When we have correct order of middlewares now the super action functionality, back to ‘lib/super_action.rb’:

class SuperAction
  include Rack::Utils

  def initialize(app)
    @app = app
  end

  def call(env)
    uri  = env['REQUEST_URI'] || env['PATH_INFO']
    wrapper = env['rack.input']
    body = wrapper.read
    wrapper.rewind

    unless body.blank?
      params, format = if uri == '/super_action.xml'
        [Hash.from_xml(body), :xml]
      end
    end

    first_response = nil

    if params
      response_array = params['requests'].map do |req|
        req_env = env.clone
        req_env['PATH_INFO']      = req['url']
        req_env['REQUEST_URI']    = req['url']
        req_env['REQUEST_METHOD'] = req['method']
        req_env['CONTENT_TYPE']   = req['content_type']
        req_env['rack.input']     = Rack::Lint::InputWrapper.new StringIO.new( req['body'] || '' )
        status, headers, req_response = @app.call(req_env)
        first_response ||= req_response
{
  :url => req['url'],
  :method => req['method'],
  :status => status,
  :body => req_response.body,
  :id => req['id']
}
      end

      response_body = case format
      when :xml then 
        xml = Builder::XmlMarkup.new
        xml.responses 'type' => 'array' do
          response_array.each do |resp|
            xml.response do
              xml.url resp[:url]
              xml.method resp[:method]
              xml.id resp[:id], 'type' => 'integer'
              xml.body do
                xml.cdata! resp[:body]
              end
            end
          end
        end
      end

      header = HeaderHash.new
      header['Content-Type'] = 'application/xml; charset=utf-8'
      header['Content-Length'] = response_body.size.to_s
      header['Cache-Control'] = 'private, max-age=0, must-revalidate'

      first_response.instance_variable_set(:@body, response_body)
      [ 200, header, response_body]
    else
      @app.call(env)
    end
  end
end

I know the code looks tricky, it realy is, but finally it allowed me to run the following test:

  test "update many books" do
    Book.delete_all

    start = Time.now

    xml = Builder::XmlMarkup.new
    xml.requests 'type' => 'array' do
      xml.request do
        xml.url '/books.xml'
        xml.method 'POST'
        xml.id 0
        xml.content_type 'application/xml'
        xml.body do
          xml.cdata! '<title>book2</title>'
        end
      end
      xml.request do
        xml.url '/books.xml'
        xml.method 'POST'
        xml.id 1
        xml.content_type 'application/xml'
        xml.body do
          xml.cdata! '<title>book3</title>'
        end
      end
      xml.request do
        xml.url '/books.xml'
        xml.method 'GET'
        xml.id 2
      end
    end

    assert_difference "Book.count", 2 do
      post '/super_action.xml', xml.target!
    end
    responses = Hash.from_xml( response.body )['responses']

    stop = Time.now
    puts "separate:#{stop-start}:"

y [ xml.target!, response.body ]

    assert_response :success
    assert_equal 3, responses.size
    responses.sort! { |a,b| a['id']  b['id'] }

    assert_equal 0, responses[0]['id']
    assert_equal 'book2', Hash.from_xml(responses[0]['body'])['book']['title']

    assert_equal 1, responses[1]['id']
    assert_equal 'book3', Hash.from_xml(responses[1]['body'])['book']['title']

    assert_equal 2, responses[2]['id']
    assert_equal 'book2', Hash.from_xml(responses[2]['body'])['books'][0]['title']
    assert_equal 'book3', Hash.from_xml(responses[2]['body'])['books'][1]['title']
  end

Speed up in integration tests is over 2x on three actions, this makes this something considerable … only if the code would be not so tricky.