Archive

Archive for the ‘Development’ Category

linux server memory management

February 27th, 2011 Comments off

It has been long time since last post, but finally something useful shown up, today I want to present simple script for monitoring memory in Linux.

But first happy news: our server is now sucessfully migrated to nginx, no more apache, now maximal memory usage is 20MB per process.

So here is script, it is almost simple, find some processes, and kill them if to much RAM used:

#!/bin/bash

threshold_mb=25
kill_at_mb=$((threshold_mb*5))
memory=$(free -m | awk '{if (FNR==2) print $2}')
threshold=$((threshold_mb*100/memory))
kill_at=$((kill_at_mb*100/memory))
log=/dev/shm/psaux.log
count=$(ps aux | awk -v threshold=$threshold '{if (FNR==1 || $4>=threshold) print}' | tee $log | wc -l)
[ $count -gt 1 ] && echo "Showing processes over ${threshold_mb}MB, killing processes over ${kill_at_mb}MB" && cat $log

cat $log | awk -v kill_at=$kill_at '{if ($4>kill_at) print $2" "$4 }' | while read pid mem;
do
used_mb=$(($(echo $mem | sed 's/\.//')*memory/1000))
# fix rounding problem in awk
if [ $used_mb -gt $kill_at_mb ]; then
kill -9 $pid
echo "pid $pid used ${used_mb}MB - killed"
fi
done
rm -f $log

Most important in this script are first two variables, threshold_mb which is a limit for showing processes (it is rough comparison) and kill_at_mb which is limit for killing processes, by default this limit is 5 times bigger then showing threshold. Cause killing is more important operation additional check for used memory is more strict to be sure, only processes over the limit are killed.

Save it to disk and in crontab:

sudo crontab -e

add the following lines:

MAILTO=<your @email>
* * * * * /root/bin/psaux.sh

That’s all, now add some filter in your mailbox to ignore mails form server – this is all automatic, but you might want to have this for later check if some important processes are missing.

jquery pure templates

November 1st, 2010 2 comments

This story will be about jquery pure templates.

During developing new project I found great library Pure by beebole and I was going to write about it in first place. but with time when using it I found more and more issues with it. My first idea was to fix it, and I started doing that, but in the middle of debugging I found out it’s not that easy, 20KB of sources is hard to change.

So I have decided to write my own code, using very similar idea but different approach. My library is still pure, does not require putting fancy markers into html in manner to fill it with data, it is enough to write code which can be matched by jquery. My first try was with putting selectors directly into data description like:

data = {'a#my':"my url",'a#my@href':"#/url"}

After I got proof of concept in only 3 hours of coding I thought it would be good to allow mapping any keys to a map of selectors, so it is no more required to return selectors as keys of JSON:

data2 = {'name':"my url",'link':"#/url"}

using simple map file:

map2 = {'name':'a#my','link':'a#my@href'}

So how to call it ? Simple :) just include lib in the header:

<script src="jquery-pure-template.js" type="text/javascript"
        charset="utf-8"></script>

And call render on jquery matching elements:

$('.user').render(data);

or using map file:

$('.user').render(data2,map2);

Thats all needs this html:

<div class="user"><a href="#"></a></div>

and finally you will get as output this html:

<div class="user"><a href="#/url">my url</a></div>

This was simple example, but it will also work with arrays, and nested arrays too, just give it a chance, and keep in mind don’t force html to change the flow, try to make data keep the flow, not view.

Ofcourse there might be certain problems with performance with the chosen approach, first of all performance. Arrays over 500 elements tend to perform quite bad, but using nesting arrays can hold up to 5000 elements without big problem on decent computer. Additionally putting id’s into collection should additionally give some 20% speed up in finding elements by jquery.

I know there are faster solutions, that can probably render lots of data, but mine is simple, only 100 lines now (70 in first version), this makes it extremely easy to maintain and change, hawing less code is better then more :)

This post is also available in Polish version.
Software described here is a plugin for jquery.

super action middleware

June 30th, 2010 Comments off

As a proof of concept I created an super action that allows to process more than one action at once.
Probably most of you will ask a question ‘why’ if there is an posibility to use nested resources. Yes you can, but what if there are more different actions that do not have common root?

Code bellow is only proof of concept, it was tested only using integration tests, if you ever try it more please let me know.

First Step is to create middleware scaffold ‘lib/super_action.rb’:

class SuperAction
  include Rack::Utils

  def initialize(app)
    @app = app
  end

  def call(env)
    @app.call(env)
  end
end

Next we have to connect it, we can not do it as advised in environment file, we also do not want to change the whole stack, the way to go is to hook into application loading process soewhere after it initilaizes all automatic middlewares. Add the following code (module) into ‘config/environment.rb’:

...
require File.join(File.dirname(__FILE__), 'boot')

module Rails
  class Initializer
    alias :old_load_application_classes :load_application_classes
    def load_application_classes
      configuration.middleware.delete ActiveRecord::QueryCache
      configuration.middleware.delete ActiveRecord::ConnectionAdapters::ConnectionManagement
      configuration.middleware.insert_before ActionController::ParamsParser, 'SuperAction'
      configuration.middleware.insert_before 'SuperAction', ActiveRecord::ConnectionAdapters::ConnectionManagement
      configuration.middleware.insert_before 'SuperAction', ActiveRecord::QueryCache
      old_load_application_classes
    end
  end
end

Rails::Initializer.run do |config|
...

We can not do the middleware changes directly in initializer using config.middleware because then it raises exception NameError on ‘ActiveRecord::QueryCache’ – it is loaded in the initialization stack after environment. So this is the current stack looks now for me:

~/projects/super_action> rake middleware
use Rack::Lock
use ActionController::Failsafe
use ActionController::Session::CookieStore, #
use ActiveRecord::ConnectionAdapters::ConnectionManagement
use ActiveRecord::QueryCache
use SuperAction
use ActionController::ParamsParser
use Rack::MethodOverride
use Rack::Head
use ActionController::StringCoercion
run ActionController::Dispatcher.new

Important here was to be before ActionController::ParamsParser, because it is responsible for providing correct params to action. Putting ConnectionManagement and QueryCache before super action should be considered as an optimization.

When we have correct order of middlewares now the super action functionality, back to ‘lib/super_action.rb’:

class SuperAction
  include Rack::Utils

  def initialize(app)
    @app = app
  end

  def call(env)
    uri  = env['REQUEST_URI'] || env['PATH_INFO']
    wrapper = env['rack.input']
    body = wrapper.read
    wrapper.rewind

    unless body.blank?
      params, format = if uri == '/super_action.xml'
        [Hash.from_xml(body), :xml]
      end
    end

    first_response = nil

    if params
      response_array = params['requests'].map do |req|
        req_env = env.clone
        req_env['PATH_INFO']      = req['url']
        req_env['REQUEST_URI']    = req['url']
        req_env['REQUEST_METHOD'] = req['method']
        req_env['CONTENT_TYPE']   = req['content_type']
        req_env['rack.input']     = Rack::Lint::InputWrapper.new StringIO.new( req['body'] || '' )
        status, headers, req_response = @app.call(req_env)
        first_response ||= req_response
{
  :url => req['url'],
  :method => req['method'],
  :status => status,
  :body => req_response.body,
  :id => req['id']
}
      end

      response_body = case format
      when :xml then
        xml = Builder::XmlMarkup.new
        xml.responses 'type' => 'array' do
          response_array.each do |resp|
            xml.response do
              xml.url resp[:url]
              xml.method resp[:method]
              xml.id resp[:id], 'type' => 'integer'
              xml.body do
                xml.cdata! resp[:body]
              end
            end
          end
        end
      end

      header = HeaderHash.new
      header['Content-Type'] = 'application/xml; charset=utf-8'
      header['Content-Length'] = response_body.size.to_s
      header['Cache-Control'] = 'private, max-age=0, must-revalidate'

      first_response.instance_variable_set(:@body, response_body)
      [ 200, header, response_body]
    else
      @app.call(env)
    end
  end
end

I know the code looks tricky, it realy is, but finally it allowed me to run the following test:

  test "update many books" do
    Book.delete_all

    start = Time.now

    xml = Builder::XmlMarkup.new
    xml.requests 'type' => 'array' do
      xml.request do
        xml.url '/books.xml'
        xml.method 'POST'
        xml.id 0
        xml.content_type 'application/xml'
        xml.body do
          xml.cdata! '<title>book2</title>'
        end
      end
      xml.request do
        xml.url '/books.xml'
        xml.method 'POST'
        xml.id 1
        xml.content_type 'application/xml'
        xml.body do
          xml.cdata! '<title>book3</title>'
        end
      end
      xml.request do
        xml.url '/books.xml'
        xml.method 'GET'
        xml.id 2
      end
    end

    assert_difference "Book.count", 2 do
      post '/super_action.xml', xml.target!
    end
    responses = Hash.from_xml( response.body )['responses']

    stop = Time.now
    puts "separate:#{stop-start}:"

y [ xml.target!, response.body ]

    assert_response :success
    assert_equal 3, responses.size
    responses.sort! { |a,b| a['id']  b['id'] }

    assert_equal 0, responses[0]['id']
    assert_equal 'book2', Hash.from_xml(responses[0]['body'])['book']['title']

    assert_equal 1, responses[1]['id']
    assert_equal 'book3', Hash.from_xml(responses[1]['body'])['book']['title']

    assert_equal 2, responses[2]['id']
    assert_equal 'book2', Hash.from_xml(responses[2]['body'])['books'][0]['title']
    assert_equal 'book3', Hash.from_xml(responses[2]['body'])['books'][1]['title']
  end

Speed up in integration tests is over 2x on three actions, this makes this something considerable … only if the code would be not so tricky.

my git prompt

May 26th, 2010 2 comments

After long playing around with my prompt I finally made it stable and thought it’s time to share :)

So edit your ~/.bashrc file and add following lines on the end:

shopt -s promptvars dotglob histappend no_empty_cmd_completion cdspell xpg_echo

function parse_git_dirty {
  echo -n $(git status 2>/dev/null | awk -v out=$1 -v std="dirty" '{ if ($0=="# Changes to be committed:") std = "uncommited"; last=$0 } END{ if(last!="" && last!="nothing to commit (working directory clean)") { if(out!="") print out; else print std } }')
}
function parse_git_branch {
  echo -n $(git branch --no-color 2>/dev/null | awk -v out=$1 '/^*/ { if(out=="") print $2; else print out}')
}
function parse_git_remote {
  echo -n $(git status 2>/dev/null | awk -v out=$1 '/# Your branch is / { if(out=="") print $5; else print out }')
}
export PS1='$(ppwd \l)\u@\h:\[33[33m\]\w\[33[0m\]$(parse_git_branch ":")\[33[36m\]$(parse_git_branch)\[33[0m\]$(parse_git_remote "(")\[33[35m\]$(parse_git_remote)\[33[0m\]$(parse_git_remote ")")\[33[0m\]$(parse_git_dirty  "[")\[33[31m\]$(parse_git_dirty )\[33[0m\]$(parse_git_dirty  "]")>'

I know it looks a bit complicated, unfortunately it is … this is wired bash rule that escape sequences are evaluated before evaluation of functions/variables evaluation.

Some examples of prompt using this script:

mpapis@papis:~/old_laptop/nicz-projects/content2:master> touch a
mpapis@papis:~/old_laptop/nicz-projects/content2:master[dirty]> git add .
mpapis@papis:~/old_laptop/nicz-projects/content2:master[uncommited]> git commit -m "added a file"
mpapis@papis:~/old_laptop/nicz-projects/content2:master(ahead)>git push origin master
mpapis@papis:~/old_laptop/nicz-projects/content2:master>

To make it more useful the prompt is also colored to distinguish between states of git repo.

For lazy users the script could be also replaced by very easy version, which prints git status before each prompt line (only where git is applicable):

export PS1='$(git status 2>/dev/null)\[33[0m\]\n$(ppwd \l)\u@\h:\[33[33m\]\w\[33[0m\]>'

Note: download the code from here http://niczsoft.com/files/2010/05/my-git-prompt.txt

Categories: Development, Linux Tags: , ,

Deep associations in rails activerecord

May 22nd, 2010 Comments off

Some time ago I wrote about complex associations, now time to add another method and corrections.

First the finder_by_sql, in that particular case It was necessary to add

:readonly => true

So the code looks now like this:

  has_many :roles,
    :readonly => true,
    :finder_sql =&gt; '
SELECT roles.name FROM roles
INNER JOIN responsibilities ON roles.id = responsibilities.role_id
INNER JOIN assigments ON responsibilities.group_id = assigments.group_id
WHERE assigments.user_id = #{id}
GROUP BY roles.id
  '

There is one realy big downside of using finder_sql – it does not work with find_by_… or named scopes, so this forced me to continue searching and this is the result:

  def roles
    Role.scoped(
     {
       :joins => { :responsibilities => { :group => { :assigments => :user } } },
       :conditions => {"users.id" => id},
       :group => "roles.id"
     }
   )
  end

and now I can write:

user.roles.by_name(:admin).count

where by_name is an named scope

  named_scope :by_name, lambda { |type| {:conditions => [ "roles.name = ", type.to_s ] } }
Get Adobe Flash playerPlugin by wpburn.com wordpress themes