Jot Thought

The other day we were setting up Capistrano to tunnel through a gateway when deploying our web application. For those who don’t know about this feature of Capistrano, it allows you to deploy your web application to servers with private ip addresses behind a firewall. To configure the gateway feature, make the following changes in your deploy.rb file.

set :gateway, "gateway.esomnie.com"
role :web, "10.5.1.102"
role :app, "10.5.1.102"
role :db,  "10.5.1.202", :primary => true

The gateway server, gateway.esomnie.com, has a public ip and is accessible via ssh; while, web, app, and db server are located behind a firewall and all have private ips. Capistrano allows you to tunnel through the gateway server to deploy on the servers with private ips. After we updated the configuration file I ran cap deploy. After running deploy, we received the following error.

Errno::ECONNREFUSED
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/session.rb:89:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/session.rb:89:in `open'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/session.rb:89:in `initialize'
/usr/local/lib/ruby/1.8/timeout.rb:48:in `timeout'
/usr/local/lib/ruby/1.8/timeout.rb:76:in `timeout'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/session.rb:88:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/services.rb:121:in `new'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/services.rb:121:in `register_services'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `synchronize'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/service-point.rb:122:in `instance'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:308:in `[]'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/connection/services.rb:60:in `register_services'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `synchronize'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/service-point.rb:122:in `instance'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:308:in `[]'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/service/process/services.rb:53:in `register_services'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `synchronize'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/service-point.rb:122:in `instance'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:308:in `get'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:380:in `method_missing'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/service/services.rb:42:in `register_services'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:365:in `__send__'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:365:in `require'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/definition-context.rb:77:in `__send__'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/definition-context.rb:77:in `require'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/session.rb:130:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:107:in `define'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/session.rb:106:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh.rb:47:in `new'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh.rb:47:in `start'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/ssh.rb:33:in `connect'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:80:in `connect_to'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:74:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:74:in `new'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:74:in `connect_to'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `establish_connection_to'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `new'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `establish_connection_to'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `establish_connections'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `map'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `establish_connections'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:553:in `execute_on_servers'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:207:in `run'
./config/deploy.rb:96:in `load'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `instance_eval'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `before_update_code'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:155:in `send'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:155:in `update_code'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/recipes/standard.rb:145:in `load'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:430:in `transaction'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/recipes/standard.rb:144:in `load'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `instance_eval'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `update'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/recipes/standard.rb:178:in `load'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `instance_eval'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `deploy'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:268:in `send'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:268:in `execute_recipes!'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:268:in `each'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:268:in `execute_recipes!'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:239:in `execute!'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:12:in `execute!'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/bin/cap:11
/usr/local/bin/cap:16:in `load'
/usr/local/bin/cap:16
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:92:in `connect_to': Could not establish connection to 10.5.1.102 (RuntimeError)
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `establish_connection_to'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `initialize'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `new'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `establish_connection_to'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `establish_connections'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `map'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `establish_connections'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:553:in `execute_on_servers'
         ... 19 levels...
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:12:in `execute!'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/bin/cap:11
        from /usr/local/bin/cap:16:in `load'
        from /usr/local/bin/cap:16
*** [before_update_code] transaction: rollback
  * [before_update_code] rolling back

After debugging for a couple of hours and becoming familiar with the Capistrano code base, we found the source of the error. For this particular site, we are running ssh on a non-standard port of 2222 on the gateway. I needed to add ssh_options[:port] = 2222 to deploy.rb to accommodate the non standard port. However, this configuration option causes a bug in Capistrano when using the gateway feature and this is the cause of the failure above. In order to solve the issue, I wrote a small patch to fix the problem in Capistrano 1.4.1. If you are running into the same issue, download the patch and run the following command to patch your Capistrano 1.4.1 installation.

% cp capistrano-141diff.gz capistrano1.4.1_home
% gunzip capistrano-141diff.gz
% patch -p1 < capistrano-141.diff
patching file lib/capistrano/ssh.rb
patching file test/ssh_test.rb

That should fix the bug in Capistrano and when you run cap deploy, it should run successfully now. I have emailed Jamis Buck and will get the patch incorporated in future versions of Capistrano 1.x. I haven’t had anytime to review Capistrano 2.0 and see if this bug exists there as well.

Early today I was talking with Dean, our new Web Developer, about how I started hacking in Ruby. Like many Ruby hackers these days, I started off using Rails and then became more and more interested in Ruby. When I developed my first small Rails web application it was rather apparent at how much faster I could write clean, testable code. I also immediately began to wonder, how does Rails perform all of its magic?

The natural place to start looking was in the Rails source code. I was immediately impressed at how dynamic Ruby was. After I started to grasp the code base and some of the patterns, I started to fall in love with Ruby. The power of Rails lies in Ruby. I know there are plenty of other beautiful languages out there, but my background was mainly with c, C++, C#, Java, and PHP, so some of the Ruby features were quite new to me. After learning Ruby, I realized that my whole career, I was solving problems in a box. My problem solving and design skills were constrained by the programming language at my disposal. The dynamic power of Ruby opened that box and I now tackle problems in a new light. The more Ruby code I read the more my toolset grows and the more ways I can look at solving a problem.

I keep thinking, what will the next programming language be that changes the way I think this much? Is it something I can even imagine right now? It’s strange how paradigms shape our world and make it so difficult to think outside the box.