Jot Thought

The other day we were setting up Capistrano to tunnel through a gateway when deploying our web application. For those who don’t know about this feature of Capistrano, it allows you to deploy your web application to servers with private ip addresses behind a firewall. To configure the gateway feature, make the following changes in your deploy.rb file.

set :gateway, "gateway.esomnie.com"
role :web, "10.5.1.102"
role :app, "10.5.1.102"
role :db,  "10.5.1.202", :primary => true

The gateway server, gateway.esomnie.com, has a public ip and is accessible via ssh; while, web, app, and db server are located behind a firewall and all have private ips. Capistrano allows you to tunnel through the gateway server to deploy on the servers with private ips. After we updated the configuration file I ran cap deploy. After running deploy, we received the following error.

Errno::ECONNREFUSED
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/session.rb:89:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/session.rb:89:in `open'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/session.rb:89:in `initialize'
/usr/local/lib/ruby/1.8/timeout.rb:48:in `timeout'
/usr/local/lib/ruby/1.8/timeout.rb:76:in `timeout'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/session.rb:88:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/services.rb:121:in `new'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/transport/services.rb:121:in `register_services'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `synchronize'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/service-point.rb:122:in `instance'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:308:in `[]'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/connection/services.rb:60:in `register_services'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `synchronize'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/service-point.rb:122:in `instance'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:308:in `[]'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/service/process/services.rb:53:in `register_services'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:42:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `synchronize'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/lifecycle/singleton.rb:40:in `call'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/service-point.rb:122:in `instance'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:308:in `get'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:380:in `method_missing'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/service/services.rb:42:in `register_services'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:365:in `__send__'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:365:in `require'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/definition-context.rb:77:in `__send__'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/definition-context.rb:77:in `require'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/session.rb:130:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/needle-1.3.0/lib/needle/container.rb:107:in `define'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh/session.rb:106:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh.rb:47:in `new'
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.1.1/lib/net/ssh.rb:47:in `start'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/ssh.rb:33:in `connect'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:80:in `connect_to'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:74:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:74:in `new'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:74:in `connect_to'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `establish_connection_to'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `new'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `establish_connection_to'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `establish_connections'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `map'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `establish_connections'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:553:in `execute_on_servers'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:207:in `run'
./config/deploy.rb:96:in `load'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `instance_eval'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `before_update_code'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:155:in `send'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:155:in `update_code'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/recipes/standard.rb:145:in `load'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:430:in `transaction'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/recipes/standard.rb:144:in `load'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `instance_eval'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `update'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/recipes/standard.rb:178:in `load'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `instance_eval'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:159:in `deploy'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:268:in `send'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:268:in `execute_recipes!'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:268:in `each'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:268:in `execute_recipes!'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:239:in `execute!'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:12:in `execute!'
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/bin/cap:11
/usr/local/bin/cap:16:in `load'
/usr/local/bin/cap:16
/usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/gateway.rb:92:in `connect_to': Could not establish connection to 10.5.1.102 (RuntimeError)
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `establish_connection_to'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `initialize'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `new'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:527:in `establish_connection_to'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `establish_connections'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `map'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:520:in `establish_connections'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/actor.rb:553:in `execute_on_servers'
         ... 19 levels...
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/cli.rb:12:in `execute!'
        from /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/bin/cap:11
        from /usr/local/bin/cap:16:in `load'
        from /usr/local/bin/cap:16
*** [before_update_code] transaction: rollback
  * [before_update_code] rolling back

After debugging for a couple of hours and becoming familiar with the Capistrano code base, we found the source of the error. For this particular site, we are running ssh on a non-standard port of 2222 on the gateway. I needed to add ssh_options[:port] = 2222 to deploy.rb to accommodate the non standard port. However, this configuration option causes a bug in Capistrano when using the gateway feature and this is the cause of the failure above. In order to solve the issue, I wrote a small patch to fix the problem in Capistrano 1.4.1. If you are running into the same issue, download the patch and run the following command to patch your Capistrano 1.4.1 installation.

% cp capistrano-141diff.gz capistrano1.4.1_home
% gunzip capistrano-141diff.gz
% patch -p1 < capistrano-141.diff
patching file lib/capistrano/ssh.rb
patching file test/ssh_test.rb

That should fix the bug in Capistrano and when you run cap deploy, it should run successfully now. I have emailed Jamis Buck and will get the patch incorporated in future versions of Capistrano 1.x. I haven’t had anytime to review Capistrano 2.0 and see if this bug exists there as well.