Unable to pull community cookbook during provisioning

[jaypipes]

Hi again, Chris,

I'm working on adding Vagabond support for a stripped-down Chef repo that is used as an example Chef repo for installing OpenStack with Chef [1].

I'm running into an issue that is making me wonder whether Vagabond needs to insert/inject some HTTP proxy settings in the various nodes in order to have the nodes be able to reach external addresses.

When I run provision on the ops node, I'm getting this:

jpipes@uberbox:~/repos/stackforge/openstack-chef-repo$ bundle exec vagabond up ops --debug
ERROR: Node already exists and is running: ops
WARNING: Node already exists: ops
Vagabond: Starting node: ops
  -> STARTED
Vagabond: Provisioning node: ops
DEBUG: sudo  knife bootstrap 10.0.3.10 -d chef-full -N ops -i /opt/hw-lxc-config/id_rsa --no-host-key-verify --run-list "role[os-ops-database],role[os-ops-messaging]"  --server-url https://10.0.3.224
Bootstrapping Chef on 10.0.3.10
10.0.3.10 Starting Chef Client, version 11.4.4
10.0.3.10 resolving cookbooks for run list: ["openstack-common", "openstack-common::logging", "openstack-ops-database::server", "openstack-ops-messaging"]
10.0.3.10 Synchronizing Cookbooks:
10.0.3.10   - rabbitmq
10.0.3.10   - openstack-common
10.0.3.10   - openstack-ops-messaging
10.0.3.10   - mysql
10.0.3.10   - openstack-identity
10.0.3.10   - database
10.0.3.10   - openstack-network
10.0.3.10   - apache2
10.0.3.10   - openstack-dashboard
10.0.3.10   - yum
10.0.3.10   - build-essential
10.0.3.10   - python
10.0.3.10   - sysctl
10.0.3.10   - selinux
10.0.3.10   - openstack-image
10.0.3.10   - openstack-compute
10.0.3.10   - apt
10.0.3.10   - erlang
10.0.3.10   - iptables
10.0.3.10   - openssh
10.0.3.10   - openstack-block-storage
10.0.3.10   - postgresql
10.0.3.10   - openstack-ops-database
10.0.3.10   - xfs
10.0.3.10   - aws
10.0.3.10   - openssl
10.0.3.10 Compiling Cookbooks...
10.0.3.10 Recipe: mysql::ruby
10.0.3.10   * execute[apt-get update] action run
10.0.3.10 
10.0.3.10     - execute apt-get update
10.0.3.10 
10.0.3.10 Recipe: build-essential::debian
10.0.3.10   * execute[apt-get-update-build-essentials] action run
10.0.3.10 
10.0.3.10     - execute apt-get update
10.0.3.10 
10.0.3.10 
10.0.3.10   * package[autoconf] action install
10.0.3.10  (up to date)
10.0.3.10   * package[binutils-doc] action install
10.0.3.10  (up to date)
10.0.3.10   * package[bison] action install
10.0.3.10  (up to date)
10.0.3.10   * package[build-essential] action install
10.0.3.10  (up to date)
10.0.3.10   * package[flex] action install
10.0.3.10  (up to date)
10.0.3.10 Recipe: mysql::client
10.0.3.10   * package[mysql-client] action install
10.0.3.10  (up to date)
10.0.3.10   * package[libmysqlclient-dev] action install
10.0.3.10  (up to date)
10.0.3.10 Recipe: mysql::ruby
10.0.3.10   * chef_gem[mysql] action install
10.0.3.10 [2013-07-01T01:06:25+00:00] WARN:  failed to find gem mysql (>= 0) from [http://rubygems.org/]
10.0.3.10 
10.0.3.10 ================================================================================
10.0.3.10 Error executing action `install` on resource 'chef_gem[mysql]'
10.0.3.10 ================================================================================
10.0.3.10 
10.0.3.10 
10.0.3.10 ArgumentError
10.0.3.10 -------------
10.0.3.10 Illformed requirement [""]
10.0.3.10 
10.0.3.10 
10.0.3.10 Cookbook Trace:
10.0.3.10 ---------------
10.0.3.10 /var/chef/cache/cookbooks/mysql/recipes/ruby.rb:36:in `from_file'
10.0.3.10 /var/chef/cache/cookbooks/openstack-ops-database/recipes/mysql-client.rb:21:in `from_file'
10.0.3.10 /var/chef/cache/cookbooks/openstack-ops-database/recipes/mysql-server.rb:34:in `from_file'
10.0.3.10 /var/chef/cache/cookbooks/openstack-ops-database/recipes/server.rb:21:in `from_file'
10.0.3.10 
10.0.3.10 
10.0.3.10 
10.0.3.10 Resource Declaration:
10.0.3.10 
10.0.3.10 ---------------------
10.0.3.10 # In /var/chef/cache/cookbooks/mysql/recipes/ruby.rb
10.0.3.10 
10.0.3.10  36: chef_gem "mysql"
10.0.3.10 
10.0.3.10 
10.0.3.10 
10.0.3.10 
10.0.3.10 Compiled Resource:
10.0.3.10 ------------------
10.0.3.10 # Declared in /var/chef/cache/cookbooks/mysql/recipes/ruby.rb:36:in `from_file'
10.0.3.10 
10.0.3.10 chef_gem("mysql") do
10.0.3.10   provider Chef::Provider::Package::Rubygems
10.0.3.10   action :install
10.0.3.10   retries 0
10.0.3.10   retry_delay 2
10.0.3.10   package_name "mysql"
10.0.3.10   cookbook_name "mysql"
10.0.3.10   recipe_name "ruby"
10.0.3.10 end
10.0.3.10 
10.0.3.10 
10.0.3.10 
10.0.3.10 
10.0.3.10 ================================================================================
10.0.3.10 Recipe Compile Error in /var/chef/cache/cookbooks/openstack-ops-database/recipes/server.rb
10.0.3.10 
10.0.3.10 ================================================================================
10.0.3.10 
10.0.3.10 
10.0.3.10 ArgumentError
10.0.3.10 -------------
10.0.3.10 chef_gem[mysql] (mysql::ruby line 36) had an error: ArgumentError: Illformed requirement [""]
10.0.3.10 
10.0.3.10 
10.0.3.10 
10.0.3.10 Cookbook Trace:
10.0.3.10 
10.0.3.10 ---------------
10.0.3.10   /var/chef/cache/cookbooks/mysql/recipes/ruby.rb:36:in `from_file'
10.0.3.10   /var/chef/cache/cookbooks/openstack-ops-database/recipes/mysql-client.rb:21:in `from_file'
10.0.3.10   /var/chef/cache/cookbooks/openstack-ops-database/recipes/mysql-server.rb:34:in `from_file'
10.0.3.10   /var/chef/cache/cookbooks/openstack-ops-database/recipes/server.rb:21:in `from_file'
10.0.3.10 
10.0.3.10 
10.0.3.10 
10.0.3.10 Relevant File Content:
10.0.3.10 ----------------------
10.0.3.10 /var/chef/cache/cookbooks/mysql/recipes/ruby.rb:
10.0.3.10 
10.0.3.10  29:  include_recipe "build-essential"
10.0.3.10  30:  include_recipe "mysql::client"
10.0.3.10  31:  
10.0.3.10  32:  node['mysql']['client']['packages'].each do |mysql_pack|
10.0.3.10  33:    resources("package[#{mysql_pack}]").run_action(:install)
10.0.3.10  34:  end
10.0.3.10  35:  
10.0.3.10  36>> chef_gem "mysql"
10.0.3.10  37:  
10.0.3.10 
10.0.3.10 
10.0.3.10 [2013-07-01T01:06:25+00:00] ERROR: Running exception handlers
10.0.3.10 [2013-07-01T01:06:25+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
10.0.3.10 [2013-07-01T01:06:25+00:00] ERROR: Exception handlers complete
10.0.3.10 Chef Client failed. 2 resources updated
10.0.3.10 [2013-07-01T01:06:25+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
10.0.3.10 [2013-07-01T01:06:25+00:00] FATAL: ArgumentError: chef_gem[mysql] (mysql::ruby line 36) had an error: ArgumentError: Illformed requirement [""]
  -> PROVISION FAILED

Note that the ops node only includes a single role:

{
    :nodes => {
        :ops => {
            :template => "ubuntu_1204",
            :ipaddress => "10.0.3.10",
            :run_list => [
                "role[os-ops-database]"
            ]
        }
    },
    :clusters => {
        :simple => [
            "ops"
        ]
    },
   :local_chef_server => {
       :zero => false,
       :berkshelf => true,
       :librarian => false,
       :enabled => true,
       :auto_upload => true
   },
   :sudo => true
}

And the os-ops-database role is really just installing mysql-server in the node's runlist, and nothing more. The Berksfile for the Chef repo looks like so:

site :opscode

cookbook 'apache2', '1.6.2'
cookbook 'apt', '1.9.2'
cookbook 'aws', '0.101.0'
cookbook 'build-essential', '1.4.0'
cookbook 'database', '~> 1.4.0'
cookbook 'erlang', '1.3.0'
cookbook 'mysql', '3.0.0'
cookbook 'openssl', '1.0.2'
cookbook 'postgresql', '3.0.0'
cookbook 'rabbitmq', '~> 2.1.0'
cookbook 'xfs', '1.1.0'
cookbook 'yum', '2.2.2'

cookbook 'openstack-block-storage', github: "stackforge/cookbook-openstack-block-storage"
cookbook 'openstack-common', github: "stackforge/cookbook-openstack-common"
cookbook 'openstack-compute', github: 'stackforge/cookbook-openstack-compute'
cookbook 'openstack-dashboard', github: 'stackforge/cookbook-openstack-dashboard'
cookbook 'openstack-identity', github: 'stackforge/cookbook-openstack-identity'
cookbook 'openstack-image', github: 'stackforge/cookbook-openstack-image'
cookbook 'openstack-network', github: 'stackforge/cookbook-openstack-network'
cookbook 'openstack-object-storage', github: "stackforge/cookbook-openstack-object-storage"
cookbook 'openstack-ops-database', github: 'stackforge/cookbook-openstack-ops-database'
cookbook 'openstack-ops-messaging', github: 'stackforge/cookbook-openstack-ops-messaging'

When researching the error that appears in the chef-client output during provision, I found this:

http://tickets.opscode.com/browse/CHEF-3912

which seems to indicate that the Illformed requirement error is due to a bad HTTP proxy setting. The suggested fix is to put the following:

http_proxy = node['http_proxy']
ENV['HTTP_PROXY'] = node['http_proxy']

# run in compilation phase
if http_proxy
  Chef::Config[:http_proxy] = node['http_proxy']
  Chef::Log.info "Setting Chef http_proxy to '#{Chef::Config[:http_proxy]}'."
end

into a recipe. So, I did that. I put the above into a cookbook recipe called testing::default and uploaded the cookbook into my Berkshelf and re-uploaded the cookbook. In that testing cookbook's attributes/default.rb file, I set the default['http_proxy'] setting to '10.0.3.132' (the IP of the local chef server). I added the cookbook to my base role's runlist and uploaded the roles to the local Chef server, I then reprovisioned the ops node but a) I did not see any output that indicated the Chef::Log.info line above was run, and I continued to get the same chef-client error about Illformed requirements.

Any ideas on what might be happening? You think it's related to the http_proxy stuff, or do you think that isn't really the source of the issue?

All the best,
-jay

[1] https://github.com/stackforge/openstack-chef-repo

[chrisroberts]

With host resolution failing (#19), does this node described have an updated resolv.conf file, and can it properly resolve rubygems.org? If so, is there actually a proxy between you and the public net?

[jaypipes]

Chris, that's exactly what it was. I removed the http_proxy recipe, upload_roles'd, ssh'd to the ops node, added the 10.0.3.1 nameserver to resolv.conf and that got everything working. Closing this one out as it is a duplicate of Issue #19.

Thanks!
-jay