1

For some reason in my organization we can't directly use the FQDN of the Chef server. All requests to it should be re-routed. Hence we have to do below setting inside the /etc/opscode/chef-server.rb file. 

api_fqdn "<some_other_ip>"

This worked fine until one day, no cookbooks could be uploaded on one of the Chef servers, yet all other knife commands like 'knife ssl check', 'knife ssl fetch' or 'knife cookbook list' could be successfully run.

Checked the chef-server.rb file and found two additional lines. 

bookshelf['vip'] = "https://<some_other_ip>:443"
bookshelf['external_url'] = "https://<some_other_ip>:443"

Bookshelf is an Amazon Simple Storage Service (S3)-compatible service used by Chef to store cookbooks. Some developer might did this for experimenting. The resulted exception is as below.

Uploading <some_cookbook> [0.10.12] ERROR: Server returned error 500 for https://<some_other_ip>/organizations/cobalt/sandboxes/6d8079e5b8c3bfcbf24b9fcc88020bd2, retrying 1/5 in 3s ERROR: Server returned error 500 for https://<some_other_ip>/organizations/cobalt/sandboxes/6d8079e5b8c3bfcbf24b9fcc88020bd2, retrying 2/5 in 8s ERROR: Server returned error 500 for https://<some_other_ip/organizations/cobalt/sandboxes/6d8079e5b8c3bfcbf24b9fcc88020bd2, retrying 3/5 in 9s ERROR: Server returned error 500 for https://<some_other_ip>/organizations/cobalt/sandboxes/6d8079e5b8c3bfcbf24b9fcc88020bd2, retrying 4/5 in 27s ERROR: Server returned error 500 for https://<some_other_ip>/organizations/cobalt/sandboxes/6d8079e5b8c3bfcbf24b9fcc88020bd2, retrying 5/5 in 51s ERROR: internal server error Response: internal service error

Not sure how this could affect as the settings are documented on Chef and there are such records on StackOverflow (see the accepted answer) also.

The Chef servers (12.2) are hosted by respective docker containers.

Community
  • 1
Ruifeng Ma
  • 181
  • 1
  • 5

1 Answers1

1

Further investigation found that such two additional lines caused below settings in /var/opt/opscode/opscode-erchef/etc/app.config on the Chef server. 

{s3_url, "https://<some_other_ip>"},
{s3_external_url, "https://<some_other_ip>:443"},

This is different other working Chef servers where the two lines are

{s3_url, "https://782492f20c53"},
{s3_external_url, host_header},

where 782492f20c53 is the docker container ID (serving as hostname inside the container, found as 127.0.0.1 782492f20c53 in the /etc/hosts) file.

Inspired by this, changed the bookshelf URL settings in the chef-server.rb file on the problematic Chef server as below. 

bookshelf['vip'] = "https://a0a994456729"
bookshelf['external_url'] = :host_header

Ran chef-server-ctl reconfigure and then the cookbook was successfully uploaded.

Not sure on the root cause since I am not experienced in network. Guess it's to do with request routing introduced by use of docker container.

Ruifeng Ma
  • 181
  • 1
  • 5