Cloud-init not finding metadata server (Openstack)

Question

I am trying to understand why my server is failing to connect to the metadata server (and thus not able to log in with the SSH key).

Similar problems online seem to be caused by Security Group problems, that I am confident is not the problem here.

It is a single host, with IP of 192.168.160.100. The guest is given a fixed IP of 10.225.134.63. The network has a neutron DHCP server at 10.225.134.64.

On the host, I have run the following commands,

# curl http://192.168.160.100:8775
1.0
...
2008-09-01
2009-04-04

# curl http://169.254.169.254
curl: (7) Failed connect to 169.254.169.254:80; No route to host

/etc/neutron/metadata_agent.ini

nova_metadata_ip = 192.168.160.100

/etc/nova/nova.conf

metadata_host=192.168.160.100

Start up log

cloud-init[827]: Cloud-init v. 0.7.5 running 'init' at Fri, 29 Sep 2017 09:50:08 +0000. Up 6.40 seconds.
cloud-init[827]: ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
cloud-init[827]: ci-info: +--------+------+---------------+---------------+-------------------+
cloud-init[827]: ci-info: | Device |  Up  |    Address    |      Mask     |     Hw-Address    |
cloud-init[827]: ci-info: +--------+------+---------------+---------------+-------------------+
cloud-init[827]: ci-info: |  lo:   | True |   127.0.0.1   |   255.0.0.0   |         .         |
cloud-init[827]: ci-info: | eth0:  | True | 10.225.134.63 | 255.255.240.0 | fa:16:3e:0b:36:43 |
cloud-init[827]: ci-info: +--------+------+---------------+---------------+-------------------+
cloud-init[827]: ci-info: ++++++++++++++++++++++++++++++++Route info+++++++++++++++++++++++++++++++++
cloud-init[827]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
cloud-init[827]: ci-info: | Route | Destination  |   Gateway    |    Genmask    | Interface | Flags |
cloud-init[827]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
cloud-init[827]: ci-info: |   0   |   0.0.0.0    | 10.225.128.1 |    0.0.0.0    |    eth0   |   UG  |
cloud-init[827]: ci-info: |   1   | 10.225.128.0 |   0.0.0.0    | 255.255.240.0 |    eth0   |   U   |
cloud-init[827]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+

cloud-init[827]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://10.225.134.64//latest/meta-data/instance-id' failed [0/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://10.225.134.64//latest/meta-data/instance-id' failed [1/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://10.225.134.64//latest/meta-data/instance-id' failed [2/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
...
cloud-init[827]: url_helper.py[WARNING]: Calling 'http://10.225.134.64//latest/meta-data/instance-id' failed [119/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
cloud-init[827]: DataSourceCloudStack.py[CRITICAL]: Giving up on waiting for the metadata from   ['http://10.225.134.64//latest/meta-data/instance-id'] after 126 seconds

192.168.160.100 and 10.225.134.63 are not on the same subnet. Why would they talk to each other? Is there a router or something between them? — bgtvfr, Oct 02 '17 at 10:50
Yes, there is a router between them. I am able to ping from 192.168.160.100 to 10.225.134.63 and SSH starts a connection from 192.168.160.100 to 10.225.134.63 (but fails at the key authorization). — Mark Perryman, Oct 02 '17 at 10:53

score 0 · Accepted Answer · answered Oct 05 '17 at 12:27

0

It turns out that some routes were missing in the Management Network namespace. They were added using the following commands.

# Where,
#  Management network ID = qdhcp-88d57d45-e9y4-4b4c-ae8a-4d5d6654e7fa
#  Management network device = tap54ff6yfd-fb
#    from `sudo ip netns exec qdhcp-88d57d45-e9y4-4b4c-ae8a-4d5d6654e7fa ip addr`
sudo ip netns exec qdhcp-88d57d45-e9y4-4b4c-ae8a-4d5d6654e7fa ip route add 169.254.0.0/16 dev  tap54ff6yfd-fb  proto kernel  scope link  src 169.254.169.254
sudo ip netns exec qdhcp-88d57d45-e9y4-4b4c-ae8a-4d5d6654e7fa ip addr add 169.254.169.254/16 brd 169.254.255.255 scope global dev tap54ff6yfd-fb

answered Oct 05 '17 at 12:27

Mark Perryman

21
1
1
5

Please can you give us more details on the solution. How did you identify the network id ( for me I have three qdhcp-*), how did you identify on which one you should run which command. Thanks. – hubert Mar 01 '21 at 05:33
Sorry, it is a long time now and I really can't remember. – Mark Perryman Mar 08 '21 at 08:07

score 0 · Answer 2 · answered Nov 29 '18 at 13:50

Even I faced the same issue so tried to reboot the Instance and after some time in the Route Ipv4 info, am able to identify the 3rd destination with which the Instance trying to take the meta-data. This is not the exact solution to the problem but a kind of alternative.

[ 131.326590] cloud-init[548]: Cloud-init v. 18.4-0ubuntu1~16.04.2 running 'init' at Thu, 29 Nov 2018 13:01:17 +0000. Up 104.95 seconds. [ 131.373908] cloud-init[548]: ci-info: +++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++ [ 131.398661] cloud-init[548]: ci-info: +--------+-------+------------------------------+---------------+--------+-------------------+ [ 131.419546] cloud-init[548]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 131.515524] cloud-init[548]: ci-info: +--------+-------+------------------------------+---------------+--------+-------------------+ [ 131.567586] cloud-init[548]: ci-info: | ens3 | True | 20.0.0.9 | 255.255.255.0 | global | fa:16:3e:fb:ba:81 | [ 131.595173] cloud-init[548]: ci-info: | ens3 | True | fe80::f816:3eff:fefb:ba81/64 | . | link | fa:16:3e:fb:ba:81 | [ 131.621358] cloud-init[548]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . | [ 131.652707] cloud-init[548]: ci-info: | lo | True | ::1/128 | . | host | . | [ 131.681617] cloud-init[548]: ci-info: | sit0 | False | . | . | . | . | [ 131.716687] cloud-init[548]: ci-info: | tunl0 | False | . | . | . | . | [ 131.747621] cloud-init[548]: ci-info: +--------+-------+------------------------------+---------------+--------+-------------------+ [ 131.828146] cloud-init[548]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++ [ 131.891154] cloud-init[548]: ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ [ 131.912601] cloud-init[548]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | [ 131.935481] cloud-init[548]: ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ [ 131.981100] cloud-init[548]: ci-info: | 0 | 0.0.0.0 | 20.0.0.1 | 0.0.0.0 | ens3 | UG | [ 132.029476] cloud-init[548]: ci-info: | 1 | 20.0.0.0 | 0.0.0.0 | 255.255.255.0 | ens3 | U | [ 132.058772] cloud-init[548]: ci-info: | 2 | 169.254.169.254 | 20.0.0.1 | 255.255.255.255 | ens3 | UGH | [ 132.097319] cloud-init[548]: ci-info: +-------+-----------------+----------+-----------------+-----------+-------+ [ 132.126704] cloud-init[548]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++ [ 132.154971] cloud-init[548]: ci-info: +-------+-------------+---------+-----------+-------+ [ 132.179131] cloud-init[548]: ci-info: | Route | Destination | Gateway | Interface | Flags | [ 132.211508] cloud-init[548]: ci-info: +-------+-------------+---------+-----------+-------+ [ 132.219367] cloud-init[548]: ci-info: | 0 | fe80::/64 | :: | ens3 | U | [ 132.259315] cloud-init[548]: ci-info: | 4 | ff00::/8 | :: | ens3 | U | [ 132.283589] cloud-init[548]: ci-info: +-------+-------------+---------+-----------+-------+ [ 132.311049] cloud-init[548]: Generating public/private rsa key pair. [ 132.318065] cloud-init[548]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key. [ 132.327114] cloud-init[548]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub. [ 132.364865] cloud-init[548]: The key fingerprint is: [ 132.378265] cloud-init[548]: SHA256:IhhdlSjJqLvj6PGiwTxA+SX7HbTHc4nSKBkqSIY3ye4 root@test1 [ 132.405438] cloud-init[548]: The key's randomart image is:

Cloud-init not finding metadata server (Openstack)

2 Answers2