Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Dev

metadata service problem

 

 

OpenStack dev RSS feed   Index | Next | Previous | View Threaded


xzhao at bnl

Aug 9, 2012, 7:31 PM

Post #1 of 6 (274 views)
Permalink
metadata service problem

Hello,

In my essex install on RHEL6, there is a problem with the metadata service.
The metadata service works for instances running on the controller node,
where
the nova-api(metadata service) is running. But for the other worker nodes,
the metadata service is intermittent, ie. the instances sometimes can
get its metadata,
sometime it fails with errors like:

$> curl -v http://169.254.169.254:8775/
* About to connect() to 169.254.169.254 port 8775 (#0)
* Trying 169.254.169.254... Connection timed out
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host

Any idea where should I start investigating this?

Thanks,
Xin

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


harlowja at yahoo-inc

Aug 9, 2012, 7:41 PM

Post #2 of 6 (270 views)
Permalink
Re: metadata service problem [In reply to]

I would start to check out iptables and routes that are being setup (in
vms and outside).

If you are running a flat (no dhcp) network that usually makes it a lot
harder also.

On 8/9/12 7:31 PM, "Xin Zhao" <xzhao [at] bnl> wrote:

>Hello,
>
>In my essex install on RHEL6, there is a problem with the metadata
>service.
>The metadata service works for instances running on the controller node,
>where
>the nova-api(metadata service) is running. But for the other worker nodes,
>the metadata service is intermittent, ie. the instances sometimes can
>get its metadata,
>sometime it fails with errors like:
>
>$> curl -v http://169.254.169.254:8775/
>* About to connect() to 169.254.169.254 port 8775 (#0)
>* Trying 169.254.169.254... Connection timed out
>* couldn't connect to host
>* Closing connection #0
>curl: (7) couldn't connect to host
>
>Any idea where should I start investigating this?
>
>Thanks,
>Xin
>
>_______________________________________________
>Mailing list: https://launchpad.net/~openstack
>Post to : openstack [at] lists
>Unsubscribe : https://launchpad.net/~openstack
>More help : https://help.launchpad.net/ListHelp


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


vishvananda at gmail

Aug 9, 2012, 8:17 PM

Post #3 of 6 (274 views)
Permalink
Re: metadata service problem [In reply to]

On Aug 9, 2012, at 7:31 PM, Xin Zhao <xzhao [at] bnl> wrote:

> Hello,
>
> In my essex install on RHEL6, there is a problem with the metadata service.
> The metadata service works for instances running on the controller node, where
> the nova-api(metadata service) is running. But for the other worker nodes,
> the metadata service is intermittent, ie. the instances sometimes can get its metadata,
> sometime it fails with errors like:
>
> $> curl -v http://169.254.169.254:8775/
> * About to connect() to 169.254.169.254 port 8775 (#0)
> * Trying 169.254.169.254... Connection timed out
> * couldn't connect to host
> * Closing connection #0
> curl: (7) couldn't connect to host
>
> Any idea where should I start investigating this?

This looks strange, the instance should not be putting port 8775 in the request,
it should be on port 80.

Vish
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


simon at gikaku

Aug 9, 2012, 10:42 PM

Post #4 of 6 (270 views)
Permalink
Re: metadata service problem [In reply to]

On 08/10/2012 12:17 PM, Vishvananda Ishaya wrote:
>> $> curl -v http://169.254.169.254:8775/
>> * About to connect() to 169.254.169.254 port 8775 (#0)
>> * Trying 169.254.169.254... Connection timed out
>> * couldn't connect to host
>> * Closing connection #0
>> curl: (7) couldn't connect to host
>>
>> Any idea where should I start investigating this?
>
> This looks strange, the instance should not be putting port 8775 in the request,
> it should be on port 80.
>

Generally iptables should be set up to translate 8775 to 80 on
169.254.169.254.

--
simonsmicrophone.com

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


xzhao at bnl

Aug 13, 2012, 3:03 PM

Post #5 of 6 (246 views)
Permalink
Re: metadata service problem [In reply to]

Hello,

How can I check the value of $my_ip?

I notice that in the arp table on the instance, the entry for ip
169.254.169.254 has a wrong mac address, which I don't see existent on
both the controller
node and the worker node. After I change the entry for ip
169.254.169.254 to map to the real mac address of the controller
host(metadata host as well),
the curl command works. But, I don't know why the initial wrong mac
address gets entered into the arp table in the first place?

Thanks,
Xin

On 8/10/2012 3:47 AM, tacy lee wrote:
> Suggest check iptables-save output,Metadata_host default set to
> $my_ip. if you often switch network,your $my_ip maybe set wrong
>
> On Fri, Aug 10, 2012 at 10:31 AM, Xin Zhao <xzhao [at] bnl
> <mailto:xzhao [at] bnl>> wrote:
>
> Hello,
>
> In my essex install on RHEL6, there is a problem with the metadata
> service.
> The metadata service works for instances running on the controller
> node, where
> the nova-api(metadata service) is running. But for the other
> worker nodes,
> the metadata service is intermittent, ie. the instances sometimes
> can get its metadata,
> sometime it fails with errors like:
>
> $> curl -v http://169.254.169.254:8775/
> * About to connect() to 169.254.169.254 port 8775 (#0)
> * Trying 169.254.169.254... Connection timed out
> * couldn't connect to host
> * Closing connection #0
> curl: (7) couldn't connect to host
>
> Any idea where should I start investigating this?
>
> Thanks,
> Xin
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> <https://launchpad.net/%7Eopenstack>
> Post to : openstack [at] lists
> <mailto:openstack [at] lists>
> Unsubscribe : https://launchpad.net/~openstack
> <https://launchpad.net/%7Eopenstack>
> More help : https://help.launchpad.net/ListHelp
>
>
Attachments: smime.p7s (3.14 KB)


xzhao at bnl

Aug 15, 2012, 6:24 AM

Post #6 of 6 (237 views)
Permalink
Re: metadata service problem [In reply to]

Hello,

Just an update to this issue.
I make it work by adding a DNAT iptables rule on the WNs, redirecting
connection destined to 169.254.169.254 to <controller ip>:8773.
I thought this is taken care of by openstack automatically, maybe not?
Or am I on the wrong track?

Thanks,
Xin

On 8/13/2012 6:03 PM, Xin Zhao wrote:
> Hello,
>
> How can I check the value of $my_ip?
>
> I notice that in the arp table on the instance, the entry for ip
> 169.254.169.254 has a wrong mac address, which I don't see existent on
> both the controller
> node and the worker node. After I change the entry for ip
> 169.254.169.254 to map to the real mac address of the controller
> host(metadata host as well),
> the curl command works. But, I don't know why the initial wrong mac
> address gets entered into the arp table in the first place?
>
> Thanks,
> Xin
>
> On 8/10/2012 3:47 AM, tacy lee wrote:
>> Suggest check iptables-save output,Metadata_host default set to
>> $my_ip. if you often switch network,your $my_ip maybe set wrong
>>
>> On Fri, Aug 10, 2012 at 10:31 AM, Xin Zhao <xzhao [at] bnl
>> <mailto:xzhao [at] bnl>> wrote:
>>
>> Hello,
>>
>> In my essex install on RHEL6, there is a problem with the
>> metadata service.
>> The metadata service works for instances running on the
>> controller node, where
>> the nova-api(metadata service) is running. But for the other
>> worker nodes,
>> the metadata service is intermittent, ie. the instances sometimes
>> can get its metadata,
>> sometime it fails with errors like:
>>
>> $> curl -v http://169.254.169.254:8775/
>> * About to connect() to 169.254.169.254 port 8775 (#0)
>> * Trying 169.254.169.254... Connection timed out
>> * couldn't connect to host
>> * Closing connection #0
>> curl: (7) couldn't connect to host
>>
>> Any idea where should I start investigating this?
>>
>> Thanks,
>> Xin
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> <https://launchpad.net/%7Eopenstack>
>> Post to : openstack [at] lists
>> <mailto:openstack [at] lists>
>> Unsubscribe : https://launchpad.net/~openstack
>> <https://launchpad.net/%7Eopenstack>
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
Attachments: smime.p7s (3.14 KB)

OpenStack dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.