Mailing List Archive: OpenStack: Operators

nova-compute won't restart (on some nodes) after Grizzly upgrade

Index | Next | Previous | View Threaded

jon at jonproulx

Aug 7, 2013, 9:36 AM

Post #1 of 3 (28 views)
Permalink

nova-compute won't restart (on some nodes) after Grizzly upgrade

Hi All,

I'm almost through my grizzly upgrade. I'd upgraded everything except
nova-compute before upgrading that (ubuntu 12.04 cloud archieve pkgs).

On most nodes the nova-compute service upgraded and restarted properly, but
on some it imediately exits with:

CRITICAL nova [-] 'instance_type_memory_mb'

It would seem like this is https://code.launchpad.net/bugs/1161022 but the
fix for that was released in March and I've verified is in the packaged
version I'm using.

The referenced bug involves the DB migration only updating non-deleted
instances in the instance-system-metatata table and the patch skips the
lookups that are broken (and irrelevant) for deleted instances.

Tracing the DB calls from the host shows it is trying to do lookups for
instances that were deleted last October, which is a bit surprising as it's
run thousands of instances since & it's not looking those up.

It is note worthy that that is around the time I upgraded from Essex ->
Folsom so it's possible their state is weirder than most having run
through that update.

There were directories for the instances in question in
/var/lib/nova/instances, so I thought "Aha!" and moved them, but on restart
I still get the same failure and same DB query for the old instances. Where
is nova getting the idea it should look these up & how can I stop it?

Suggestions for other solutions welcome as well.

Thanks,
-Jon

devendran.jagadisan at gmail

Aug 7, 2013, 6:45 PM

Post #2 of 3 (25 views)
Permalink

Re: nova-compute won't restart (on some nodes) after Grizzly upgrade [In reply to]

Hi Jonathan,

You could try to run virsh list --all on the node and check if there are
deleted instances defined.

Had the same error message during grizzly update and running virsh undefine
the deleted instances solved the issue.

Devendran

On Thu, Aug 8, 2013 at 2:36 AM, Jonathan Proulx <jon [at] jonproulx> wrote:

> Hi All,
>
> I'm almost through my grizzly upgrade. I'd upgraded everything except
> nova-compute before upgrading that (ubuntu 12.04 cloud archieve pkgs).
>
> On most nodes the nova-compute service upgraded and restarted properly,
> but on some it imediately exits with:
>
> CRITICAL nova [-] 'instance_type_memory_mb'
>
> It would seem like this is https://code.launchpad.net/bugs/1161022 but
> the fix for that was released in March and I've verified is in the packaged
> version I'm using.
>
> The referenced bug involves the DB migration only updating non-deleted
> instances in the instance-system-metatata table and the patch skips the
> lookups that are broken (and irrelevant) for deleted instances.
>
> Tracing the DB calls from the host shows it is trying to do lookups for
> instances that were deleted last October, which is a bit surprising as it's
> run thousands of instances since & it's not looking those up.
>
> It is note worthy that that is around the time I upgraded from Essex ->
> Folsom so it's possible their state is weirder than most having run
> through that update.
>
> There were directories for the instances in question in
> /var/lib/nova/instances, so I thought "Aha!" and moved them, but on restart
> I still get the same failure and same DB query for the old instances. Where
> is nova getting the idea it should look these up & how can I stop it?
>
> Suggestions for other solutions welcome as well.
>
> Thanks,
> -Jon
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators [at] lists
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>

jon at jonproulx

Aug 8, 2013, 7:26 AM

Post #3 of 3 (22 views)
Permalink

Re: nova-compute won't restart (on some nodes) after Grizzly upgrade [In reply to]

Thank you Devendran. That did it

-Jon

On Wed, Aug 7, 2013 at 9:45 PM, Devendran Jagadisan <
devendran.jagadisan [at] gmail> wrote:

> Hi Jonathan,
>
> You could try to run virsh list --all on the node and check if there are
> deleted instances defined.
>
> Had the same error message during grizzly update and running virsh undefine
> the deleted instances solved the issue.
>
> Devendran
>
>
>
> On Thu, Aug 8, 2013 at 2:36 AM, Jonathan Proulx <jon [at] jonproulx> wrote:
>
>> Hi All,
>>
>> I'm almost through my grizzly upgrade. I'd upgraded everything except
>> nova-compute before upgrading that (ubuntu 12.04 cloud archieve pkgs).
>>
>> On most nodes the nova-compute service upgraded and restarted properly,
>> but on some it imediately exits with:
>>
>> CRITICAL nova [-] 'instance_type_memory_mb'
>>
>> It would seem like this is https://code.launchpad.net/bugs/1161022 but
>> the fix for that was released in March and I've verified is in the packaged
>> version I'm using.
>>
>> The referenced bug involves the DB migration only updating non-deleted
>> instances in the instance-system-metatata table and the patch skips the
>> lookups that are broken (and irrelevant) for deleted instances.
>>
>> Tracing the DB calls from the host shows it is trying to do lookups for
>> instances that were deleted last October, which is a bit surprising as it's
>> run thousands of instances since & it's not looking those up.
>>
>> It is note worthy that that is around the time I upgraded from Essex ->
>> Folsom so it's possible their state is weirder than most having run
>> through that update.
>>
>> There were directories for the instances in question in
>> /var/lib/nova/instances, so I thought "Aha!" and moved them, but on restart
>> I still get the same failure and same DB query for the old instances. Where
>> is nova getting the idea it should look these up & how can I stop it?
>>
>> Suggestions for other solutions welcome as well.
>>
>> Thanks,
>> -Jon
>>
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators [at] lists
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads