Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Dev

HELP: All instances automatically rebooted.

 

 

OpenStack dev RSS feed   Index | Next | Previous | View Threaded


susltd.su at gmail

Aug 27, 2012, 5:09 PM

Post #1 of 8 (360 views)
Permalink
HELP: All instances automatically rebooted.

Hi,

I have an Essex cluster with 6 compute nodes and one control nodes. All
compute nodes are working not any interrupted, for some reason all
instances in my cluster automatically rebooted. I am trying to but not
figured out why this happened in these couple of days.

It's much appreciated if someone can give me some hints about how to deal
with this situation.

Logs in my /var/log/upstart/nova-compute.log:
http://pastebin.com/WYJtS5a5

Let me know if need more info.

TIA,
Sam


gabe.westmaas at RACKSPACE

Aug 27, 2012, 5:21 PM

Post #2 of 8 (345 views)
Permalink
Re: HELP: All instances automatically rebooted. [In reply to]

Hey Sam,

Is it possible your hypervisors restarted? I see this entry in the logs:

2012-08-23 06:35:02 INFO nova.compute.manager [req-f1598257-3f35-40e6-b5aa-d47a0e93bfba None None] [instance: ce00ff1d-cf46-44de-9557-c5a0f91c8d67] Rebooting instance after nova-compute restart.

Gabe

From: openstack-bounces+gabe.westmaas=rackspace.com [at] lists [mailto:openstack-bounces+gabe.westmaas=rackspace.com [at] lists] On Behalf Of Sam Su
Sent: Monday, August 27, 2012 8:10 PM
To: openstack
Subject: [Openstack] HELP: All instances automatically rebooted.

Hi,

I have an Essex cluster with 6 compute nodes and one control nodes. All compute nodes are working not any interrupted, for some reason all instances in my cluster automatically rebooted. I am trying to but not figured out why this happened in these couple of days.

It's much appreciated if someone can give me some hints about how to deal with this situation.

Logs in my /var/log/upstart/nova-compute.log:
http://pastebin.com/WYJtS5a5

Let me know if need more info.

TIA,
Sam


alejandro.comisario at mercadolibre

Aug 27, 2012, 5:56 PM

Post #3 of 8 (343 views)
Permalink
Re: HELP: All instances automatically rebooted. [In reply to]

One of the things i dont like in essex.
That the "autostart" flag in nova.conf with KVM doesnt work with the
autostart feature of libvirt/kvm, so if, for some reason you need to
restart nova-compute to apply some kind of modification, the instances get
soft/hard rebooted because now nova-compute handles the "autostart" flag
from nova.conf

Why is that if some one can explain ?

On Mon, Aug 27, 2012 at 9:21 PM, Gabe Westmaas
<gabe.westmaas [at] rackspace>wrote:

> Hey Sam,****
>
> ** **
>
> Is it possible your hypervisors restarted? I see this entry in the logs:*
> ***
>
> ** **
>
> 2012-08-23 06:35:02 INFO nova.compute.manager
> [req-f1598257-3f35-40e6-b5aa-d47a0e93bfba None None] [instance:
> ce00ff1d-cf46-44de-9557-c5a0f91c8d67] Rebooting instance after nova-compute
> restart.****
>
> ** **
>
> Gabe****
>
> ** **
>
> *From:* openstack-bounces+gabe.westmaas=rackspace.com [at] lists[mailto:
> openstack-bounces+gabe.westmaas=rackspace.com [at] lists] *On
> Behalf Of *Sam Su
> *Sent:* Monday, August 27, 2012 8:10 PM
> *To:* openstack
> *Subject:* [Openstack] HELP: All instances automatically rebooted.****
>
> ** **
>
> Hi, ****
>
> ** **
>
> I have an Essex cluster with 6 compute nodes and one control nodes. All
> compute nodes are working not any interrupted, for some reason all
> instances in my cluster automatically rebooted. I am trying to but not
> figured out why this happened in these couple of days. ****
>
> ** **
>
> It's much appreciated if someone can give me some hints about how to deal
> with this situation.****
>
> ** **
>
> Logs in my /var/log/upstart/nova-compute.log:****
>
> http://pastebin.com/WYJtS5a5 ****
>
> ** **
>
> Let me know if need more info.****
>
>
> TIA,****
>
> Sam****
>
> ** **
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>


--
*Alejandro*


susltd.su at gmail

Aug 27, 2012, 7:42 PM

Post #4 of 8 (343 views)
Permalink
Re: HELP: All instances automatically rebooted. [In reply to]

Thanks for your guys help!
I guess this problem may be caused by nova packages auto upgrade.

I just found these two lines in the file /var/log/kern.log:
Aug 23 06:34:33 cnode-01 kernel: [4955691.256036] init: nova-network main
process (9191) terminated with status 143
Aug 23 06:34:35 cnode-01 kernel: [4955693.402082] init: nova-compute main
process (9275) terminated with status 143

Here is the link for detail of /var/log/kern.log
http://pastebin.com/GqH1ju1R

Also found these info in /var/log/dpkg.log:
2012-08-23 06:34:33 upgrade nova-network
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012.1+stable~20120612-3ee026e-0ubuntu1.3
2012-08-23 06:34:33 status half-configured nova-network
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:33 status unpacked nova-network
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:33 status half-installed nova-network
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
2012-08-23 06:34:34 status half-installed nova-network
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
2012-08-23 06:34:34 status triggers-pending man-db 2.6.1-2
2012-08-23 06:34:34 status half-installed nova-network
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:34 status half-installed nova-network
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:34 status unpacked nova-network
2012.1+stable~20120612-3ee026e-0ubuntu1.3
2012-08-23 06:34:34 status unpacked nova-network
2012.1+stable~20120612-3ee026e-0ubuntu1.3
2012-08-23 06:34:34 upgrade nova-compute-kvm
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012.1+stable~20120612-3ee026e-0ubuntu1.3
2012-08-23 06:34:34 status half-configured nova-compute-kvm
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:35 status unpacked nova-compute-kvm
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:35 status half-installed nova-compute-kvm
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:35 status half-installed nova-compute-kvm
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012-08-23 06:34:35 status unpacked nova-compute-kvm
2012.1+stable~20120612-3ee026e-0ubuntu1.3
2012-08-23 06:34:35 status unpacked nova-compute-kvm
2012.1+stable~20120612-3ee026e-0ubuntu1.3
2012-08-23 06:34:35 upgrade nova-compute
2012.1+stable~20120612-3ee026e-0ubuntu1.2
2012.1+stable~20120612-3ee026e-0ubuntu1.3

Here is detail:
http://pastebin.com/juiSxCue

But I am not 100% sure of this. Does anyone knows what it means that a
process was terminated with status 143 in Ubuntu 12.04?

Thanks ahead,
Sam


On Mon, Aug 27, 2012 at 5:56 PM, Alejandro Comisario <
alejandro.comisario [at] mercadolibre> wrote:

> One of the things i dont like in essex.
> That the "autostart" flag in nova.conf with KVM doesnt work with the
> autostart feature of libvirt/kvm, so if, for some reason you need to
> restart nova-compute to apply some kind of modification, the instances get
> soft/hard rebooted because now nova-compute handles the "autostart" flag
> from nova.conf
>
> Why is that if some one can explain ?
>
> On Mon, Aug 27, 2012 at 9:21 PM, Gabe Westmaas <
> gabe.westmaas [at] rackspace> wrote:
>
>> Hey Sam,****
>>
>> ** **
>>
>> Is it possible your hypervisors restarted? I see this entry in the logs:
>> ****
>>
>> ** **
>>
>> 2012-08-23 06:35:02 INFO nova.compute.manager
>> [req-f1598257-3f35-40e6-b5aa-d47a0e93bfba None None] [instance:
>> ce00ff1d-cf46-44de-9557-c5a0f91c8d67] Rebooting instance after nova-compute
>> restart.****
>>
>> ** **
>>
>> Gabe****
>>
>> ** **
>>
>> *From:* openstack-bounces+gabe.westmaas=rackspace.com [at] lists[mailto:
>> openstack-bounces+gabe.westmaas=rackspace.com [at] lists] *On
>> Behalf Of *Sam Su
>> *Sent:* Monday, August 27, 2012 8:10 PM
>> *To:* openstack
>> *Subject:* [Openstack] HELP: All instances automatically rebooted.****
>>
>> ** **
>>
>> Hi, ****
>>
>> ** **
>>
>> I have an Essex cluster with 6 compute nodes and one control nodes. All
>> compute nodes are working not any interrupted, for some reason all
>> instances in my cluster automatically rebooted. I am trying to but not
>> figured out why this happened in these couple of days. ****
>>
>> ** **
>>
>> It's much appreciated if someone can give me some hints about how to deal
>> with this situation.****
>>
>> ** **
>>
>> Logs in my /var/log/upstart/nova-compute.log:****
>>
>> http://pastebin.com/WYJtS5a5 ****
>>
>> ** **
>>
>> Let me know if need more info.****
>>
>>
>> TIA,****
>>
>> Sam****
>>
>> ** **
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack [at] lists
>> Unsubscribe : https://launchpad.net/~openstack
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> *Alejandro*
>


tagliapietra.alessandro at gmail

Aug 30, 2012, 9:59 AM

Post #5 of 8 (339 views)
Permalink
Re: HELP: All instances automatically rebooted. [In reply to]

Try to disable

# start_guests_on_host_boot=false

in nova.conf, if nova-compute dies it doesn't means that the hypervisor is death.

Il giorno 28/ago/2012, alle ore 04:42, Sam Su <susltd.su [at] gmail> ha scritto:

> Thanks for your guys help!
> I guess this problem may be caused by nova packages auto upgrade.
>
> I just found these two lines in the file /var/log/kern.log:
> Aug 23 06:34:33 cnode-01 kernel: [4955691.256036] init: nova-network main process (9191) terminated with status 143
> Aug 23 06:34:35 cnode-01 kernel: [4955693.402082] init: nova-compute main process (9275) terminated with status 143
>
> Here is the link for detail of /var/log/kern.log
> http://pastebin.com/GqH1ju1R
>
> Also found these info in /var/log/dpkg.log:
> 2012-08-23 06:34:33 upgrade nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> 2012-08-23 06:34:33 status half-configured nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:33 status unpacked nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:33 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
> 2012-08-23 06:34:34 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
> 2012-08-23 06:34:34 status triggers-pending man-db 2.6.1-2
> 2012-08-23 06:34:34 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:34 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:34 status unpacked nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> 2012-08-23 06:34:34 status unpacked nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> 2012-08-23 06:34:34 upgrade nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> 2012-08-23 06:34:34 status half-configured nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:35 status unpacked nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:35 status half-installed nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:35 status half-installed nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012-08-23 06:34:35 status unpacked nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> 2012-08-23 06:34:35 status unpacked nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> 2012-08-23 06:34:35 upgrade nova-compute 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012.1+stable~20120612-3ee026e-0ubuntu1.3
>
> Here is detail:
> http://pastebin.com/juiSxCue
>
> But I am not 100% sure of this. Does anyone knows what it means that a process was terminated with status 143 in Ubuntu 12.04?
>
> Thanks ahead,
> Sam
>
>
> On Mon, Aug 27, 2012 at 5:56 PM, Alejandro Comisario <alejandro.comisario [at] mercadolibre> wrote:
> One of the things i dont like in essex.
> That the "autostart" flag in nova.conf with KVM doesnt work with the autostart feature of libvirt/kvm, so if, for some reason you need to restart nova-compute to apply some kind of modification, the instances get soft/hard rebooted because now nova-compute handles the "autostart" flag from nova.conf
>
> Why is that if some one can explain ?
>
> On Mon, Aug 27, 2012 at 9:21 PM, Gabe Westmaas <gabe.westmaas [at] rackspace> wrote:
> Hey Sam,
>
>
>
> Is it possible your hypervisors restarted? I see this entry in the logs:
>
>
>
> 2012-08-23 06:35:02 INFO nova.compute.manager [req-f1598257-3f35-40e6-b5aa-d47a0e93bfba None None] [instance: ce00ff1d-cf46-44de-9557-c5a0f91c8d67] Rebooting instance after nova-compute restart.
>
>
>
> Gabe
>
>
>
> From: openstack-bounces+gabe.westmaas=rackspace.com [at] lists [mailto:openstack-bounces+gabe.westmaas=rackspace.com [at] lists] On Behalf Of Sam Su
> Sent: Monday, August 27, 2012 8:10 PM
> To: openstack
> Subject: [Openstack] HELP: All instances automatically rebooted.
>
>
>
> Hi,
>
>
>
> I have an Essex cluster with 6 compute nodes and one control nodes. All compute nodes are working not any interrupted, for some reason all instances in my cluster automatically rebooted. I am trying to but not figured out why this happened in these couple of days.
>
>
>
> It's much appreciated if someone can give me some hints about how to deal with this situation.
>
>
>
> Logs in my /var/log/upstart/nova-compute.log:
>
> http://pastebin.com/WYJtS5a5
>
>
>
> Let me know if need more info.
>
>
> TIA,
>
> Sam
>
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>
>
>
> --
> Alejandro
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


alejandro.comisario at mercadolibre

Aug 30, 2012, 10:38 AM

Post #6 of 8 (339 views)
Permalink
Re: HELP: All instances automatically rebooted. [In reply to]

but what if still ... i want that if the compute node to autostart de vms
if for any reason, it gets rebooted ? i dont want to start them by hand (
or issue an API reboot )
Thats why i think it was great that this feature is managed ( at least in
KVM ) from the libvirt side.
*
*
*Alejandro*


On Thu, Aug 30, 2012 at 1:59 PM, Alessandro Tagliapietra <
tagliapietra.alessandro [at] gmail> wrote:

> Try to disable
>
> # start_guests_on_host_boot=false
>
> in nova.conf, if nova-compute dies it doesn't means that the hypervisor is
> death.
>
> Il giorno 28/ago/2012, alle ore 04:42, Sam Su <susltd.su [at] gmail> ha
> scritto:
>
> > Thanks for your guys help!
> > I guess this problem may be caused by nova packages auto upgrade.
> >
> > I just found these two lines in the file /var/log/kern.log:
> > Aug 23 06:34:33 cnode-01 kernel: [4955691.256036] init: nova-network
> main process (9191) terminated with status 143
> > Aug 23 06:34:35 cnode-01 kernel: [4955693.402082] init: nova-compute
> main process (9275) terminated with status 143
> >
> > Here is the link for detail of /var/log/kern.log
> > http://pastebin.com/GqH1ju1R
> >
> > Also found these info in /var/log/dpkg.log:
> > 2012-08-23 06:34:33 upgrade nova-network
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:33 status half-configured nova-network
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:33 status unpacked nova-network
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:33 status half-installed nova-network
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
> > 2012-08-23 06:34:34 status half-installed nova-network
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
> > 2012-08-23 06:34:34 status triggers-pending man-db 2.6.1-2
> > 2012-08-23 06:34:34 status half-installed nova-network
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:34 status half-installed nova-network
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:34 status unpacked nova-network
> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:34 status unpacked nova-network
> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:34 upgrade nova-compute-kvm
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:34 status half-configured nova-compute-kvm
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:35 status unpacked nova-compute-kvm
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:35 status half-installed nova-compute-kvm
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:35 status half-installed nova-compute-kvm
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:35 status unpacked nova-compute-kvm
> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:35 status unpacked nova-compute-kvm
> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:35 upgrade nova-compute
> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> >
> > Here is detail:
> > http://pastebin.com/juiSxCue
> >
> > But I am not 100% sure of this. Does anyone knows what it means that a
> process was terminated with status 143 in Ubuntu 12.04?
> >
> > Thanks ahead,
> > Sam
> >
> >
> > On Mon, Aug 27, 2012 at 5:56 PM, Alejandro Comisario <
> alejandro.comisario [at] mercadolibre> wrote:
> > One of the things i dont like in essex.
> > That the "autostart" flag in nova.conf with KVM doesnt work with the
> autostart feature of libvirt/kvm, so if, for some reason you need to
> restart nova-compute to apply some kind of modification, the instances get
> soft/hard rebooted because now nova-compute handles the "autostart" flag
> from nova.conf
> >
> > Why is that if some one can explain ?
> >
> > On Mon, Aug 27, 2012 at 9:21 PM, Gabe Westmaas <
> gabe.westmaas [at] rackspace> wrote:
> > Hey Sam,
> >
> >
> >
> > Is it possible your hypervisors restarted? I see this entry in the logs:
> >
> >
> >
> > 2012-08-23 06:35:02 INFO nova.compute.manager
> [req-f1598257-3f35-40e6-b5aa-d47a0e93bfba None None] [instance:
> ce00ff1d-cf46-44de-9557-c5a0f91c8d67] Rebooting instance after nova-compute
> restart.
> >
> >
> >
> > Gabe
> >
> >
> >
> > From: openstack-bounces+gabe.westmaas=rackspace.com [at] lists[mailto:
> openstack-bounces+gabe.westmaas=rackspace.com [at] lists] On
> Behalf Of Sam Su
> > Sent: Monday, August 27, 2012 8:10 PM
> > To: openstack
> > Subject: [Openstack] HELP: All instances automatically rebooted.
> >
> >
> >
> > Hi,
> >
> >
> >
> > I have an Essex cluster with 6 compute nodes and one control nodes. All
> compute nodes are working not any interrupted, for some reason all
> instances in my cluster automatically rebooted. I am trying to but not
> figured out why this happened in these couple of days.
> >
> >
> >
> > It's much appreciated if someone can give me some hints about how to
> deal with this situation.
> >
> >
> >
> > Logs in my /var/log/upstart/nova-compute.log:
> >
> > http://pastebin.com/WYJtS5a5
> >
> >
> >
> > Let me know if need more info.
> >
> >
> > TIA,
> >
> > Sam
> >
> >
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
> >
> >
> >
> >
> > --
> > Alejandro
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
>
>


vishvananda at gmail

Aug 30, 2012, 3:28 PM

Post #7 of 8 (332 views)
Permalink
Re: HELP: All instances automatically rebooted. [In reply to]

You want:
resume_guests_state_on_host_boot=true
start_guests_on_host_boot=false

The start_ option will essentially force a reboot every time nova is started, but the resume_ option will only attempt to reboot them if they are supposed to be running and the driver says they are not.

Vish

On Aug 30, 2012, at 10:38 AM, Alejandro Comisario <alejandro.comisario [at] mercadolibre> wrote:

> but what if still ... i want that if the compute node to autostart de vms if for any reason, it gets rebooted ? i dont want to start them by hand ( or issue an API reboot )
> Thats why i think it was great that this feature is managed ( at least in KVM ) from the libvirt side.
>
> Alejandro
>
>
> On Thu, Aug 30, 2012 at 1:59 PM, Alessandro Tagliapietra <tagliapietra.alessandro [at] gmail> wrote:
> Try to disable
>
> # start_guests_on_host_boot=false
>
> in nova.conf, if nova-compute dies it doesn't means that the hypervisor is death.
>
> Il giorno 28/ago/2012, alle ore 04:42, Sam Su <susltd.su [at] gmail> ha scritto:
>
> > Thanks for your guys help!
> > I guess this problem may be caused by nova packages auto upgrade.
> >
> > I just found these two lines in the file /var/log/kern.log:
> > Aug 23 06:34:33 cnode-01 kernel: [4955691.256036] init: nova-network main process (9191) terminated with status 143
> > Aug 23 06:34:35 cnode-01 kernel: [4955693.402082] init: nova-compute main process (9275) terminated with status 143
> >
> > Here is the link for detail of /var/log/kern.log
> > http://pastebin.com/GqH1ju1R
> >
> > Also found these info in /var/log/dpkg.log:
> > 2012-08-23 06:34:33 upgrade nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:33 status half-configured nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:33 status unpacked nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:33 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
> > 2012-08-23 06:34:34 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
> > 2012-08-23 06:34:34 status triggers-pending man-db 2.6.1-2
> > 2012-08-23 06:34:34 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:34 status half-installed nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:34 status unpacked nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:34 status unpacked nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:34 upgrade nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:34 status half-configured nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:35 status unpacked nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:35 status half-installed nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:35 status half-installed nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.2
> > 2012-08-23 06:34:35 status unpacked nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:35 status unpacked nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> > 2012-08-23 06:34:35 upgrade nova-compute 2012.1+stable~20120612-3ee026e-0ubuntu1.2 2012.1+stable~20120612-3ee026e-0ubuntu1.3
> >
> > Here is detail:
> > http://pastebin.com/juiSxCue
> >
> > But I am not 100% sure of this. Does anyone knows what it means that a process was terminated with status 143 in Ubuntu 12.04?
> >
> > Thanks ahead,
> > Sam
> >
> >
> > On Mon, Aug 27, 2012 at 5:56 PM, Alejandro Comisario <alejandro.comisario [at] mercadolibre> wrote:
> > One of the things i dont like in essex.
> > That the "autostart" flag in nova.conf with KVM doesnt work with the autostart feature of libvirt/kvm, so if, for some reason you need to restart nova-compute to apply some kind of modification, the instances get soft/hard rebooted because now nova-compute handles the "autostart" flag from nova.conf
> >
> > Why is that if some one can explain ?
> >
> > On Mon, Aug 27, 2012 at 9:21 PM, Gabe Westmaas <gabe.westmaas [at] rackspace> wrote:
> > Hey Sam,
> >
> >
> >
> > Is it possible your hypervisors restarted? I see this entry in the logs:
> >
> >
> >
> > 2012-08-23 06:35:02 INFO nova.compute.manager [req-f1598257-3f35-40e6-b5aa-d47a0e93bfba None None] [instance: ce00ff1d-cf46-44de-9557-c5a0f91c8d67] Rebooting instance after nova-compute restart.
> >
> >
> >
> > Gabe
> >
> >
> >
> > From: openstack-bounces+gabe.westmaas=rackspace.com [at] lists [mailto:openstack-bounces+gabe.westmaas=rackspace.com [at] lists] On Behalf Of Sam Su
> > Sent: Monday, August 27, 2012 8:10 PM
> > To: openstack
> > Subject: [Openstack] HELP: All instances automatically rebooted.
> >
> >
> >
> > Hi,
> >
> >
> >
> > I have an Essex cluster with 6 compute nodes and one control nodes. All compute nodes are working not any interrupted, for some reason all instances in my cluster automatically rebooted. I am trying to but not figured out why this happened in these couple of days.
> >
> >
> >
> > It's much appreciated if someone can give me some hints about how to deal with this situation.
> >
> >
> >
> > Logs in my /var/log/upstart/nova-compute.log:
> >
> > http://pastebin.com/WYJtS5a5
> >
> >
> >
> > Let me know if need more info.
> >
> >
> > TIA,
> >
> > Sam
> >
> >
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
> >
> >
> >
> >
> > --
> > Alejandro
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help : https://help.launchpad.net/ListHelp
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp


alejandro.comisario at mercadolibre

Aug 30, 2012, 4:36 PM

Post #8 of 8 (329 views)
Permalink
Re: HELP: All instances automatically rebooted. [In reply to]

Nice Vish !
I dont know why i didnt see that in the docs, my bad.

Thanks a lot!
*Alejandro*


On Thu, Aug 30, 2012 at 7:28 PM, Vishvananda Ishaya
<vishvananda [at] gmail>wrote:

> You want:
> resume_guests_state_on_host_boot=true
> start_guests_on_host_boot=false
>
> The start_ option will essentially force a reboot every time nova is
> started, but the resume_ option will only attempt to reboot them if they
> are supposed to be running and the driver says they are not.
>
> Vish
>
> On Aug 30, 2012, at 10:38 AM, Alejandro Comisario <
> alejandro.comisario [at] mercadolibre> wrote:
>
> but what if still ... i want that if the compute node to autostart de vms
> if for any reason, it gets rebooted ? i dont want to start them by hand (
> or issue an API reboot )
> Thats why i think it was great that this feature is managed ( at least in
> KVM ) from the libvirt side.
> *
> *
> *Alejandro*
>
>
> On Thu, Aug 30, 2012 at 1:59 PM, Alessandro Tagliapietra <
> tagliapietra.alessandro [at] gmail> wrote:
>
>> Try to disable
>>
>> # start_guests_on_host_boot=false
>>
>> in nova.conf, if nova-compute dies it doesn't means that the hypervisor
>> is death.
>>
>> Il giorno 28/ago/2012, alle ore 04:42, Sam Su <susltd.su [at] gmail> ha
>> scritto:
>>
>> > Thanks for your guys help!
>> > I guess this problem may be caused by nova packages auto upgrade.
>> >
>> > I just found these two lines in the file /var/log/kern.log:
>> > Aug 23 06:34:33 cnode-01 kernel: [4955691.256036] init: nova-network
>> main process (9191) terminated with status 143
>> > Aug 23 06:34:35 cnode-01 kernel: [4955693.402082] init: nova-compute
>> main process (9275) terminated with status 143
>> >
>> > Here is the link for detail of /var/log/kern.log
>> > http://pastebin.com/GqH1ju1R
>> >
>> > Also found these info in /var/log/dpkg.log:
>> > 2012-08-23 06:34:33 upgrade nova-network
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
>> > 2012-08-23 06:34:33 status half-configured nova-network
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:33 status unpacked nova-network
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:33 status half-installed nova-network
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
>> > 2012-08-23 06:34:34 status half-installed nova-network
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:34 status triggers-pending ureadahead 0.100.0-12
>> > 2012-08-23 06:34:34 status triggers-pending man-db 2.6.1-2
>> > 2012-08-23 06:34:34 status half-installed nova-network
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:34 status half-installed nova-network
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:34 status unpacked nova-network
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
>> > 2012-08-23 06:34:34 status unpacked nova-network
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
>> > 2012-08-23 06:34:34 upgrade nova-compute-kvm
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
>> > 2012-08-23 06:34:34 status half-configured nova-compute-kvm
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:35 status unpacked nova-compute-kvm
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:35 status half-installed nova-compute-kvm
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:35 status half-installed nova-compute-kvm
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> > 2012-08-23 06:34:35 status unpacked nova-compute-kvm
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
>> > 2012-08-23 06:34:35 status unpacked nova-compute-kvm
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
>> > 2012-08-23 06:34:35 upgrade nova-compute
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.2
>> 2012.1+stable~20120612-3ee026e-0ubuntu1.3
>> >
>> > Here is detail:
>> > http://pastebin.com/juiSxCue
>> >
>> > But I am not 100% sure of this. Does anyone knows what it means that a
>> process was terminated with status 143 in Ubuntu 12.04?
>> >
>> > Thanks ahead,
>> > Sam
>> >
>> >
>> > On Mon, Aug 27, 2012 at 5:56 PM, Alejandro Comisario <
>> alejandro.comisario [at] mercadolibre> wrote:
>> > One of the things i dont like in essex.
>> > That the "autostart" flag in nova.conf with KVM doesnt work with the
>> autostart feature of libvirt/kvm, so if, for some reason you need to
>> restart nova-compute to apply some kind of modification, the instances get
>> soft/hard rebooted because now nova-compute handles the "autostart" flag
>> from nova.conf
>> >
>> > Why is that if some one can explain ?
>> >
>> > On Mon, Aug 27, 2012 at 9:21 PM, Gabe Westmaas <
>> gabe.westmaas [at] rackspace> wrote:
>> > Hey Sam,
>> >
>> >
>> >
>> > Is it possible your hypervisors restarted? I see this entry in the
>> logs:
>> >
>> >
>> >
>> > 2012-08-23 06:35:02 INFO nova.compute.manager
>> [req-f1598257-3f35-40e6-b5aa-d47a0e93bfba None None] [instance:
>> ce00ff1d-cf46-44de-9557-c5a0f91c8d67] Rebooting instance after nova-compute
>> restart.
>> >
>> >
>> >
>> > Gabe
>> >
>> >
>> >
>> > From: openstack-bounces+gabe.westmaas=rackspace.com [at] lists[mailto:
>> openstack-bounces+gabe.westmaas=rackspace.com [at] lists] On
>> Behalf Of Sam Su
>> > Sent: Monday, August 27, 2012 8:10 PM
>> > To: openstack
>> > Subject: [Openstack] HELP: All instances automatically rebooted.
>> >
>> >
>> >
>> > Hi,
>> >
>> >
>> >
>> > I have an Essex cluster with 6 compute nodes and one control nodes. All
>> compute nodes are working not any interrupted, for some reason all
>> instances in my cluster automatically rebooted. I am trying to but not
>> figured out why this happened in these couple of days.
>> >
>> >
>> >
>> > It's much appreciated if someone can give me some hints about how to
>> deal with this situation.
>> >
>> >
>> >
>> > Logs in my /var/log/upstart/nova-compute.log:
>> >
>> > http://pastebin.com/WYJtS5a5
>> >
>> >
>> >
>> > Let me know if need more info.
>> >
>> >
>> > TIA,
>> >
>> > Sam
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to : openstack [at] lists
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help : https://help.launchpad.net/ListHelp
>> >
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~openstack
>> > Post to : openstack [at] lists
>> > Unsubscribe : https://launchpad.net/~openstack
>> > More help : https://help.launchpad.net/ListHelp
>>
>>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>
>

OpenStack dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.