Mailing List Archive: OpenStack: Operators

no VM "really" up after complete hardware reboot

Index | Next | Previous | View Threaded

trapni at gmail

May 24, 2012, 1:43 AM

Post #1 of 6 (317 views)
Permalink

no VM "really" up after complete hardware reboot

Hi all,

due to some circumstances I did shutdown all my OpenStack nodes
(controller, network, computes).

Now, after booting every node back up again, the dashboard and CLI command
pretend to have every VM up again,
but the truth is... NOT. No VM booted.

the /var/log/nova/nova-compute.log says, that it first wanted to shutdown
the machine (hm?) and then could not access
the bridge br102 (102 is the VLAN ID), of course, this bridge doesn't exist
yet, because the host just booted up.

Any ideas why I could not get the service back up?

Many thanks in advance,
Christian Parpart.

diego.parrilla.santamaria at gmail

May 24, 2012, 1:49 AM

Post #2 of 6 (311 views)
Permalink

Re: no VM "really" up after complete hardware reboot [In reply to]

You have two interesting parameters you can use to automatically launch the
images after reboot:

resume_guests_state_on_host_boot
start_guests_on_host_boot

Good luck!
Diego
--
Diego Parrilla
<http://www.stackops.com/>*CEO*
*www.stackops.com | * diego.parrilla [at] stackops** | +34 649 94 43 29 |
skype:diegoparrilla*
* <http://www.stackops.com/>
*

*

On Thu, May 24, 2012 at 10:43 AM, Christian Parpart <trapni [at] gmail>wrote:

> Hi all,
>
> due to some circumstances I did shutdown all my OpenStack nodes
> (controller, network, computes).
>
> Now, after booting every node back up again, the dashboard and CLI command
> pretend to have every VM up again,
> but the truth is... NOT. No VM booted.
>
> the /var/log/nova/nova-compute.log says, that it first wanted to shutdown
> the machine (hm?) and then could not access
> the bridge br102 (102 is the VLAN ID), of course, this bridge doesn't
> exist yet, because the host just booted up.
>
> Any ideas why I could not get the service back up?
>
> Many thanks in advance,
> Christian Parpart.
>
> _______________________________________________
> Openstack-operators mailing list
> Openstack-operators [at] lists
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>

trapni at gmail

May 24, 2012, 1:57 AM

Post #3 of 6 (307 views)
Permalink

Re: no VM "really" up after complete hardware reboot [In reply to]

On Thu, May 24, 2012 at 10:49 AM, Diego Parrilla Santamaría <
diego.parrilla.santamaria [at] gmail> wrote:

> You have two interesting parameters you can use to automatically launch
> the images after reboot:
>
> resume_guests_state_on_host_boot
> start_guests_on_host_boot
>

Yep, I first thought it might be something like this, too, however, the
dashboard actually pretends that they're up.
but nothing is running. The `nova list` output even says that they're
running, too, but they aren't :)
So I then tried to push the "reboot" button in the dashboard, and since
then they're in infinite "restarting" state.

This somehow also crunched my nova.projects table (again). <--- veeeery odd.

Christian.

mwjpiero at gmail

May 24, 2012, 1:59 AM

Post #4 of 6 (324 views)
Permalink

Re: no VM "really" up after complete hardware reboot [In reply to]

Try to reboot the vms or use virsh start them

-- Sent from my HP TouchPad

On 24 May 2012 16:44, Christian Parpart <trapni [at] gmail> wrote:
Hi all,
due to some circumstances I did shutdown all my OpenStack nodes (controller, network, computes).
Now, after booting every node back up again, the dashboard and CLI command pretend to have every VM up again, but the truth is... NOT. No VM booted.
the /var/log/nova/nova-compute.log says, that it first wanted to shutdown the machine (hm?) and then could not accessthe bridge br102 (102 is the VLAN ID), of course, this bridge doesn't exist yet, because the host just booted up.
Any ideas why I could not get the service back up?
Many thanks in advance,Christian Parpart.

andrew.glen-young at canonical

May 24, 2012, 2:32 AM

Post #5 of 6 (310 views)
Permalink

Re: no VM "really" up after complete hardware reboot [In reply to]

On Thu 24-05-2012 10:57 â€º, Christian Parpart wrote:
> On Thu, May 24, 2012 at 10:49 AM, Diego Parrilla SantamarÃa <
> diego.parrilla.santamaria [at] gmail> wrote:
>
> > You have two interesting parameters you can use to automatically launch
> > the images after reboot:
> >
> > resume_guests_state_on_host_boot
> > start_guests_on_host_boot
> >
>
> Yep, I first thought it might be something like this, too, however, the
> dashboard actually pretends that they're up.
> but nothing is running. The `nova list` output even says that they're
> running, too, but they aren't :)
> So I then tried to push the "reboot" button in the dashboard, and since
> then they're in infinite "restarting" state.

The running status may be a result of the 'reboot' definition in the
compute API.

The VM state is first set to active and then the VM is told to reboot.
If the VM fails to come up, the database will have the VM marked as
active even if it really is not.

nova/compute/api.py:

def reboot(self, context, instance, reboot_type):
"""Reboot the given instance."""
state = {'SOFT': task_states.REBOOTING,
'HARD': task_states.REBOOTING_HARD}[reboot_type]
self.update(context,
instance,
vm_state=vm_states.ACTIVE,
task_state=state)
self._cast_compute_message('reboot_instance', context, instance,
params={'reboot_type': reboot_type})

To get the system into a good state, I have needed to terminate the
instances. The termination throws an exception on the compute nodes, but
the VMs are no longer marked as running.

>
> This somehow also crunched my nova.projects table (again). <--- veeeery odd.
>

Due to the above explanation, I do not believe this to be related [1][2].
It is, however worrying.

[1]: http://xkcd.com/552/
[2]: http://en.wikipedia.org/wiki/Correlation_does_not_imply_causation

--
Regards.
Andrew Glen-Young
_______________________________________________
Openstack-operators mailing list
Openstack-operators [at] lists
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

trapni at gmail

May 25, 2012, 5:00 PM

Post #6 of 6 (313 views)
Permalink

Re: no VM "really" up after complete hardware reboot [In reply to]

On Thu, May 24, 2012 at 11:32 AM, Andrew Glen-Young <
andrew.glen-young [at] canonical> wrote:

> On Thu 24-05-2012 10:57 â€º, Christian Parpart wrote:
> > On Thu, May 24, 2012 at 10:49 AM, Diego Parrilla SantamarÃa <
> > diego.parrilla.santamaria [at] gmail> wrote:
> >
> > > You have two interesting parameters you can use to automatically launch
> > > the images after reboot:
> > >
> > > resume_guests_state_on_host_boot
> > > start_guests_on_host_boot
> > >
> >
> > Yep, I first thought it might be something like this, too, however, the
> > dashboard actually pretends that they're up.
> > but nothing is running. The `nova list` output even says that they're
> > running, too, but they aren't :)
> > So I then tried to push the "reboot" button in the dashboard, and since
> > then they're in infinite "restarting" state.
>
> The running status may be a result of the 'reboot' definition in the
> compute API.
>
> The VM state is first set to active and then the VM is told to reboot.
> If the VM fails to come up, the database will have the VM marked as
> active even if it really is not.
>
>
> nova/compute/api.py:
>
>
> def reboot(self, context, instance, reboot_type):
> """Reboot the given instance."""
> state = {'SOFT': task_states.REBOOTING,
> 'HARD': task_states.REBOOTING_HARD}[reboot_type]
> self.update(context,
> instance,
> vm_state=vm_states.ACTIVE,
> task_state=state)
> self._cast_compute_message('reboot_instance', context, instance,
> params={'reboot_type': reboot_type})
>
>
> To get the system into a good state, I have needed to terminate the
> instances. The termination throws an exception on the compute nodes, but
> the VMs are no longer marked as running.

Hey,

and this means, that it is a bug in Nova Compute.
Trying to follow your hint, to (fake) terminate, to reset the state, did
not work,
as it is now also in infinite Termination state.

Am I too rude to my install, or is it that Nova is not really as production
ready as the internet showed me.

Best regards,
Christian.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads