Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Dev

Deleting a volume stuck in "attaching" state?

 

 

OpenStack dev RSS feed   Index | Next | Previous | View Threaded


lars at seas

Jun 19, 2012, 6:40 PM

Post #1 of 8 (1993 views)
Permalink
Deleting a volume stuck in "attaching" state?

I attempted to attach a volume to a running instance, but later
deleted the instance, leaving the volume stuck in the "attaching"
state:

# nova volume-list
+----+-----------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+----+-----------+--------------+------+-------------+-------------+
| 9 | attaching | None | 1 | None | |
+----+-----------+--------------+------+-------------+-------------+

It doesn't appear to be possible to delete this with "nova
volume-delete":

# nova volume-delete
nova volume-delete 9
ERROR: Invalid volume: Volume status must be available or error (HTTP 400)

Other than directly editing the database (and I've had to do that an
awful lot already), how do I recover from this situation?

--
Lars Kellogg-Stedman <lars [at] seas> |
Senior Technologist | http://ac.seas.harvard.edu/
Academic Computing | http://code.seas.harvard.edu/
Harvard School of Engineering and Applied Sciences |


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


john.griffith at solidfire

Jun 19, 2012, 9:02 PM

Post #2 of 8 (1979 views)
Permalink
Re: Deleting a volume stuck in "attaching" state? [In reply to]

On Tue, Jun 19, 2012 at 7:40 PM, Lars Kellogg-Stedman
<lars [at] seas> wrote:
> I attempted to attach a volume to a running instance, but later
> deleted the instance, leaving the volume stuck in the "attaching"
> state:
>
>  # nova volume-list
>  +----+-----------+--------------+------+-------------+-------------+
>  | ID |   Status  | Display Name | Size | Volume Type | Attached to |
>  +----+-----------+--------------+------+-------------+-------------+
>  | 9  | attaching | None         | 1    | None        |             |
>  +----+-----------+--------------+------+-------------+-------------+
>
> It doesn't appear to be possible to delete this with "nova
> volume-delete":
>
>  # nova volume-delete
>   nova volume-delete 9
>   ERROR: Invalid volume: Volume status must be available or error (HTTP 400)
>
> Other than directly editing the database (and I've had to do that an
> awful lot already), how do I recover from this situation?
>
> --
> Lars Kellogg-Stedman <lars [at] seas>       |
> Senior Technologist                                | http://ac.seas.harvard.edu/
> Academic Computing                                 | http://code.seas.harvard.edu/
> Harvard School of Engineering and Applied Sciences |
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

Hi Lars,

Unfortunately manipulating the database might be your best bet for
now. We do have plans to come up with another option in the Cinder
project, but unfortunately that won't help you much right now.

If somebody has a better method, I'm sure they'll speak up and reply
to this email, but I think right now that's your best bet.

Thanks,
John

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


duncan.thomas at hp

Jun 20, 2012, 7:30 AM

Post #3 of 8 (1983 views)
Permalink
Re: Deleting a volume stuck in "attaching" state? [In reply to]

"nova-manage volume delete" on a nova host works for this, though if the attach operation is still underway then this might cause some weirdness. If the attach cast to nova compute has timeouted out or been lost / errored then "nova-manage volume delete" should do the job.

--
Duncan Thomas
HP Cloud Services, Galway


> -----Original Message-----
> From: openstack-bounces+duncan.thomas=hp.com [at] lists
> [mailto:openstack-bounces+duncan.thomas=hp.com [at] lists] On
> Behalf Of John Griffith
> Sent: 20 June 2012 05:02
> To: Lars Kellogg-Stedman
> Cc: openstack [at] lists
> Subject: Re: [Openstack] Deleting a volume stuck in "attaching" state?
>
> On Tue, Jun 19, 2012 at 7:40 PM, Lars Kellogg-Stedman
> <lars [at] seas> wrote:
> > I attempted to attach a volume to a running instance, but later
> > deleted the instance, leaving the volume stuck in the "attaching"
> > state:
> >
> >  # nova volume-list
> >  +----+-----------+--------------+------+-------------+-------------+
> >  | ID |   Status  | Display Name | Size | Volume Type | Attached to |
> >  +----+-----------+--------------+------+-------------+-------------+
> >  | 9  | attaching | None         | 1    | None        |             |
> >  +----+-----------+--------------+------+-------------+-------------+
> >
> > It doesn't appear to be possible to delete this with "nova
> > volume-delete":
> >
> >  # nova volume-delete
> >   nova volume-delete 9
> >   ERROR: Invalid volume: Volume status must be available or error
> (HTTP 400)
> >
> > Other than directly editing the database (and I've had to do that an
> > awful lot already), how do I recover from this situation?
> >
> > --
> > Lars Kellogg-Stedman <lars [at] seas>       |
> > Senior Technologist                                |
> http://ac.seas.harvard.edu/
> > Academic Computing                                 |
> http://code.seas.harvard.edu/
> > Harvard School of Engineering and Applied Sciences |
> >
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack [at] lists
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
>
> Hi Lars,
>
> Unfortunately manipulating the database might be your best bet for
> now. We do have plans to come up with another option in the Cinder
> project, but unfortunately that won't help you much right now.
>
> If somebody has a better method, I'm sure they'll speak up and reply
> to this email, but I think right now that's your best bet.
>
> Thanks,
> John
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


yunmao at gmail

Jun 20, 2012, 7:33 AM

Post #4 of 8 (1980 views)
Permalink
Re: Deleting a volume stuck in "attaching" state? [In reply to]

John,

A strategy we are making in Nova (WIP) is to allow instance
termination no matter what. Perhaps a similar strategy could be
adopted for volumes too? Thanks,

Yun

On Wed, Jun 20, 2012 at 12:02 AM, John Griffith
<john.griffith [at] solidfire> wrote:
> On Tue, Jun 19, 2012 at 7:40 PM, Lars Kellogg-Stedman
> <lars [at] seas> wrote:
>> I attempted to attach a volume to a running instance, but later
>> deleted the instance, leaving the volume stuck in the "attaching"
>> state:
>>
>>  # nova volume-list
>>  +----+-----------+--------------+------+-------------+-------------+
>>  | ID |   Status  | Display Name | Size | Volume Type | Attached to |
>>  +----+-----------+--------------+------+-------------+-------------+
>>  | 9  | attaching | None         | 1    | None        |             |
>>  +----+-----------+--------------+------+-------------+-------------+
>>
>> It doesn't appear to be possible to delete this with "nova
>> volume-delete":
>>
>>  # nova volume-delete
>>   nova volume-delete 9
>>   ERROR: Invalid volume: Volume status must be available or error (HTTP 400)
>>
>> Other than directly editing the database (and I've had to do that an
>> awful lot already), how do I recover from this situation?
>>
>> --
>> Lars Kellogg-Stedman <lars [at] seas>       |
>> Senior Technologist                                | http://ac.seas.harvard.edu/
>> Academic Computing                                 | http://code.seas.harvard.edu/
>> Harvard School of Engineering and Applied Sciences |
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack [at] lists
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>
> Hi Lars,
>
> Unfortunately manipulating the database might be your best bet for
> now.  We do have plans to come up with another option in the Cinder
> project, but unfortunately that won't help you much right now.
>
> If somebody has a better method, I'm sure they'll speak up and reply
> to this email, but I think right now that's your best bet.
>
> Thanks,
> John
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack [at] lists
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


lars at seas

Jun 20, 2012, 8:50 AM

Post #5 of 8 (1979 views)
Permalink
Re: Deleting a volume stuck in "attaching" state? [In reply to]

On Wed, Jun 20, 2012 at 02:30:12PM +0000, Thomas, Duncan wrote:
> "nova-manage volume delete" on a nova host works for this...

Ah, that appears to do it. I wasn't previously aware that there were
volume management commands in both 'nova' and 'nova-manage'. Thanks,

--
Lars Kellogg-Stedman <lars [at] seas> |
Senior Technologist | http://ac.seas.harvard.edu/
Academic Computing | http://code.seas.harvard.edu/
Harvard School of Engineering and Applied Sciences |


_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


lars at seas

Jun 20, 2012, 8:52 AM

Post #6 of 8 (1981 views)
Permalink
Re: Deleting a volume stuck in "attaching" state? [In reply to]

> A strategy we are making in Nova (WIP) is to allow instance
> termination no matter what. Perhaps a similar strategy could be
> adopted for volumes too? Thanks,

The 'nova-manage volume delete ...' solution worked just fine in this
case...but in general, as a consumer of the software, we would really
prefer to be able to delete things regardless of their state using
established tools, rather than manipulating the database directly.
I'm always worried that I'll screw something up due to my incomplete
understanding of the database schema.

--
Lars Kellogg-Stedman <lars [at] seas> |
Senior Technologist | http://ac.seas.harvard.edu/
Academic Computing | http://code.seas.harvard.edu/
Harvard School of Engineering and Applied Sciences |

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


jaypipes at gmail

Jun 20, 2012, 10:16 AM

Post #7 of 8 (1978 views)
Permalink
Re: Deleting a volume stuck in "attaching" state? [In reply to]

On 06/20/2012 11:52 AM, Lars Kellogg-Stedman wrote:
>> A strategy we are making in Nova (WIP) is to allow instance
>> termination no matter what. Perhaps a similar strategy could be
>> adopted for volumes too? Thanks,
>
> The 'nova-manage volume delete ...' solution worked just fine in this
> case...but in general, as a consumer of the software, we would really
> prefer to be able to delete things regardless of their state using
> established tools, rather than manipulating the database directly.
> I'm always worried that I'll screw something up due to my incomplete
> understanding of the database schema.

Agreed. I think that the DELETE API calls should support a "force"
parameter that would stop any pending operation, cleanup any mess, and
delete the resource.

The fact that nova-manage volume delete works is really just because the
nova-manage tool talks directly to the database, and I know that Vish
and others are keen to have all tools speak only the REST APIs and not
any backdoor interfaces or database queries directly...

Best,
-jay

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp


cthier at gmail

Jun 20, 2012, 10:47 AM

Post #8 of 8 (1977 views)
Permalink
Re: Deleting a volume stuck in "attaching" state? [In reply to]

On Wed, Jun 20, 2012 at 12:16 PM, Jay Pipes <jaypipes [at] gmail> wrote:
> On 06/20/2012 11:52 AM, Lars Kellogg-Stedman wrote:
>>>
>>> A strategy we are making in Nova (WIP) is to allow instance
>>> termination no matter what. Perhaps a similar strategy could be
>>> adopted for volumes too? Thanks,
>>
>>
>> The 'nova-manage volume delete ...' solution worked just fine in this
>> case...but in general, as a consumer of the software, we would really
>> prefer to be able to delete things regardless of their state using
>> established tools, rather than manipulating the database directly.
>> I'm always worried that I'll screw something up due to my incomplete
>> understanding of the database schema.
>
>
> Agreed. I think that the DELETE API calls should support a "force" parameter
> that would stop any pending operation, cleanup any mess, and delete the
> resource.
>
> The fact that nova-manage volume delete works is really just because the
> nova-manage tool talks directly to the database, and I know that Vish and
> others are keen to have all tools speak only the REST APIs and not any
> backdoor interfaces or database queries directly...
>
> Best,
> -jay
>

Just as an FYI, there is a bug related to this

https://bugs.launchpad.net/nova/+bug/944383

and similar to

https://bugs.launchpad.net/nova/+bug/953594

We would like to see this functionality as well, but if it is exposed
as an api call, we need to make sure other artifacts are cleaned up as
well. The fix for the latter bug at least allows us to delete a
volume that is an error state.

In a similar vein, it would be really nice to have a force detach as
described in

https://bugs.launchpad.net/nova/+bug/944383

--
Chuck

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

OpenStack dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.