Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Netapp: toasters

SMVI "Operation timed out" during queice / vmware-level snapshot

 

 

Netapp toasters RSS feed   Index | Next | Previous | View Threaded


kwillia at smud

May 20, 2009, 11:35 AM

Post #1 of 6 (1407 views)
Permalink
SMVI "Operation timed out" during queice / vmware-level snapshot

Anyone seeing these types of issues?

I have a FAS3070 cluster serving VMFS datastore over fiber channel to a
6 server cluster of ESX 3.5 latest SP. I seem to get a couple VMs
(randomly) that are unable to perform the VMWare snapshot and cause the
snapshot to fail. The problem seems to be hit or miss.

Here's an excerpt from the log:
2009-05-19 19:18:47,556 INFO - FLOW-11012: Operation requested retry
2009-05-19 19:18:57,681 INFO - FLOW-11006: Starting operation in
threadpool backup
2009-05-19 19:18:57,697 INFO - FLOW-11012: Operation requested retry
2009-05-19 19:19:07,822 INFO - FLOW-11006: Starting operation in
threadpool backup
2009-05-19 19:19:07,837 WARN - VMware Task "CreateSnapshot_Task" for
entity "snqfpapp01" failed with the following error: Operation timed
out.
2009-05-19 19:19:07,837 ERROR - VM "snqfpapp01" will not be backed up
since vmware snapshot create operation failed.

I have a case open with Netapp but I'd like to see if the community has
seen this.
So has anyone else seen this problem? Any suggestions?
__________________________________________________________
Ken Williams
Storage Administrator, Business Technology Operations
Sacramento Municipal Utility District
E-Mail: kwillia[at]smud.org
Phone: (916) 732-6744
Cell: (916) 240-4213


netbacker at gmail

May 20, 2009, 2:17 PM

Post #2 of 6 (1315 views)
Permalink
Re: SMVI "Operation timed out" during queice / vmware-level snapshot [In reply to]

We have seen something similar recently. Is "snqfpapp01" a Windows VM?
Check the version of VMware tools installed on that VM and
reinstall/upgrade if necessary. That fixed the issue for us.
Hope that helps

-Sto Rage

On Wed, May 20, 2009 at 11:35 AM, Ken Williams <kwillia[at]smud.org> wrote:
> Anyone seeing these types of issues?
>
> I have a FAS3070 cluster serving VMFS datastore over fiber channel to a  6
> server cluster of ESX 3.5 latest SP. I seem to get a couple VMs (randomly)
> that are unable to perform the VMWare snapshot and cause the snapshot to
> fail. The problem seems to be hit or miss.
>
> Here's an excerpt from the log:
> 2009-05-19 19:18:47,556 INFO - FLOW-11012: Operation requested retry
> 2009-05-19 19:18:57,681 INFO - FLOW-11006: Starting operation in threadpool
> backup
> 2009-05-19 19:18:57,697 INFO - FLOW-11012: Operation requested retry
> 2009-05-19 19:19:07,822 INFO - FLOW-11006: Starting operation in threadpool
> backup
> 2009-05-19 19:19:07,837 WARN - VMware Task "CreateSnapshot_Task" for entity
> "snqfpapp01" failed with the following error: Operation timed out.
> 2009-05-19 19:19:07,837 ERROR - VM "snqfpapp01" will not be backed up since
> vmware snapshot create operation failed.
>
> I have a case open with Netapp but I'd like to see if the community has seen
> this.
> So has anyone else seen this problem? Any suggestions?
> __________________________________________________________
> Ken Williams
> Storage Administrator, Business Technology Operations
> Sacramento Municipal Utility District
> E-Mail: kwillia[at]smud.org
> Phone: (916) 732-6744
> Cell: (916) 240-4213


kwillia at smud

May 21, 2009, 9:17 AM

Post #3 of 6 (1291 views)
Permalink
RE: SMVI "Operation timed out" during queice / vmware-level snapshot [In reply to]

We just moved these VMs to this datastore, They are freshly rebooted
after having the Vmtools updated...



-----Original Message-----
From: Matt Hallmark [mailto:matt[at]cosmictoaster.com]
Sent: Wednesday, May 20, 2009 4:51 PM
To: Sto Rage(c)
Cc: Ken Williams; toasters[at]mathworks.com
Subject: Re: SMVI "Operation timed out" during queice / vmware-level
snapshot

Is this a new VM, or with tools newly installed? Try rebooting the VM,
that sometimes fixes these symptoms for me.

Matt Hallmark

On May 20, 2009, at 2:17 PM, Sto Rage(c) wrote:

> We have seen something similar recently. Is "snqfpapp01" a Windows VM?
> Check the version of VMware tools installed on that VM and
> reinstall/upgrade if necessary. That fixed the issue for us.
> Hope that helps
>
> -Sto Rage
>
> On Wed, May 20, 2009 at 11:35 AM, Ken Williams <kwillia[at]smud.org>
> wrote:
>> Anyone seeing these types of issues?
>>
>> I have a FAS3070 cluster serving VMFS datastore over fiber channel to

>> a 6 server cluster of ESX 3.5 latest SP. I seem to get a couple VMs
>> (randomly)
>> that are unable to perform the VMWare snapshot and cause the snapshot

>> to fail. The problem seems to be hit or miss.
>>
>> Here's an excerpt from the log:
>> 2009-05-19 19:18:47,556 INFO - FLOW-11012: Operation requested retry
>> 2009-05-19 19:18:57,681 INFO - FLOW-11006: Starting operation in
>> threadpool backup
>> 2009-05-19 19:18:57,697 INFO - FLOW-11012: Operation requested retry
>> 2009-05-19 19:19:07,822 INFO - FLOW-11006: Starting operation in
>> threadpool backup
>> 2009-05-19 19:19:07,837 WARN - VMware Task "CreateSnapshot_Task"
>> for entity
>> "snqfpapp01" failed with the following error: Operation timed out.
>> 2009-05-19 19:19:07,837 ERROR - VM "snqfpapp01" will not be backed up

>> since vmware snapshot create operation failed.
>>
>> I have a case open with Netapp but I'd like to see if the community
>> has seen this.
>> So has anyone else seen this problem? Any suggestions?
>> __________________________________________________________
>> Ken Williams
>> Storage Administrator, Business Technology Operations Sacramento
>> Municipal Utility District
>> E-Mail: kwillia[at]smud.org
>> Phone: (916) 732-6744
>> Cell: (916) 240-4213
>


Darren.Sykes at csr

May 22, 2009, 12:39 AM

Post #4 of 6 (1292 views)
Permalink
RE: SMVI "Operation timed out" during queice / vmware-level snapshot [In reply to]

Not much help I know, but we see this issue on some VM's occasionally.

Are the failing VM's different from the rest? I seem to remember linking
our failures to VM's with iSCSI disks in the guests.

The other thing to check is that you're running the VSS driver as part
of VMWare tools (if you continually upgrade VMWare tools from older
versions as we have then you may inadvertently miss the install of VSS).


Darren


-----Original Message-----
From: owner-toasters[at]mathworks.com [mailto:owner-toasters[at]mathworks.com]
On Behalf Of Ken Williams
Sent: 21 May 2009 17:18
To: Matt Hallmark; Sto RageC
Cc: toasters[at]mathworks.com
Subject: RE: SMVI "Operation timed out" during queice / vmware-level
snapshot

We just moved these VMs to this datastore, They are freshly rebooted
after having the Vmtools updated...



-----Original Message-----
From: Matt Hallmark [mailto:matt[at]cosmictoaster.com]
Sent: Wednesday, May 20, 2009 4:51 PM
To: Sto Rage(c)
Cc: Ken Williams; toasters[at]mathworks.com
Subject: Re: SMVI "Operation timed out" during queice / vmware-level
snapshot

Is this a new VM, or with tools newly installed? Try rebooting the VM,
that sometimes fixes these symptoms for me.

Matt Hallmark

On May 20, 2009, at 2:17 PM, Sto Rage(c) wrote:

> We have seen something similar recently. Is "snqfpapp01" a Windows VM?
> Check the version of VMware tools installed on that VM and
> reinstall/upgrade if necessary. That fixed the issue for us.
> Hope that helps
>
> -Sto Rage
>
> On Wed, May 20, 2009 at 11:35 AM, Ken Williams <kwillia[at]smud.org>
> wrote:
>> Anyone seeing these types of issues?
>>
>> I have a FAS3070 cluster serving VMFS datastore over fiber channel to

>> a 6 server cluster of ESX 3.5 latest SP. I seem to get a couple VMs
>> (randomly)
>> that are unable to perform the VMWare snapshot and cause the snapshot

>> to fail. The problem seems to be hit or miss.
>>
>> Here's an excerpt from the log:
>> 2009-05-19 19:18:47,556 INFO - FLOW-11012: Operation requested retry
>> 2009-05-19 19:18:57,681 INFO - FLOW-11006: Starting operation in
>> threadpool backup
>> 2009-05-19 19:18:57,697 INFO - FLOW-11012: Operation requested retry
>> 2009-05-19 19:19:07,822 INFO - FLOW-11006: Starting operation in
>> threadpool backup
>> 2009-05-19 19:19:07,837 WARN - VMware Task "CreateSnapshot_Task"
>> for entity
>> "snqfpapp01" failed with the following error: Operation timed out.
>> 2009-05-19 19:19:07,837 ERROR - VM "snqfpapp01" will not be backed up

>> since vmware snapshot create operation failed.
>>
>> I have a case open with Netapp but I'd like to see if the community
>> has seen this.
>> So has anyone else seen this problem? Any suggestions?
>> __________________________________________________________
>> Ken Williams
>> Storage Administrator, Business Technology Operations Sacramento
>> Municipal Utility District
>> E-Mail: kwillia[at]smud.org
>> Phone: (916) 732-6744
>> Cell: (916) 240-4213
>






To report this email as spam click
https://www.mailcontrol.com/sr/qbWPulKgB9bTndxI!oX7Uu4ItyQZZf3fgvDIMwks4
!XPFJ9TGtWz1TxjQfCV8S+crFqheWj7AFz5CCz92bNxfg== .


Anupama.Kirpekar at netapp

May 22, 2009, 9:11 AM

Post #5 of 6 (1273 views)
Permalink
RE: SMVI "Operation timed out" during queice / vmware-level snapshot [In reply to]

What verions of ESX are you seeing this on? The recommendation from
Vmware is to install ESX 3.5 Update 4 in order to see reduced instances
of "Operation timed out".



-----Original Message-----
From: Darren Sykes [mailto:Darren.Sykes[at]csr.com]
Sent: Friday, May 22, 2009 12:40 AM
To: Ken Williams; Matt Hallmark; Sto RageC
Cc: toasters[at]mathworks.com
Subject: RE: SMVI "Operation timed out" during queice / vmware-level
snapshot

Not much help I know, but we see this issue on some VM's occasionally.

Are the failing VM's different from the rest? I seem to remember linking
our failures to VM's with iSCSI disks in the guests.

The other thing to check is that you're running the VSS driver as part
of VMWare tools (if you continually upgrade VMWare tools from older
versions as we have then you may inadvertently miss the install of VSS).


Darren


-----Original Message-----
From: owner-toasters[at]mathworks.com [mailto:owner-toasters[at]mathworks.com]
On Behalf Of Ken Williams
Sent: 21 May 2009 17:18
To: Matt Hallmark; Sto RageC
Cc: toasters[at]mathworks.com
Subject: RE: SMVI "Operation timed out" during queice / vmware-level
snapshot

We just moved these VMs to this datastore, They are freshly rebooted
after having the Vmtools updated...



-----Original Message-----
From: Matt Hallmark [mailto:matt[at]cosmictoaster.com]
Sent: Wednesday, May 20, 2009 4:51 PM
To: Sto Rage(c)
Cc: Ken Williams; toasters[at]mathworks.com
Subject: Re: SMVI "Operation timed out" during queice / vmware-level
snapshot

Is this a new VM, or with tools newly installed? Try rebooting the VM,
that sometimes fixes these symptoms for me.

Matt Hallmark

On May 20, 2009, at 2:17 PM, Sto Rage(c) wrote:

> We have seen something similar recently. Is "snqfpapp01" a Windows VM?
> Check the version of VMware tools installed on that VM and
> reinstall/upgrade if necessary. That fixed the issue for us.
> Hope that helps
>
> -Sto Rage
>
> On Wed, May 20, 2009 at 11:35 AM, Ken Williams <kwillia[at]smud.org>
> wrote:
>> Anyone seeing these types of issues?
>>
>> I have a FAS3070 cluster serving VMFS datastore over fiber channel to

>> a 6 server cluster of ESX 3.5 latest SP. I seem to get a couple VMs
>> (randomly)
>> that are unable to perform the VMWare snapshot and cause the snapshot

>> to fail. The problem seems to be hit or miss.
>>
>> Here's an excerpt from the log:
>> 2009-05-19 19:18:47,556 INFO - FLOW-11012: Operation requested retry
>> 2009-05-19 19:18:57,681 INFO - FLOW-11006: Starting operation in
>> threadpool backup
>> 2009-05-19 19:18:57,697 INFO - FLOW-11012: Operation requested retry
>> 2009-05-19 19:19:07,822 INFO - FLOW-11006: Starting operation in
>> threadpool backup
>> 2009-05-19 19:19:07,837 WARN - VMware Task "CreateSnapshot_Task"
>> for entity
>> "snqfpapp01" failed with the following error: Operation timed out.
>> 2009-05-19 19:19:07,837 ERROR - VM "snqfpapp01" will not be backed up

>> since vmware snapshot create operation failed.
>>
>> I have a case open with Netapp but I'd like to see if the community
>> has seen this.
>> So has anyone else seen this problem? Any suggestions?
>> __________________________________________________________
>> Ken Williams
>> Storage Administrator, Business Technology Operations Sacramento
>> Municipal Utility District
>> E-Mail: kwillia[at]smud.org
>> Phone: (916) 732-6744
>> Cell: (916) 240-4213
>






To report this email as spam click
https://www.mailcontrol.com/sr/qbWPulKgB9bTndxI!oX7Uu4ItyQZZf3fgvDIMwks4
!XPFJ9TGtWz1TxjQfCV8S+crFqheWj7AFz5CCz92bNxfg== .


kwillia at smud

May 22, 2009, 9:56 AM

Post #6 of 6 (1277 views)
Permalink
RE: SMVI "Operation timed out" during queice / vmware-level snapshot [In reply to]

Kirpekar,

Thanks! We're at ESX 3.5 Update 4 right now.

Darren,

I'll have the VMWare admins check the individual VMs for this. They were
disabling VSS a while back... It's a possibility they did not turn it
back on.

-----Original Message-----
From: Kirpekar, Anupama [mailto:Anupama.Kirpekar[at]netapp.com]
Sent: Friday, May 22, 2009 9:11 AM
To: Darren Sykes; Ken Williams; Matt Hallmark; Sto RageC
Cc: toasters[at]mathworks.com
Subject: RE: SMVI "Operation timed out" during queice / vmware-level
snapshot

What verions of ESX are you seeing this on? The recommendation from
Vmware is to install ESX 3.5 Update 4 in order to see reduced instances
of "Operation timed out".



-----Original Message-----
From: Darren Sykes [mailto:Darren.Sykes[at]csr.com]
Sent: Friday, May 22, 2009 12:40 AM
To: Ken Williams; Matt Hallmark; Sto RageC
Cc: toasters[at]mathworks.com
Subject: RE: SMVI "Operation timed out" during queice / vmware-level
snapshot

Not much help I know, but we see this issue on some VM's occasionally.

Are the failing VM's different from the rest? I seem to remember linking
our failures to VM's with iSCSI disks in the guests.

The other thing to check is that you're running the VSS driver as part
of VMWare tools (if you continually upgrade VMWare tools from older
versions as we have then you may inadvertently miss the install of VSS).


Darren


-----Original Message-----
From: owner-toasters[at]mathworks.com [mailto:owner-toasters[at]mathworks.com]
On Behalf Of Ken Williams
Sent: 21 May 2009 17:18
To: Matt Hallmark; Sto RageC
Cc: toasters[at]mathworks.com
Subject: RE: SMVI "Operation timed out" during queice / vmware-level
snapshot

We just moved these VMs to this datastore, They are freshly rebooted
after having the Vmtools updated...



-----Original Message-----
From: Matt Hallmark [mailto:matt[at]cosmictoaster.com]
Sent: Wednesday, May 20, 2009 4:51 PM
To: Sto Rage(c)
Cc: Ken Williams; toasters[at]mathworks.com
Subject: Re: SMVI "Operation timed out" during queice / vmware-level
snapshot

Is this a new VM, or with tools newly installed? Try rebooting the VM,
that sometimes fixes these symptoms for me.

Matt Hallmark

On May 20, 2009, at 2:17 PM, Sto Rage(c) wrote:

> We have seen something similar recently. Is "snqfpapp01" a Windows VM?
> Check the version of VMware tools installed on that VM and
> reinstall/upgrade if necessary. That fixed the issue for us.
> Hope that helps
>
> -Sto Rage
>
> On Wed, May 20, 2009 at 11:35 AM, Ken Williams <kwillia[at]smud.org>
> wrote:
>> Anyone seeing these types of issues?
>>
>> I have a FAS3070 cluster serving VMFS datastore over fiber channel to

>> a 6 server cluster of ESX 3.5 latest SP. I seem to get a couple VMs
>> (randomly)
>> that are unable to perform the VMWare snapshot and cause the snapshot

>> to fail. The problem seems to be hit or miss.
>>
>> Here's an excerpt from the log:
>> 2009-05-19 19:18:47,556 INFO - FLOW-11012: Operation requested retry
>> 2009-05-19 19:18:57,681 INFO - FLOW-11006: Starting operation in
>> threadpool backup
>> 2009-05-19 19:18:57,697 INFO - FLOW-11012: Operation requested retry
>> 2009-05-19 19:19:07,822 INFO - FLOW-11006: Starting operation in
>> threadpool backup
>> 2009-05-19 19:19:07,837 WARN - VMware Task "CreateSnapshot_Task"
>> for entity
>> "snqfpapp01" failed with the following error: Operation timed out.
>> 2009-05-19 19:19:07,837 ERROR - VM "snqfpapp01" will not be backed up

>> since vmware snapshot create operation failed.
>>
>> I have a case open with Netapp but I'd like to see if the community
>> has seen this.
>> So has anyone else seen this problem? Any suggestions?
>> __________________________________________________________
>> Ken Williams
>> Storage Administrator, Business Technology Operations Sacramento
>> Municipal Utility District
>> E-Mail: kwillia[at]smud.org
>> Phone: (916) 732-6744
>> Cell: (916) 240-4213
>






To report this email as spam click
https://www.mailcontrol.com/sr/qbWPulKgB9bTndxI!oX7Uu4ItyQZZf3fgvDIMwks4
!XPFJ9TGtWz1TxjQfCV8S+crFqheWj7AFz5CCz92bNxfg== .

Netapp toasters RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact lists@gossamer-threads.com
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.