Mailing List Archive: Netapp: toasters

High CPU utilizations and file I/O

Index | Next | Previous | View Threaded

G.Milazzo at sinergy

Oct 1, 2009, 8:28 AM

Post #1 of 8 (2599 views)
Permalink

High CPU utilizations and file I/O

Hi all,

From some days the CPU of a 3020C reaches 100% slowing down all processes.
Maybe some area of a 3020HA is stressed on I/O.

There's a VMware on NFS farm using both servers and VDI also! The customer states that in the NFS volume for VMware there's an high activity of continous destruction and building of VMware vdisk...

Is there some tool that can let me understand which volume is under stress and in case move it on the other head?
Or maybe this filer is undersized for the job that has been asked...

Regards,

Dott. Giacomo Milazzo
[cid:image001.jpg [at] 01CA42BC]
Technical Account Manager
Sinergy SpA
Filiale di Roma
* 00148. viale Castello della Magliana, 38
' (+39) 3406001045 0665970252
7 +39 02 26922048
* Giacomo.Milazzo [at] Sinergy

Attachments:

image001.jpg (1.37 KB)

jeremy.page at gilbarco

Oct 1, 2009, 9:00 AM

Post #2 of 8 (2508 views)
Permalink

RE: High CPU utilizations and file I/O [In reply to]

Show stats will let you look at per volume IO.

stats show -i 15 -e volume:(volumes):(nfs)

Replace volumes with something that matches all the volume names, -e
means it's looked at as a regular expression.

Jeremy Page____________________

Systems Architect

* email:Jeremy.Page [at] gilbarco - * phone: 336.547.5399 - 6 fax:
336.547.5163 - * cell: 336.601.7274

________________________________

From: owner-toasters [at] mathworks [mailto:owner-toasters [at] mathworks]
On Behalf Of Milazzo Giacomo
Sent: Thursday, October 01, 2009 11:28 AM
To: toasters [at] mathworks
Subject: High CPU utilizations and file I/O

Hi all,

From some days the CPU of a 3020C reaches 100% slowing down all
processes.

Maybe some area of a 3020HA is stressed on I/O.

There's a VMware on NFS farm using both servers and VDI also! The
customer states that in the NFS volume for VMware there's an high
activity of continous destruction and building of VMware vdisk...

Is there some tool that can let me understand which volume is under
stress and in case move it on the other head?

Or maybe this filer is undersized for the job that has been asked...

Regards,

Dott. Giacomo Milazzo

Technical Account Manager

Sinergy SpA
Filiale di Roma
* 00148. viale Castello della Magliana, 38

' (+39) 3406001045 0665970252

7 +39 02 26922048
* Giacomo.Milazzo [at] Sinergy

Please be advised that this email may contain confidential information.
If you are not the intended recipient, please do not read, copy or
re-transmit this email. If you have received this email in error,
please notify us by email by replying to the sender and by telephone
(call us collect at +1 202-828-0850) and delete this message and any
attachments. Thank you in advance for your cooperation and assistance.

In addition, Danaher and its subsidiaries disclaim that the content of
this email constitutes an offer to enter into, or the acceptance of,
any
contract or agreement or any amendment thereto; provided that the
foregoing disclaimer does not invalidate the binding effect of any
digital or other electronic reproduction of a manual signature that is
included in any attachment to this email.

Attachments:

image001.jpg (1.37 KB)

romeotheriault at gmail

Oct 1, 2009, 8:54 PM

Post #3 of 8 (2502 views)
Permalink

Re: High CPU utilizations and file I/O [In reply to]

>
> Is there some tool that can let me understand which volume is under stress
> and in case move it on the other head?
>

I wrote a tool called "topvol" a while ago that does just this. You can find
it here:

http://communities.netapp.com/docs/DOC-1262

I also have an updated version that allows you to filter on aggregate too,
if you'd be interested.

--
Romeo Theriault
System Administrator
Information Technology Services

max.reid at saikonetworks

Oct 1, 2009, 10:16 PM

Post #4 of 8 (2501 views)
Permalink

Re: High CPU utilizations and file I/O [In reply to]

Hi Romeo,

I think everyone would be interested in the updated version!

~Max

On Oct 1, 2009, at 8:54 PM, Romeo Theriault wrote:

> Is there some tool that can let me understand which volume is under
> stress and in case move it on the other head?
>
> I wrote a tool called "topvol" a while ago that does just this. You
> can find it here:
>
> http://communities.netapp.com/docs/DOC-1262
>
> I also have an updated version that allows you to filter on
> aggregate too, if you'd be interested.
>
>
>
> --
> Romeo Theriault
> System Administrator
> Information Technology Services

romeotheriault at gmail

Oct 3, 2009, 12:35 AM

Post #5 of 8 (2473 views)
Permalink

Re: High CPU utilizations and file I/O [In reply to]

>
> I think everyone would be interested in the updated version!
>

Great! Ok, I'll put up the version that can also filter on aggregates on
Monday. I hadn't put it up earlier mainly because it takes a bit more setup
to get working. I used the netapp perl api to access the aggregate
information so it also requires a user account with the appropriate api
permissions and the associated netapp perl api modules. I just haven't
gotten around to figuring out if I can legally bundle the netapp perl
modules and just generally putting it in a easily usable package for the
general public. I'll try to figure out the details on monday and see what I
can get out, with or without the netapp modules.

--
Romeo Theriault
System Administrator
Information Technology Services

phigmov at gmail

Oct 4, 2009, 9:58 PM

Post #6 of 8 (2456 views)
Permalink

Re: High CPU utilizations and file I/O [In reply to]

As a matter of interest how much latency is to much (I know, how many
angels can you fit on a pinhead) ?

We've had a few DBA types come and comment on long SQL queue lengths
associated with disk i/o - the SQL luns are connected via iSCSI.

As far as the end user is concerned performance seems fine for things
like SharePoint and CRM. Even Exchange. However a few apps with high
i/o (mail archiving to database and SCOM which seems to thrash the
database) do seem to suffer a little in terms of slow response time.

Do people have tips for optimising iSCSI performance (targets are ESX
servers, Exchange on physical hardware and SQL on a mix of physical
and virtual servers) ?

What key counters should I keep an eye on ?

Fibre Channel is still prohibitively expensive plus 10GbE seems very
promising and more pervasive than a few years ago.

Cheers,
Raj.

On 10/3/09, Romeo Theriault <romeotheriault [at] gmail> wrote:
>>
>> I think everyone would be interested in the updated version!
>>
>
> Great! Ok, I'll put up the version that can also filter on aggregates on
> Monday. I hadn't put it up earlier mainly because it takes a bit more setup
> to get working. I used the netapp perl api to access the aggregate
> information so it also requires a user account with the appropriate api
> permissions and the associated netapp perl api modules. I just haven't
> gotten around to figuring out if I can legally bundle the netapp perl
> modules and just generally putting it in a easily usable package for the
> general public. I'll try to figure out the details on monday and see what I
> can get out, with or without the netapp modules.
>
> --
> Romeo Theriault
> System Administrator
> Information Technology Services
>

romeotheriault at gmail

Oct 5, 2009, 1:28 AM

Post #7 of 8 (2461 views)
Permalink

Re: High CPU utilizations and file I/O [In reply to]

>
> I think everyone would be interested in the updated version!
>

Ok, I've put up the newer version of topvol which allows you to optionally
filter the volumes based on aggregates. Here is an example of using topvol
without aggregate filtering:

./topvol filername 4 5 ops

and a example of filtering on an aggregate called aggrsata750. This will
only show you the volumes that are in aggrsata750.

./topvol filername:aggrsata750 4 5 ops

You can find the download here:

http://communities.netapp.com/docs/DOC-1262

Please let me know if you have any questions or comments.

Enjoy,

--
Romeo Theriault
System Administrator
Information Technology Services

phigmov at gmail

Oct 6, 2009, 1:14 PM

Post #8 of 8 (2442 views)
Permalink

Re: High CPU utilizations and file I/O [In reply to]

Thanks for all the advice - thanks to Romeo Theriaults awesome tool
I'll run some stats over time and see how things look.

I'll probably give Logicmonitor a quick look and see how it compares to DFM.

Cheers,
Raj.

On 10/7/09, Kevin M. Parker <kevin [at] theparkerz> wrote:
> Raj,
> I'll share knowledge I've received specific to NFS environments (and
> this may be specific to VMware over NFS - sorry I can't confirm). The
> way I understand it is that clients start experiencing negative
> performance after 20ms disk latency. I do not know if that same latency
> applies to iSCSI traffic.
>
> As it relates to the disks making up your aggregates, I can offer the
> following numbers as they relate to disk IOPS, which is also somewhat
> related to the original request in this thread. Numbers below are the
> IOPS that disk can sustain before experiencing a >= 20ms delay.
> FC [at] 10KRP: 120IOPS
> FC [at] 15KRP: 220IOPS
> SATA: 40IOPS
>
> I had looked on NOW for this type info but did not come up w/anything
> quickly - perhaps you could reach out to your SE.
>
> -/-
> Kevin Parker
> 919.521.8413
> http://theparkerz.com
> Blackle.com - Saving energy one search at a time.
>
>
> -----Original Message-----
> From: owner-toasters [at] mathworks [mailto:owner-toasters [at] mathworks]
> On Behalf Of Raj Patel
> Sent: Monday, October 05, 2009 12:59 AM
> To: Romeo Theriault
> Cc: Maxwell Reid; vera.lee [at] avagotech; Milazzo Giacomo;
> toasters [at] mathworks
> Subject: Re: High CPU utilizations and file I/O
>
> As a matter of interest how much latency is to much (I know, how many
> angels can you fit on a pinhead) ?
>
> We've had a few DBA types come and comment on long SQL queue lengths
> associated with disk i/o - the SQL luns are connected via iSCSI.
>
> As far as the end user is concerned performance seems fine for things
> like SharePoint and CRM. Even Exchange. However a few apps with high
> i/o (mail archiving to database and SCOM which seems to thrash the
> database) do seem to suffer a little in terms of slow response time.
>
> Do people have tips for optimising iSCSI performance (targets are ESX
> servers, Exchange on physical hardware and SQL on a mix of physical
> and virtual servers) ?
>
> What key counters should I keep an eye on ?
>
> Fibre Channel is still prohibitively expensive plus 10GbE seems very
> promising and more pervasive than a few years ago.
>
> Cheers,
> Raj.
>
> On 10/3/09, Romeo Theriault <romeotheriault [at] gmail> wrote:
>>>
>>> I think everyone would be interested in the updated version!
>>>
>>
>> Great! Ok, I'll put up the version that can also filter on aggregates
> on
>> Monday. I hadn't put it up earlier mainly because it takes a bit more
> setup
>> to get working. I used the netapp perl api to access the aggregate
>> information so it also requires a user account with the appropriate
> api
>> permissions and the associated netapp perl api modules. I just haven't
>> gotten around to figuring out if I can legally bundle the netapp perl
>> modules and just generally putting it in a easily usable package for
> the
>> general public. I'll try to figure out the details on monday and see
> what I
>> can get out, with or without the netapp modules.
>>
>> --
>> Romeo Theriault
>> System Administrator
>> Information Technology Services
>>
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads