Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Netapp: toasters

nfsiostat and client NFS connections

 

 

Netapp toasters RSS feed   Index | Next | Previous | View Threaded


rrue at fhcrc

Feb 1, 2012, 2:33 PM

Post #1 of 11 (3378 views)
Permalink
nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not
specifically a NetApp question but if nothing else, the tool I'm using
appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and
uploading the results to a timestamped log file served on a web server so
I can reach them. I have a script that parses those pages, takes the last
timestamped entry from each file (one file for each client), and tallies
up a) the average RTT time for all mounts to the vFiler and b) the worst
single instance.

Now that I have it all working, the numbers don't believe like I think
they should. The clients are adding an entry to each file every minute. I
can see that from the timestamps before each entry. But the numbers don't
change. Or they change by a few hundredths of a ms. My gather script
returns an average that varies only by .01ms or so, and my "worst" figure
has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings
or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue



_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


andrey.borzenkov at ts

Feb 1, 2012, 7:33 PM

Post #2 of 11 (3283 views)
Permalink
RE: nfsiostat and client NFS connections [In reply to]

You do not say how you process results nor how you run nfsiostat.

nfsiostat (just like iostat, vmstat, ...) reports delta for measurement interval. The very first line is always totals since kernel boot. Usually nfsiostat is run with interval argument ("nfsiostat 60" will give you stats every minute). If you invoke it every minute you have to compute delta yourself.

There are other ways to collect information in form easier to postprocess than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or other formats (which are part of the same sysstat package), collectl (can maintain RRD database) etc.

________________________________________
From: toasters-bounces [at] teaparty [toasters-bounces [at] teaparty] On Behalf Of Randy Rue [rrue [at] fhcrc]
Sent: Thursday, February 02, 2012 02:33
To: toasters [at] teaparty
Subject: nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not
specifically a NetApp question but if nothing else, the tool I'm using
appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and
uploading the results to a timestamped log file served on a web server so
I can reach them. I have a script that parses those pages, takes the last
timestamped entry from each file (one file for each client), and tallies
up a) the average RTT time for all mounts to the vFiler and b) the worst
single instance.

Now that I have it all working, the numbers don't believe like I think
they should. The clients are adding an entry to each file every minute. I
can see that from the timestamps before each entry. But the numbers don't
change. Or they change by a few hundredths of a ms. My gather script
returns an average that varies only by .01ms or so, and my "worst" figure
has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings
or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue



_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


rrue at fhcrc

Feb 1, 2012, 8:30 PM

Post #3 of 11 (3286 views)
Permalink
Re: nfsiostat and client NFS connections [In reply to]

I appreciate your feedback.

I'm attempting to get some real-time view of current latency for NFS mounts, but have figured out that running nfsiostat returns the average since boot. Running it with multiple iterations in one call seems to return transient data, but running a single 1 second iteration doesn't.

Our ultimate goal is to capture instantaneous latency figures from several clients, post the average of those latencies and the single worst figure into an RRD database, and give a graph showing average and worst case. Can you suggest a way to do this with nfsiostat, or a better way with some other tool?



----- Original Message -----
From: "Andrey Borzenkov" <andrey.borzenkov [at] ts>
To: "Randy Rue" <rrue [at] fhcrc>, toasters [at] teaparty
Sent: Wednesday, February 1, 2012 7:33:02 PM
Subject: RE: nfsiostat and client NFS connections

You do not say how you process results nor how you run nfsiostat.

nfsiostat (just like iostat, vmstat, ...) reports delta for measurement interval. The very first line is always totals since kernel boot. Usually nfsiostat is run with interval argument ("nfsiostat 60" will give you stats every minute). If you invoke it every minute you have to compute delta yourself.

There are other ways to collect information in form easier to postprocess than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or other formats (which are part of the same sysstat package), collectl (can maintain RRD database) etc.

________________________________________
From: toasters-bounces [at] teaparty [toasters-bounces [at] teaparty] On Behalf Of Randy Rue [rrue [at] fhcrc]
Sent: Thursday, February 02, 2012 02:33
To: toasters [at] teaparty
Subject: nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not
specifically a NetApp question but if nothing else, the tool I'm using
appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and
uploading the results to a timestamped log file served on a web server so
I can reach them. I have a script that parses those pages, takes the last
timestamped entry from each file (one file for each client), and tallies
up a) the average RTT time for all mounts to the vFiler and b) the worst
single instance.

Now that I have it all working, the numbers don't believe like I think
they should. The clients are adding an entry to each file every minute. I
can see that from the timestamps before each entry. But the numbers don't
change. Or they change by a few hundredths of a ms. My gather script
returns an average that varies only by .01ms or so, and my "worst" figure
has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings
or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue



_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


speedtoys.racing at gmail

Feb 1, 2012, 8:51 PM

Post #4 of 11 (3275 views)
Permalink
Re: nfsiostat and client NFS connections [In reply to]

Hmm...what reason would you expect the line latency to be _different_ from
host to host, to the same filer?



On Wed, Feb 1, 2012 at 11:30 PM, Randy Rue <rrue [at] fhcrc> wrote:

> I appreciate your feedback.
>
> I'm attempting to get some real-time view of current latency for NFS
> mounts, but have figured out that running nfsiostat returns the average
> since boot. Running it with multiple iterations in one call seems to return
> transient data, but running a single 1 second iteration doesn't.
>
> Our ultimate goal is to capture instantaneous latency figures from several
> clients, post the average of those latencies and the single worst figure
> into an RRD database, and give a graph showing average and worst case. Can
> you suggest a way to do this with nfsiostat, or a better way with some
> other tool?
>
>
>
> ----- Original Message -----
> From: "Andrey Borzenkov" <andrey.borzenkov [at] ts>
> To: "Randy Rue" <rrue [at] fhcrc>, toasters [at] teaparty
> Sent: Wednesday, February 1, 2012 7:33:02 PM
> Subject: RE: nfsiostat and client NFS connections
>
> You do not say how you process results nor how you run nfsiostat.
>
> nfsiostat (just like iostat, vmstat, ...) reports delta for measurement
> interval. The very first line is always totals since kernel boot. Usually
> nfsiostat is run with interval argument ("nfsiostat 60" will give you stats
> every minute). If you invoke it every minute you have to compute delta
> yourself.
>
> There are other ways to collect information in form easier to postprocess
> than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or
> other formats (which are part of the same sysstat package), collectl (can
> maintain RRD database) etc.
>
> ________________________________________
> From: toasters-bounces [at] teaparty [toasters-bounces [at] teaparty] On
> Behalf Of Randy Rue [rrue [at] fhcrc]
> Sent: Thursday, February 02, 2012 02:33
> To: toasters [at] teaparty
> Subject: nfsiostat and client NFS connections
>
> Hello All,
>
> Trying to troubleshoot performance problems on our v3170. This is not
> specifically a NetApp question but if nothing else, the tool I'm using
> appears to have been written by a NetApp engineer, does that count?
>
> Running nfsiostat on a handful of impacted clients on another network and
> uploading the results to a timestamped log file served on a web server so
> I can reach them. I have a script that parses those pages, takes the last
> timestamped entry from each file (one file for each client), and tallies
> up a) the average RTT time for all mounts to the vFiler and b) the worst
> single instance.
>
> Now that I have it all working, the numbers don't believe like I think
> they should. The clients are adding an entry to each file every minute. I
> can see that from the timestamps before each entry. But the numbers don't
> change. Or they change by a few hundredths of a ms. My gather script
> returns an average that varies only by .01ms or so, and my "worst" figure
> has been steady for an hour now.
>
> We're running nfsiostat with no arguments. Am I getting real time readings
> or some average? If average, since when? Is there a way to get real-time?
>
> Hope to hear from you,
>
> Randy Rue
>
>
>
> _______________________________________________
> Toasters mailing list
> Toasters [at] teaparty
> http://www.teaparty.net/mailman/listinfo/toasters
> _______________________________________________
> Toasters mailing list
> Toasters [at] teaparty
> http://www.teaparty.net/mailman/listinfo/toasters
>



--
---
Gustatus Similis Pullus


rrue at fhcrc

Feb 1, 2012, 8:56 PM

Post #5 of 11 (3290 views)
Permalink
Re: nfsiostat and client NFS connections [In reply to]

Good question. I'm trying to resolve the concerns of a customer that is having critical performance issues they believe to be the fault of the filer. The filer is indeed struggling but from the perspective of the volumes on the vFiler they own, latencies are generally pretty low. They've agreed to also let us run some monitoring on a few of their impacted systems so we can have some indication of when they're struggling.


----- Original Message -----

From: "Jeff Mohler" <speedtoys.racing [at] gmail>
To: "Randy Rue" <rrue [at] fhcrc>
Cc: "Andrey Borzenkov" <andrey.borzenkov [at] ts>, toasters [at] teaparty
Sent: Wednesday, February 1, 2012 8:51:21 PM
Subject: Re: nfsiostat and client NFS connections

Hmm...what reason would you expect the line latency to be _different_ from host to host, to the same filer?




On Wed, Feb 1, 2012 at 11:30 PM, Randy Rue < rrue [at] fhcrc > wrote:


I appreciate your feedback.

I'm attempting to get some real-time view of current latency for NFS mounts, but have figured out that running nfsiostat returns the average since boot. Running it with multiple iterations in one call seems to return transient data, but running a single 1 second iteration doesn't.

Our ultimate goal is to capture instantaneous latency figures from several clients, post the average of those latencies and the single worst figure into an RRD database, and give a graph showing average and worst case. Can you suggest a way to do this with nfsiostat, or a better way with some other tool?






----- Original Message -----
From: "Andrey Borzenkov" < andrey.borzenkov [at] ts >
To: "Randy Rue" < rrue [at] fhcrc >, toasters [at] teaparty
Sent: Wednesday, February 1, 2012 7:33:02 PM
Subject: RE: nfsiostat and client NFS connections

You do not say how you process results nor how you run nfsiostat.

nfsiostat (just like iostat, vmstat, ...) reports delta for measurement interval. The very first line is always totals since kernel boot. Usually nfsiostat is run with interval argument ("nfsiostat 60" will give you stats every minute). If you invoke it every minute you have to compute delta yourself.

There are other ways to collect information in form easier to postprocess than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or other formats (which are part of the same sysstat package), collectl (can maintain RRD database) etc.

________________________________________
From: toasters-bounces [at] teaparty [ toasters-bounces [at] teaparty ] On Behalf Of Randy Rue [ rrue [at] fhcrc ]
Sent: Thursday, February 02, 2012 02:33
To: toasters [at] teaparty
Subject: nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not
specifically a NetApp question but if nothing else, the tool I'm using
appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and
uploading the results to a timestamped log file served on a web server so
I can reach them. I have a script that parses those pages, takes the last
timestamped entry from each file (one file for each client), and tallies
up a) the average RTT time for all mounts to the vFiler and b) the worst
single instance.

Now that I have it all working, the numbers don't believe like I think
they should. The clients are adding an entry to each file every minute. I
can see that from the timestamps before each entry. But the numbers don't
change. Or they change by a few hundredths of a ms. My gather script
returns an average that varies only by .01ms or so, and my "worst" figure
has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings
or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue



_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters





--
---
Gustatus Similis Pullus


andrey.borzenkov at ts

Feb 1, 2012, 10:08 PM

Post #6 of 11 (3277 views)
Permalink
RE: nfsiostat and client NFS connections [In reply to]

Hmm ... but nfsiostat does not really display latency, unless you have some customized version. You want nfs-iostat.py for it: http://communities.netapp.com/thread/13445


---
With best regards

Andrey Borzenkov
Senior system engineer
Service operations

-----Original Message-----
From: Randy Rue [mailto:rrue [at] fhcrc]
Sent: Thursday, February 02, 2012 8:31 AM
To: Borzenkov, Andrey
Cc: toasters [at] teaparty
Subject: Re: nfsiostat and client NFS connections

I appreciate your feedback.

I'm attempting to get some real-time view of current latency for NFS mounts, but have figured out that running nfsiostat returns the average since boot. Running it with multiple iterations in one call seems to return transient data, but running a single 1 second iteration doesn't.

Our ultimate goal is to capture instantaneous latency figures from several clients, post the average of those latencies and the single worst figure into an RRD database, and give a graph showing average and worst case. Can you suggest a way to do this with nfsiostat, or a better way with some other tool?



----- Original Message -----
From: "Andrey Borzenkov" <andrey.borzenkov [at] ts>
To: "Randy Rue" <rrue [at] fhcrc>, toasters [at] teaparty
Sent: Wednesday, February 1, 2012 7:33:02 PM
Subject: RE: nfsiostat and client NFS connections

You do not say how you process results nor how you run nfsiostat.

nfsiostat (just like iostat, vmstat, ...) reports delta for measurement interval. The very first line is always totals since kernel boot. Usually nfsiostat is run with interval argument ("nfsiostat 60" will give you stats every minute). If you invoke it every minute you have to compute delta yourself.

There are other ways to collect information in form easier to postprocess than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or other formats (which are part of the same sysstat package), collectl (can maintain RRD database) etc.

________________________________________
From: toasters-bounces [at] teaparty [toasters-bounces [at] teaparty] On Behalf Of Randy Rue [rrue [at] fhcrc]
Sent: Thursday, February 02, 2012 02:33
To: toasters [at] teaparty
Subject: nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not
specifically a NetApp question but if nothing else, the tool I'm using
appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and
uploading the results to a timestamped log file served on a web server so
I can reach them. I have a script that parses those pages, takes the last
timestamped entry from each file (one file for each client), and tallies
up a) the average RTT time for all mounts to the vFiler and b) the worst
single instance.

Now that I have it all working, the numbers don't believe like I think
they should. The clients are adding an entry to each file every minute. I
can see that from the timestamps before each entry. But the numbers don't
change. Or they change by a few hundredths of a ms. My gather script
returns an average that varies only by .01ms or so, and my "worst" figure
has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings
or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue



_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


rrue at fhcrc

Feb 1, 2012, 10:38 PM

Post #7 of 11 (3410 views)
Permalink
Re: nfsiostat and client NFS connections [In reply to]

I started with nfs-iostat.py, Realized that nfsiostat that ships with CentOS seems to be the same python script. Output is identical and if you diff the two scripts they're almost identical.

Under the impression that RTT (round trip time) is the same as latency.

But in either case the RTT figures given are averaged since the last reboot if you run it with no arguments, and if you run it with arguments of "1 1" it's not clear what you're actually getting.

I'm looking for something I can run once a minute that will tell me the latency of the NFS mounts in real time.

----- Original Message -----
From: "Andrey Borzenkov" <andrey.borzenkov [at] ts>
To: "Randy Rue" <rrue [at] fhcrc>
Cc: toasters [at] teaparty
Sent: Wednesday, February 1, 2012 10:08:14 PM
Subject: RE: nfsiostat and client NFS connections

Hmm ... but nfsiostat does not really display latency, unless you have some customized version. You want nfs-iostat.py for it: http://communities.netapp.com/thread/13445


---
With best regards

Andrey Borzenkov
Senior system engineer
Service operations

-----Original Message-----
From: Randy Rue [mailto:rrue [at] fhcrc]
Sent: Thursday, February 02, 2012 8:31 AM
To: Borzenkov, Andrey
Cc: toasters [at] teaparty
Subject: Re: nfsiostat and client NFS connections

I appreciate your feedback.

I'm attempting to get some real-time view of current latency for NFS mounts, but have figured out that running nfsiostat returns the average since boot. Running it with multiple iterations in one call seems to return transient data, but running a single 1 second iteration doesn't.

Our ultimate goal is to capture instantaneous latency figures from several clients, post the average of those latencies and the single worst figure into an RRD database, and give a graph showing average and worst case. Can you suggest a way to do this with nfsiostat, or a better way with some other tool?



----- Original Message -----
From: "Andrey Borzenkov" <andrey.borzenkov [at] ts>
To: "Randy Rue" <rrue [at] fhcrc>, toasters [at] teaparty
Sent: Wednesday, February 1, 2012 7:33:02 PM
Subject: RE: nfsiostat and client NFS connections

You do not say how you process results nor how you run nfsiostat.

nfsiostat (just like iostat, vmstat, ...) reports delta for measurement interval. The very first line is always totals since kernel boot. Usually nfsiostat is run with interval argument ("nfsiostat 60" will give you stats every minute). If you invoke it every minute you have to compute delta yourself.

There are other ways to collect information in form easier to postprocess than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or other formats (which are part of the same sysstat package), collectl (can maintain RRD database) etc.

________________________________________
From: toasters-bounces [at] teaparty [toasters-bounces [at] teaparty] On Behalf Of Randy Rue [rrue [at] fhcrc]
Sent: Thursday, February 02, 2012 02:33
To: toasters [at] teaparty
Subject: nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not
specifically a NetApp question but if nothing else, the tool I'm using
appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and
uploading the results to a timestamped log file served on a web server so
I can reach them. I have a script that parses those pages, takes the last
timestamped entry from each file (one file for each client), and tallies
up a) the average RTT time for all mounts to the vFiler and b) the worst
single instance.

Now that I have it all working, the numbers don't believe like I think
they should. The clients are adding an entry to each file every minute. I
can see that from the timestamps before each entry. But the numbers don't
change. Or they change by a few hundredths of a ms. My gather script
returns an average that varies only by .01ms or so, and my "worst" figure
has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings
or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue



_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


andrey.borzenkov at ts

Feb 1, 2012, 11:28 PM

Post #8 of 11 (3410 views)
Permalink
RE: nfsiostat and client NFS connections [In reply to]

OK, that's confusing. Recent enough sysstat includes "nfsiostat" command as well, but it gives totally different statistic.

"1 1" gives you single sample which is cumulative since system boot. Use "1 2" and ignore the first sample. But then you get data for 1 second only. At least use "60 2" to get one minute stats. You still will miss tiny amount of time between consecutive invocations.

I am not aware offhand of any other tool that would show NFS latency, unfortunately.


---
With best regards

Andrey Borzenkov
Senior system engineer
Service operations


-----Original Message-----
From: Randy Rue [mailto:rrue [at] fhcrc]
Sent: Thursday, February 02, 2012 10:39 AM
To: Borzenkov, Andrey
Cc: toasters [at] teaparty
Subject: Re: nfsiostat and client NFS connections

I started with nfs-iostat.py, Realized that nfsiostat that ships with CentOS seems to be the same python script. Output is identical and if you diff the two scripts they're almost identical.

Under the impression that RTT (round trip time) is the same as latency.

But in either case the RTT figures given are averaged since the last reboot if you run it with no arguments, and if you run it with arguments of "1 1" it's not clear what you're actually getting.

I'm looking for something I can run once a minute that will tell me the latency of the NFS mounts in real time.

----- Original Message -----
From: "Andrey Borzenkov" <andrey.borzenkov [at] ts>
To: "Randy Rue" <rrue [at] fhcrc>
Cc: toasters [at] teaparty
Sent: Wednesday, February 1, 2012 10:08:14 PM
Subject: RE: nfsiostat and client NFS connections

Hmm ... but nfsiostat does not really display latency, unless you have some customized version. You want nfs-iostat.py for it: http://communities.netapp.com/thread/13445


---
With best regards

Andrey Borzenkov
Senior system engineer
Service operations

-----Original Message-----
From: Randy Rue [mailto:rrue [at] fhcrc]
Sent: Thursday, February 02, 2012 8:31 AM
To: Borzenkov, Andrey
Cc: toasters [at] teaparty
Subject: Re: nfsiostat and client NFS connections

I appreciate your feedback.

I'm attempting to get some real-time view of current latency for NFS mounts, but have figured out that running nfsiostat returns the average since boot. Running it with multiple iterations in one call seems to return transient data, but running a single 1 second iteration doesn't.

Our ultimate goal is to capture instantaneous latency figures from several clients, post the average of those latencies and the single worst figure into an RRD database, and give a graph showing average and worst case. Can you suggest a way to do this with nfsiostat, or a better way with some other tool?



----- Original Message -----
From: "Andrey Borzenkov" <andrey.borzenkov [at] ts>
To: "Randy Rue" <rrue [at] fhcrc>, toasters [at] teaparty
Sent: Wednesday, February 1, 2012 7:33:02 PM
Subject: RE: nfsiostat and client NFS connections

You do not say how you process results nor how you run nfsiostat.

nfsiostat (just like iostat, vmstat, ...) reports delta for measurement interval. The very first line is always totals since kernel boot. Usually nfsiostat is run with interval argument ("nfsiostat 60" will give you stats every minute). If you invoke it every minute you have to compute delta yourself.

There are other ways to collect information in form easier to postprocess than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or other formats (which are part of the same sysstat package), collectl (can maintain RRD database) etc.

________________________________________
From: toasters-bounces [at] teaparty [toasters-bounces [at] teaparty] On Behalf Of Randy Rue [rrue [at] fhcrc]
Sent: Thursday, February 02, 2012 02:33
To: toasters [at] teaparty
Subject: nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not
specifically a NetApp question but if nothing else, the tool I'm using
appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and
uploading the results to a timestamped log file served on a web server so
I can reach them. I have a script that parses those pages, takes the last
timestamped entry from each file (one file for each client), and tallies
up a) the average RTT time for all mounts to the vFiler and b) the worst
single instance.

Now that I have it all working, the numbers don't believe like I think
they should. The clients are adding an entry to each file every minute. I
can see that from the timestamps before each entry. But the numbers don't
change. Or they change by a few hundredths of a ms. My gather script
returns an average that varies only by .01ms or so, and my "worst" figure
has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings
or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue



_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


johnpc at xs4all

Feb 2, 2012, 1:00 AM

Post #9 of 11 (3277 views)
Permalink
Re: nfsiostat and client NFS connections [In reply to]

On 2012-2-2 8:28 , Borzenkov, Andrey wrote:
> I am not aware offhand of any other tool that would show NFS latency, unfortunately.

There's "stats show volume:*:avg_latency" on the filer itself, for example (and several other latency counters, both under 'volume' and under 'nfsv3', see
"stats list counters nfsv3" and "stats list counters volume").

Of course, that ignores any network latency to the clients, but it might still be a useful thing to measure. The same values are also available via the API, but not via SNMP.

--
Jan-Pieter Cornet
SSL is only keeping your connection safe from hackers, crooks and three
letter agencies by the least secured, least likely to refuse money from
strangers, and least bullying-proof of several hundred companies worldwide.
Attachments: signature.asc (0.33 KB)


andrey.borzenkov at ts

Feb 2, 2012, 2:39 AM

Post #10 of 11 (3274 views)
Permalink
RE: nfsiostat and client NFS connections [In reply to]

I could not find information what exactly is measured on filer. I.e. does it show total time from receiving request on LAN port until response is sent out of LAN port or there may be extra delay e.g. due to packet queuing in transport layer after response has been submitted.

So it is definitely useful for trend analysis, but as I understand here the problem is to find out exact place that incurs delays.


---
With best regards

Andrey Borzenkov
Senior system engineer
Service operations


-----Original Message-----
From: Jan-Pieter Cornet [mailto:johnpc [at] xs4all]
Sent: Thursday, February 02, 2012 1:01 PM
To: Borzenkov, Andrey
Cc: Randy Rue; toasters [at] teaparty
Subject: Re: nfsiostat and client NFS connections

On 2012-2-2 8:28 , Borzenkov, Andrey wrote:
> I am not aware offhand of any other tool that would show NFS latency, unfortunately.

There's "stats show volume:*:avg_latency" on the filer itself, for example (and several other latency counters, both under 'volume' and under 'nfsv3', see "stats list counters nfsv3" and "stats list counters volume").

Of course, that ignores any network latency to the clients, but it might still be a useful thing to measure. The same values are also available via the API, but not via SNMP.

--
Jan-Pieter Cornet
SSL is only keeping your connection safe from hackers, crooks and three letter agencies by the least secured, least likely to refuse money from strangers, and least bullying-proof of several hundred companies worldwide.


_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


rrue at fhcrc

Feb 8, 2012, 11:06 AM

Post #11 of 11 (3263 views)
Permalink
RE: nfsiostat and client NFS connections [In reply to]

Been away from this issue for a little while.

Took a look at "stats list counters nfsv3" and "stats show nfsv3."

Looks like exactly what I'm looking for but it's showing for either the
root filer or the total physical filer. Any way to show it for a specific
vFiler?

Randy


-----Original Message-----
From: Borzenkov, Andrey [mailto:andrey.borzenkov [at] ts]
Sent: Thursday, February 02, 2012 2:39 AM
To: Jan-Pieter Cornet
Cc: Randy Rue; toasters [at] teaparty
Subject: RE: nfsiostat and client NFS connections

I could not find information what exactly is measured on filer. I.e. does
it show total time from receiving request on LAN port until response is
sent out of LAN port or there may be extra delay e.g. due to packet
queuing in transport layer after response has been submitted.

So it is definitely useful for trend analysis, but as I understand here
the problem is to find out exact place that incurs delays.


---
With best regards

Andrey Borzenkov
Senior system engineer
Service operations


-----Original Message-----
From: Jan-Pieter Cornet [mailto:johnpc [at] xs4all]
Sent: Thursday, February 02, 2012 1:01 PM
To: Borzenkov, Andrey
Cc: Randy Rue; toasters [at] teaparty
Subject: Re: nfsiostat and client NFS connections

On 2012-2-2 8:28 , Borzenkov, Andrey wrote:
> I am not aware offhand of any other tool that would show NFS latency,
unfortunately.

There's "stats show volume:*:avg_latency" on the filer itself, for example
(and several other latency counters, both under 'volume' and under
'nfsv3', see "stats list counters nfsv3" and "stats list counters
volume").

Of course, that ignores any network latency to the clients, but it might
still be a useful thing to measure. The same values are also available via
the API, but not via SNMP.

--
Jan-Pieter Cornet
SSL is only keeping your connection safe from hackers, crooks and three
letter agencies by the least secured, least likely to refuse money from
strangers, and least bullying-proof of several hundred companies
worldwide.

_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters

Netapp toasters RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.