Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Netapp: toasters

deduplication, snapshots and performance questions

 

 

Netapp toasters RSS feed   Index | Next | Previous | View Threaded


rrue at fhcrc

Feb 15, 2012, 4:13 PM

Post #1 of 5 (3264 views)
Permalink
deduplication, snapshots and performance questions

Hello All,


We're troubleshooting some performance issues and trying to determine how deduplication might be might be muddying the waters on an already busy v3170 HA filer pair. There's also some likelihood that snapshots are involved.


In a nutshell we have two Vfilers running on one node. One serves NFS datastores to our VMWare farm and all its volumes are de-duped. We recently migrated the other to the node, it's serving NFS to a wide variety of hosts and IO tends to be very heavy on metadata (get_attr and set_attr).


We've found a pretty clear correlation between running de-dupes and performance complaints from customers of the other Vfiler. Whether we're running one de-dupe session or eight, CPU3 stays in the mid-high nineties. This is consistent with what NetApp has told us: that de-dupe processes live in the "Kahuna" domain that always ends up on one proc (in our case, the last of the four). They've also told us that de-dupe processes are heavily "niced" and shouldn't impact other processes, but after just having finished reading their 75 page PDF TR-3505 ("NetApp Deduplication for FAS and V-Series Deployment and Implementation Guide"), it's pretty clear the answer is a huge "it depends" and that de-dupe processes have a very real impact on performance. Especially if we're talking about a system that's otherwise heavily used, that is a higher model number, and that is using ATA drives (bingo on all three).


Does anyone have any real-world experience with de-dupe processes impacting performance? Any suggestions for tuning? We've spent some time moving the schedules around but kept finding that efforts to spread them out were pretty fruitless: some volumes took much longer than others and eventually they started overlapping and stacking up. Right now we're running a script from another server that checks to see how many de-dupes are running and launches new ones as old ones complete, depending on what time it is. The end result is we run a constant loop over a list of fourteen VMWare datastore volumes with two de-dupes running on nights and weekends and none during the day. This means it takes about three days for that list to repeat; we used to run each volume every other day so we're watching the size numbers to make sure we don't hit any walls on the aggregates.


Here's a tangent question. Does anyone know if, everything else being equal, two equal de-dupe jobs will complete faster if they're run sequentially or in parallel? If this process really is confined to a single proc and the work to be done is directly related to the ops available the blocks to be changed, it seems like should be a wash, but we haven't collected enough data to figure this out for ourselves (I'm afraid if we spend too much time running at only one process we'll have space problems).


To add snapshots to the mix: we're running hourly snapshots on most or all of the volumes on both Vfilers. NetApp has told us this adds considerable performance load, most notably the work involved in releasing a snapshot as the oldest one rolls off the stack. They've said they can see instances in the logs where a snapshot begins before the previous one is finished. We're considering splitting the volumes into two groups and snapping them at alternate hours, so we get protection at two-hour granularity but each hour we're only processing half the volumes.


However. Won't we then be processing twice as many changed blocks for each snapshot? Do we gain anything? We had this exact problem, for example, with a different storage vendor with a similar snapshot implementation. In that case their solution was actually to step up to snapshots every half hour. It ended up that processing smaller hunks of changed blocks more often solved our problem.


Right now we're leaning toward trying a hybrid of these two approaches on our NetApp, splitting the volumes into two groups and running them at the top and bottom of the hour. We still get hourly snapshots on all volumes but only need to parse half as many in each pass.


Anyway, I've rambled a lot and asked not many specific questions. What I'm hoping to hear is about:


* Whether de-dupe processes scale linearly (do two jobs in parallel take the same total time as sequentially)
* Anyone's experience tuning de-dupe processes and/or snapshots to minimize performance impact.




Hope to hear from you,


Randy


bryer at sfu

Feb 15, 2012, 4:44 PM

Post #2 of 5 (3179 views)
Permalink
Re: deduplication, snapshots and performance questions [In reply to]

We ran into similar performance problems using dedupe on a 3040 cluster. We don't use vFilers.
For us the performance problems really hit our FC LUNs for VMWare. We didn't get too many complaints
about the performance on the other volumes on the system (our DB/2 and Oracle volumes, CIFS and NFS shares).

Like you, we saw the high Kahuna domain utilization.

I wish I could suggest solutions or things to tune. But after months of getting nowhere, we switched to
another vendor for our VMWare storage.

----- Original Message -----
From: "Randy Rue" <rrue [at] fhcrc>
To: toasters [at] teaparty
Sent: Wednesday, February 15, 2012 4:13:48 PM
Subject: deduplication, snapshots and performance questions





Hello All,


We're troubleshooting some performance issues and trying to determine how deduplication might be might be muddying the waters on an already busy v3170 HA filer pair. There's also some likelihood that snapshots are involved.


In a nutshell we have two Vfilers running on one node. One serves NFS datastores to our VMWare farm and all its volumes are de-duped. We recently migrated the other to the node, it's serving NFS to a wide variety of hosts and IO tends to be very heavy on metadata (get_attr and set_attr).


We've found a pretty clear correlation between running de-dupes and performance complaints from customers of the other Vfiler. Whether we're running one de-dupe session or eight, CPU3 stays in the mid-high nineties. This is consistent with what NetApp has told us: that de-dupe processes live in the "Kahuna" domain that always ends up on one proc (in our case, the last of the four). They've also told us that de-dupe processes are heavily "niced" and shouldn't impact other processes, but after just having finished reading their 75 page PDF TR-3505 ("NetApp Deduplication for FAS and V-Series Deployment and Implementation Guide"), it's pretty clear the answer is a huge "it depends" and that de-dupe processes have a very real impact on performance. Especially if we're talking about a system that's otherwise heavily used, that is a higher model number, and that is using ATA drives (bingo on all three).


Does anyone have any real-world experience with de-dupe processes impacting performance? Any suggestions for tuning? We've spent some time moving the schedules around but kept finding that efforts to spread them out were pretty fruitless: some volumes took much longer than others and eventually they started overlapping and stacking up. Right now we're running a script from another server that checks to see how many de-dupes are running and launches new ones as old ones complete, depending on what time it is. The end result is we run a constant loop over a list of fourteen VMWare datastore volumes with two de-dupes running on nights and weekends and none during the day. This means it takes about three days for that list to repeat; we used to run each volume every other day so we're watching the size numbers to make sure we don't hit any walls on the aggregates.


Here's a tangent question. Does anyone know if, everything else being equal, two equal de-dupe jobs will complete faster if they're run sequentially or in parallel? If this process really is confined to a single proc and the work to be done is directly related to the ops available the blocks to be changed, it seems like should be a wash, but we haven't collected enough data to figure this out for ourselves (I'm afraid if we spend too much time running at only one process we'll have space problems).


To add snapshots to the mix: we're running hourly snapshots on most or all of the volumes on both Vfilers. NetApp has told us this adds considerable performance load, most notably the work involved in releasing a snapshot as the oldest one rolls off the stack. They've said they can see instances in the logs where a snapshot begins before the previous one is finished. We're considering splitting the volumes into two groups and snapping them at alternate hours, so we get protection at two-hour granularity but each hour we're only processing half the volumes.


However. Won't we then be processing twice as many changed blocks for each snapshot? Do we gain anything? We had this exact problem, for example, with a different storage vendor with a similar snapshot implementation. In that case their solution was actually to step up to snapshots every half hour. It ended up that processing smaller hunks of changed blocks more often solved our problem.


Right now we're leaning toward trying a hybrid of these two approaches on our NetApp, splitting the volumes into two groups and running them at the top and bottom of the hour. We still get hourly snapshots on all volumes but only need to parse half as many in each pass.


Anyway, I've rambled a lot and asked not many specific questions. What I'm hoping to hear is about:


* Whether de-dupe processes scale linearly (do two jobs in parallel take the same total time as sequentially)
* Anyone's experience tuning de-dupe processes and/or snapshots to minimize performance impact.




Hope to hear from you,


Randy
_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


jack1729 at gmail

Feb 15, 2012, 5:36 PM

Post #3 of 5 (3175 views)
Permalink
Re: deduplication, snapshots and performance questions [In reply to]

Have you looked at vmdk alignment. On the nfs datastore - we saw performance issues related to snapshots deletion during high utilization

Make sure the dedupe savings are worth the 'effort'. We had issues with deduping large volumes but only saw 2-3% savings. We ended up stopping deupe on those volumes.

Jack
Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Randy Rue <rrue [at] fhcrc>
Sender: toasters-bounces [at] teaparty
Date: Wed, 15 Feb 2012 16:13:48
To: <toasters [at] teaparty>
Subject: deduplication, snapshots and performance questions

_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters



_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


netbacker at gmail

Feb 15, 2012, 7:05 PM

Post #4 of 5 (3172 views)
Permalink
Re: deduplication, snapshots and performance questions [In reply to]

On Wed, Feb 15, 2012 at 4:13 PM, Randy Rue <rrue [at] fhcrc> wrote:
> Hello All,
>
> We're troubleshooting some performance issues and trying to determine how
> deduplication might be might be muddying the waters on an already busy v3170
> HA filer pair. There's also some likelihood that snapshots are involved.

Why do you need to run the de-dup process daily? In our environment,
we set the schedule to auto & let the filer manage it based on the
threshold and still see very good de-dupe ratio and good performance.
e.g.:
Filesystem used saved %saved
/vol/vmstore1/ 446GB 301GB 40%
/vol/vmstore2/ 607GB 281GB 32%
/vol/vmstore3/ 693GB 245GB 26%
/vol/vmstore4/ 494GB 197GB 29%
/vol/vmswap/ 81GB 95GB 54%
/vol/vmstore5/ 1378GB 461GB 25%
/vol/vmstore6/ 70GB 57GB 45%
/vol/vmstore7/ 4598MB 624MB 12%
/vol/vmstore8/ 329GB 109GB 25%
/vol/vmstore9/ 446GB 90GB 17%

sis status
/vol/vmstore1 Enabled Idle Idle for 43:34:28
/vol/vmstore2 Enabled Idle Idle for 10875:57:18
/vol/vmstore3 Enabled Idle Idle for 41:50:19
/vol/vmstore4 Enabled Idle Idle for 163:47:09
/vol/vmstore5 Enabled Idle Idle for 25:51:08
/vol/vmstore6 Enabled Idle Idle for 32:22:59
/vol/vmstore7 Enabled Idle Idle for 27:20:30
/vol/vmstore8 Enabled Idle Idle for 56:33:27
/vol/vmstore9 Enabled Idle Idle for 50:51:38
/vol/vmswap Enabled Idle Idle for 39:46:32

Hope that helps

-net
_______________________________________________
Toasters mailing list
Toasters [at] teaparty
http://www.teaparty.net/mailman/listinfo/toasters


sklise at hotmail

Feb 16, 2012, 11:00 AM

Post #5 of 5 (3164 views)
Permalink
RE: deduplication, snapshots and performance questions [In reply to]

Just to add to the fodder..

Have you looked at performance advisor? I use it from a high level on whats really busy.

Also, if you've reciently added spindles to an existing aggr, may need to run a re-allocate.

Maybe don't have enough spindles for the work loads. Little value add here, but maybe you need (if you don't already have it) a PAM card with the appropriate size.

Just thought I would throw that in.



> Date: Wed, 15 Feb 2012 19:05:24 -0800
> Subject: Re: deduplication, snapshots and performance questions
> From: netbacker [at] gmail
> To: rrue [at] fhcrc
> CC: toasters [at] teaparty
>
> On Wed, Feb 15, 2012 at 4:13 PM, Randy Rue <rrue [at] fhcrc> wrote:
> > Hello All,
> >
> > We're troubleshooting some performance issues and trying to determine how
> > deduplication might be might be muddying the waters on an already busy v3170
> > HA filer pair. There's also some likelihood that snapshots are involved.
>
> Why do you need to run the de-dup process daily? In our environment,
> we set the schedule to auto & let the filer manage it based on the
> threshold and still see very good de-dupe ratio and good performance.
> e.g.:
> Filesystem used saved %saved
> /vol/vmstore1/ 446GB 301GB 40%
> /vol/vmstore2/ 607GB 281GB 32%
> /vol/vmstore3/ 693GB 245GB 26%
> /vol/vmstore4/ 494GB 197GB 29%
> /vol/vmswap/ 81GB 95GB 54%
> /vol/vmstore5/ 1378GB 461GB 25%
> /vol/vmstore6/ 70GB 57GB 45%
> /vol/vmstore7/ 4598MB 624MB 12%
> /vol/vmstore8/ 329GB 109GB 25%
> /vol/vmstore9/ 446GB 90GB 17%
>
> sis status
> /vol/vmstore1 Enabled Idle Idle for 43:34:28
> /vol/vmstore2 Enabled Idle Idle for 10875:57:18
> /vol/vmstore3 Enabled Idle Idle for 41:50:19
> /vol/vmstore4 Enabled Idle Idle for 163:47:09
> /vol/vmstore5 Enabled Idle Idle for 25:51:08
> /vol/vmstore6 Enabled Idle Idle for 32:22:59
> /vol/vmstore7 Enabled Idle Idle for 27:20:30
> /vol/vmstore8 Enabled Idle Idle for 56:33:27
> /vol/vmstore9 Enabled Idle Idle for 50:51:38
> /vol/vmswap Enabled Idle Idle for 39:46:32
>
> Hope that helps
>
> -net
> _______________________________________________
> Toasters mailing list
> Toasters [at] teaparty
> http://www.teaparty.net/mailman/listinfo/toasters

Netapp toasters RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.