Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DRBD: Users

Weird DRBD performance problem

 

 

DRBD users RSS feed   Index | Next | Previous | View Threaded


MRoof at admin

Feb 1, 2012, 5:04 PM

Post #1 of 3 (753 views)
Permalink
Weird DRBD performance problem

Hi Everyone,

I have a DRBD performance problem that has got me completely confused. I hoping that someone can help with this one as my other servers that use the same type of RAID cards and DRBD don't have this problem.

For the hardware, I have two Dell R515 servers with the H700 card, basically an LSI Megaraid based card, and running SLES 11 SP1. This problem shows up on drbd 8.3.11, 8.3.12, and 8.4.1 but I haven't tested other versions.

here is the simple config I made based on the servers that don't have any issues:

global {
# We don't want to be bother by the usage count numbers
usage-count no;
}
common {
protocol C;
net {
cram-hmac-alg md5;
shared-secret "P4ss";
}
}
resource r0 {
on san1 {
device /dev/drbd0;
disk /dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part2;
address 10.60.60.1:63000;
flexible-meta-disk /dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part1;
}
on san2 {
device /dev/drbd0;
disk /dev/disk/by-id/scsi-36782bcb0698b6e00167bb1d107a77a47-part2;
address 10.60.60.2:63000;
flexible-meta-disk /dev/disk/by-id/scsi-36782bcb0698b6e00167bb1d107a77a47-part1;
}
startup {
wfc-timeout 5;
}
syncer {
rate 50M;
cpu-mask 4;
}
disk {
on-io-error detach;
no-disk-barrier;
no-disk-flushes;
no-disk-drain;
no-md-flushes;
}
}

version: 8.3.11 (api:88/proto:86-96)
GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by phil [at] fat-tyr <mailto:phil [at] fat-tyr> , 2011-06-29 11:37:11
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----s
ns:0 nr:0 dw:8501248 dr:551 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:n oos:3397375600

So, when I'm running just with one server and no replication the performance hit with DRBD is huge. The backing device shows a throughput of:
----
san1:~ # dd if=/dev/zero of=/dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part2 bs=1M count=16384
16384+0 records in
16384+0 records out
17179869184 bytes (17 GB) copied, 16.4434 s, 1.0 GB/s
----
san1:~ # dd if=/dev/zero of=/dev/drbd/by-res/r0 bs=1M count=16384
16384+0 records in
16384+0 records out
17179869184 bytes (17 GB) copied, 93.457 s, 184 MB/s
-------

using iostat I see part of the problem:

avg-cpu: %user %nice %system %iowait %steal %idle
0.08 0.00 16.76 0.00 0.00 83.17
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.00 0.00 0.00 0 0
sdb 20565.00 0.00 360.00 0 719
drbd0 737449.50 0.00 360.08 0 720
avg-cpu: %user %nice %system %iowait %steal %idle
0.07 0.00 28.87 1.37 0.00 69.69
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 1.50 0.00 0.01 0 0
sdb 57859.50 0.00 177.22 0 354
drbd0 362787.00 0.00 177.14 0 354

the drbd device is showing a TPS about 10x - 20x of the backing store. When I do this on my other servers I don't see anything like it. The working servers are also running the same kernel and drbd versions.

Does anyone have any ideas of how this might be resolved or fixed? I'm at a loss right now.


lars.ellenberg at linbit

Feb 2, 2012, 6:11 AM

Post #2 of 3 (721 views)
Permalink
Re: Weird DRBD performance problem [In reply to]

On Wed, Feb 01, 2012 at 06:04:18PM -0700, Roof, Morey R. wrote:
> Hi Everyone,
>
> I have a DRBD performance problem that has got me completely confused.
> I hoping that someone can help with this one as my other servers that
> use the same type of RAID cards and DRBD don't have this problem.
>
> For the hardware, I have two Dell R515 servers with the H700 card,
> basically an LSI Megaraid based card, and running SLES 11 SP1. This
> problem shows up on drbd 8.3.11, 8.3.12, and 8.4.1 but I haven't
> tested other versions.
>
> here is the simple config I made based on the servers that don't have
> any issues:
>
> global {
> # We don't want to be bother by the usage count numbers
> usage-count no;
> }
> common {
> protocol C;
> net {
> cram-hmac-alg md5;
> shared-secret "P4ss";
> }
> }
> resource r0 {
> on san1 {
> device /dev/drbd0;
> disk /dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part2;
> address 10.60.60.1:63000;
> flexible-meta-disk /dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part1;
> }
> on san2 {
> device /dev/drbd0;
> disk /dev/disk/by-id/scsi-36782bcb0698b6e00167bb1d107a77a47-part2;
> address 10.60.60.2:63000;
> flexible-meta-disk /dev/disk/by-id/scsi-36782bcb0698b6e00167bb1d107a77a47-part1;
> }
> startup {
> wfc-timeout 5;
> }
> syncer {
> rate 50M;
> cpu-mask 4;
> }
> disk {
> on-io-error detach;
> no-disk-barrier;
> no-disk-flushes;
> no-disk-drain;


Will people please STOP using no-disk-drain. On most hardware, it does
not provide measurable performance gain, but may risk data integrity
because of potential violation of write-after-write dependencies!

> no-md-flushes;
> }
> }
>
> version: 8.3.11 (api:88/proto:86-96)
> GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by phil [at] fat-tyr <mailto:phil [at] fat-tyr> , 2011-06-29 11:37:11
> 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----s
> ns:0 nr:0 dw:8501248 dr:551 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:n oos:3397375600
>
> So, when I'm running just with one server and no replication the performance hit with DRBD is huge. The backing device shows a throughput of:
> ----
> san1:~ # dd if=/dev/zero of=/dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part2 bs=1M count=16384

Hope you are not writing to the page cache only?
add oflag=direct, or oflag=dsync, or conv=fsync combinations thereof.

> san1:~ # dd if=/dev/zero of=/dev/drbd/by-res/r0 bs=1M count=16384
> 16384+0 records in
> 16384+0 records out
> 17179869184 bytes (17 GB) copied, 93.457 s, 184 MB/s

See if moving the drbd meta data to raid 1 helps.

> -------
>
> using iostat I see part of the problem:
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.08 0.00 16.76 0.00 0.00 83.17
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> sda 0.00 0.00 0.00 0 0
> sdb 20565.00 0.00 360.00 0 719
> drbd0 737449.50 0.00 360.08 0 720
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.07 0.00 28.87 1.37 0.00 69.69
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> sda 1.50 0.00 0.01 0 0
> sdb 57859.50 0.00 177.22 0 354
> drbd0 362787.00 0.00 177.14 0 354
>
> the drbd device is showing a TPS about 10x - 20x of the backing store.
> When I do this on my other servers I don't see anything like it. The
> working servers are also running the same kernel and drbd versions.

The rest of the IO stack is the same as well, including driver,
firmware, settings, health of controller cache battery?
Not implying anything, that's just something to check...

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


MRoof at admin

Feb 4, 2012, 1:14 AM

Post #3 of 3 (743 views)
Permalink
Re: Weird DRBD performance problem [In reply to]

Hi Lars and Everyone else,

I put the no-disk-drain in the hope that it might help me find the problem. I leave it out on production.

The run with the various dd's with different commands was interesting:

san1:/dev # dd if=/dev/zero of=/dev/sdb2 bs=1M count=32768 oflag=direct
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 38.9945 s, 881 MB/s
san1:/dev # dd if=/dev/zero of=/dev/sdb2 bs=1M count=32768 oflag=dsync
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 88.8506 s, 387 MB/s
san1:/dev # dd if=/dev/zero of=/dev/sdb2 bs=1M count=32768 conv=fsync
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 72.9384 s, 471 MB/s

Not sure why direct would be so much faster than the others. Something to look into. Anyone have some thoughts on that one?

Since I'm trying to figure out what is going on I decided to put the DRBD metadata on a RAM disk and see what happens and did two sets of runs in the same format as the above.

Here are the runs with the metadata on the regular RAID system:

san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 oflag=direct
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 53.1025 s, 647 MB/s
san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 oflag=dsync
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 147.199 s, 233 MB/s
san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 conv=fsync
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 156.27 s, 220 MB/s

Here are the runs with the metadata on the RAM disk:

san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 oflag=direct
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 53.6152 s, 641 MB/s
san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 oflag=dsync
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 129.425 s, 265 MB/s
san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 conv=fsync
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 148.076 s, 232 MB/s

I did run the tests about 10 times each and they came out to about the same numbers (+/- 15MB/s). Basically, the difference between the metadata being on the disk and something extremely fast doesn't make a difference in this setup. The odd thing is the DRBD overhead is just really high on this system for some reason and I can't seem to find the answer.

The firmware between the raid controllers isn't the same. The big difference is that these servers are using the dell H700 card and the other server are using the LSI Megaraid. Both cards are really made by LSI but Dell has its own firmware but they all use the same driver.

I also don't understand why the transactions per second as seem by iostat is so high on the drbd device but when it gets to the backing store the transactions drop. Also, when I just run the dd command on the backing store the transactions per second are not high and look just like they do in the drbd test run for the backing device. For some reason the drbd device shows a huge number of transactions.

Anymore have some more thoughts about where the overhead hit might be coming from?

thanks



________________________________

From: drbd-user-bounces [at] lists on behalf of Lars Ellenberg
Sent: Thu 2/2/2012 7:11 AM
To: drbd-user [at] lists
Subject: Re: [DRBD-user] Weird DRBD performance problem



On Wed, Feb 01, 2012 at 06:04:18PM -0700, Roof, Morey R. wrote:
> Hi Everyone,
>
> I have a DRBD performance problem that has got me completely confused.
> I hoping that someone can help with this one as my other servers that
> use the same type of RAID cards and DRBD don't have this problem.
>
> For the hardware, I have two Dell R515 servers with the H700 card,
> basically an LSI Megaraid based card, and running SLES 11 SP1. This
> problem shows up on drbd 8.3.11, 8.3.12, and 8.4.1 but I haven't
> tested other versions.
>
> here is the simple config I made based on the servers that don't have
> any issues:
>
> global {
> # We don't want to be bother by the usage count numbers
> usage-count no;
> }
> common {
> protocol C;
> net {
> cram-hmac-alg md5;
> shared-secret "P4ss";
> }
> }
> resource r0 {
> on san1 {
> device /dev/drbd0;
> disk /dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part2;
> address 10.60.60.1:63000;
> flexible-meta-disk /dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part1;
> }
> on san2 {
> device /dev/drbd0;
> disk /dev/disk/by-id/scsi-36782bcb0698b6e00167bb1d107a77a47-part2;
> address 10.60.60.2:63000;
> flexible-meta-disk /dev/disk/by-id/scsi-36782bcb0698b6e00167bb1d107a77a47-part1;
> }
> startup {
> wfc-timeout 5;
> }
> syncer {
> rate 50M;
> cpu-mask 4;
> }
> disk {
> on-io-error detach;
> no-disk-barrier;
> no-disk-flushes;
> no-disk-drain;


Will people please STOP using no-disk-drain. On most hardware, it does
not provide measurable performance gain, but may risk data integrity
because of potential violation of write-after-write dependencies!

> no-md-flushes;
> }
> }
>
> version: 8.3.11 (api:88/proto:86-96)
> GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by phil [at] fat-tyr <mailto:phil [at] fat-tyr <mailto:phil [at] fat-tyr> > , 2011-06-29 11:37:11
> 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----s
> ns:0 nr:0 dw:8501248 dr:551 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:n oos:3397375600
>
> So, when I'm running just with one server and no replication the performance hit with DRBD is huge. The backing device shows a throughput of:
> ----
> san1:~ # dd if=/dev/zero of=/dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part2 bs=1M count=16384

Hope you are not writing to the page cache only?
add oflag=direct, or oflag=dsync, or conv=fsync combinations thereof.

> san1:~ # dd if=/dev/zero of=/dev/drbd/by-res/r0 bs=1M count=16384
> 16384+0 records in
> 16384+0 records out
> 17179869184 bytes (17 GB) copied, 93.457 s, 184 MB/s

See if moving the drbd meta data to raid 1 helps.

> -------
>
> using iostat I see part of the problem:
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.08 0.00 16.76 0.00 0.00 83.17
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> sda 0.00 0.00 0.00 0 0
> sdb 20565.00 0.00 360.00 0 719
> drbd0 737449.50 0.00 360.08 0 720
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.07 0.00 28.87 1.37 0.00 69.69
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> sda 1.50 0.00 0.01 0 0
> sdb 57859.50 0.00 177.22 0 354
> drbd0 362787.00 0.00 177.14 0 354
>
> the drbd device is showing a TPS about 10x - 20x of the backing store.
> When I do this on my other servers I don't see anything like it. The
> working servers are also running the same kernel and drbd versions.

The rest of the IO stack is the same as well, including driver,
firmware, settings, health of controller cache battery?
Not implying anything, that's just something to check...

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com <http://www.linbit.com/>
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user <http://lists.linbit.com/mailman/listinfo/drbd-user>

DRBD users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.