Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DRBD: Users

drbd write performance slow, per disk 40M/s by dd command

 

 

DRBD users RSS feed   Index | Next | Previous | View Threaded


zf5984599 at gmail

Apr 25, 2012, 1:28 AM

Post #1 of 10 (1864 views)
Permalink
drbd write performance slow, per disk 40M/s by dd command

hi, dear all:

When I use drbd, I found the write performance very slow, against testing without drbd module.

1. the environment:
-) CentOS 5.6
-) 2.6.18 kernel
-) drbd 8.4.1
-) drbd.conf
resource r0
{
protocol B;

net
{
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 512K;
}

disk
{
al-extents 3389;
}


on OSS211
{
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.100.231:7788;
meta-disk internal;
}

on OSS213
{
device /dev/drbd0;
disk /dev/sde1;
address 192.168.100.213:7788;
meta-disk internal;
}

}


2. Test scenario:
*) without drbd module,
dd to write 1G stream into one disk, which formatted to ext3:
[para]# !echo
echo 3 > /proc/sys/vm/drop_caches
[para]# !dd
dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.9905 seconds, 95.4 MB/s

*) with drbd module,
dd 1G stream to the disk, which is ext3 too:
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----
ns:1260036 nr:0 dw:1260036 dr:297 al:330 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
[para]#
[para]# dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 26.7392 seconds, 39.2 MB/s
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----

All the upper test writing disk are same. From the upper result, if i use DRBD to test, the performance
is 39 MB/s; while if i do not use, the performance is about 95M/s.

3. My question is:
-) this write performance decays so large is normal or not?
I had read the following from the DRBD website:
"15.1. Hardware considerations:
.... A single, reasonably recent, SCSI or SAS disk will
typically allow streaming writes of roughly 40MB/s to the single disk."
But this is very slow.

-if this is not normal, how can i turn this? is the config file something not correct?

thanks a lot
BRs,
feng


zf5984599 at gmail

Apr 25, 2012, 5:38 AM

Post #2 of 10 (1790 views)
Permalink
drbd write performance slow, per disk 40M/s by dd command [In reply to]

hi, dear all:

When I use drbd, I found the write performance very slow, against
testing without drbd module.

1. the environment:
-) CentOS 5.6
-) 2.6.18 kernel
-) drbd 8.4.1
-) drbd.conf
resource r0
{
protocol B;

net
{
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 512K;
}

disk
{
al-extents 3389;
}


on OSS211
{
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.100.231:7788;
meta-disk internal;
}

on OSS213
{
device /dev/drbd0;
disk /dev/sde1;
address 192.168.100.213:7788;
meta-disk internal;
}

}


2. Test scenario:
*) without drbd module,
dd to write 1G stream into one disk, which formatted to ext3:
[para]# !echo
echo 3 > /proc/sys/vm/drop_caches
[para]# !dd
dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.9905 seconds, 95.4 MB/s

*) with drbd module,
dd 1G stream to the disk, which is ext3 too:
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----
ns:1260036 nr:0 dw:1260036 dr:297 al:330 bm:0 lo:0 pe:0 ua:0 ap:0
ep:1 wo:b oos:0
[para]#
[para]# dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 26.7392 seconds, 39.2 MB/s
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----

All the upper test writing disk are same. From the upper result, if i
use DRBD to test, the performance
is 39 MB/s; while if i do not use, the performance is about 95M/s.

3. My question is:
-) this write performance decays so large is normal or not?
I had read the following from the DRBD website:
"15.1. Hardware considerations:
.... A single, reasonably recent, SCSI or SAS disk will
typically allow streaming writes of roughly 40MB/s to the single disk."
But this is very slow.

-if this is not normal, how can i turn this? is the config file
something not correct?

thanks a lot
BRs,
feng
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


chrisd1100 at gmail

Apr 25, 2012, 7:35 AM

Post #3 of 10 (1793 views)
Permalink
Re: drbd write performance slow, per disk 40M/s by dd command [In reply to]

Try turning off disk-barrier and disk-flushes and see if that makes a
difference.

2012/4/25 feng zheng <zf5984599 [at] gmail>

> hi, dear all:
>
> When I use drbd, I found the write performance very slow, against
> testing without drbd module.
>
> 1. the environment:
> -) CentOS 5.6
> -) 2.6.18 kernel
> -) drbd 8.4.1
> -) drbd.conf:
> resource r0
> {
> protocol B;
>
> net
> {
> max-buffers 8000;
> max-epoch-size 8000;
> sndbuf-size 512K;
> }
>
> disk
> {
> al-extents 3389;
> }
>
>
> on OSS211
> {
> device /dev/drbd0;
> disk /dev/sdb1;
> address 192.168.100.231:7788;
> meta-disk internal;
> }
>
> on OSS213
> {
> device /dev/drbd0;
> disk /dev/sde1;
> address 192.168.100.213:7788;
> meta-disk internal;
> }
>
> }
>
>
> 2. Test scenario:
> *) without drbd module,
> dd to write 1G stream into one disk, which formatted to ext3:
> [para]# !echo
> echo 3 > /proc/sys/vm/drop_caches
> [para]# !dd
> dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 10.9905 seconds, 95.4 MB/s
>
> *) with drbd module,
> dd 1G stream to the disk, which is ext3 too:
> [para]# cat /proc/drbd
> version: 8.4.1 (api:1/proto:86-100)
> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
> root [at] OSS21, 2012-04-16 21:38:36
> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----
> ns:1260036 nr:0 dw:1260036 dr:297 al:330 bm:0 lo:0 pe:0 ua:0 ap:0
> ep:1 wo:b oos:0
> [para]#
> [para]# dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 26.7392 seconds, 39.2 MB/s
> [para]# cat /proc/drbd
> version: 8.4.1 (api:1/proto:86-100)
> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
> root [at] OSS21, 2012-04-16 21:38:36
> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----
>
> All the upper test writing disk are same. From the upper result, if i
> use DRBD to test, the performance
> is 39 MB/s; while if i do not use, the performance is about 95M/s.
>
> 3. My question is:
> -) this write performance decays so large is normal or not?
> I had read the following from the DRBD website:
> "15.1. Hardware considerations:
> .... A single, reasonably recent, SCSI or SAS disk will
> typically allow streaming writes of roughly 40MB/s to the single disk."
> But this is very slow.
>
> -)if this is not normal, how can i turn this? is the config file
> something not correct?
>
> thanks a lot
> BRs,
> feng
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
>


chrisd1100 at gmail

Apr 25, 2012, 7:40 AM

Post #4 of 10 (1792 views)
Permalink
Re: drbd write performance slow, per disk 40M/s by dd command [In reply to]

Also use oflag=direct in both tests and perform them a few times, sometimes
high speeds are the result of caching.

On Wed, Apr 25, 2012 at 10:35 AM, Chris Dickson <chrisd1100 [at] gmail>wrote:

> Try turning off disk-barrier and disk-flushes and see if that makes a
> difference.
>
>
> 2012/4/25 feng zheng <zf5984599 [at] gmail>
>
>> hi, dear all:
>>
>> When I use drbd, I found the write performance very slow, against
>> testing without drbd module.
>>
>> 1. the environment:
>> -) CentOS 5.6
>> -) 2.6.18 kernel
>> -) drbd 8.4.1
>> -) drbd.conf:
>> resource r0
>> {
>> protocol B;
>>
>> net
>> {
>> max-buffers 8000;
>> max-epoch-size 8000;
>> sndbuf-size 512K;
>> }
>>
>> disk
>> {
>> al-extents 3389;
>> }
>>
>>
>> on OSS211
>> {
>> device /dev/drbd0;
>> disk /dev/sdb1;
>> address 192.168.100.231:7788;
>> meta-disk internal;
>> }
>>
>> on OSS213
>> {
>> device /dev/drbd0;
>> disk /dev/sde1;
>> address 192.168.100.213:7788;
>> meta-disk internal;
>> }
>>
>> }
>>
>>
>> 2. Test scenario:
>> *) without drbd module,
>> dd to write 1G stream into one disk, which formatted to ext3:
>> [para]# !echo
>> echo 3 > /proc/sys/vm/drop_caches
>> [para]# !dd
>> dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
>> 1000+0 records in
>> 1000+0 records out
>> 1048576000 bytes (1.0 GB) copied, 10.9905 seconds, 95.4 MB/s
>>
>> *) with drbd module,
>> dd 1G stream to the disk, which is ext3 too:
>> [para]# cat /proc/drbd
>> version: 8.4.1 (api:1/proto:86-100)
>> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
>> root [at] OSS21, 2012-04-16 21:38:36
>> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----
>> ns:1260036 nr:0 dw:1260036 dr:297 al:330 bm:0 lo:0 pe:0 ua:0 ap:0
>> ep:1 wo:b oos:0
>> [para]#
>> [para]# dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
>> 1000+0 records in
>> 1000+0 records out
>> 1048576000 bytes (1.0 GB) copied, 26.7392 seconds, 39.2 MB/s
>> [para]# cat /proc/drbd
>> version: 8.4.1 (api:1/proto:86-100)
>> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
>> root [at] OSS21, 2012-04-16 21:38:36
>> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----
>>
>> All the upper test writing disk are same. From the upper result, if i
>> use DRBD to test, the performance
>> is 39 MB/s; while if i do not use, the performance is about 95M/s.
>>
>> 3. My question is:
>> -) this write performance decays so large is normal or not?
>> I had read the following from the DRBD website:
>> "15.1. Hardware considerations:
>> .... A single, reasonably recent, SCSI or SAS disk will
>> typically allow streaming writes of roughly 40MB/s to the single disk."
>> But this is very slow.
>>
>> -)if this is not normal, how can i turn this? is the config file
>> something not correct?
>>
>> thanks a lot
>> BRs,
>> feng
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user [at] lists
>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>
>
>


themsley at voiceflex

Apr 25, 2012, 8:13 AM

Post #5 of 10 (1823 views)
Permalink
Re: drbd write performance slow, per disk 40M/s by dd command [In reply to]

Did you test your network connection to make sure that it can transfer
at a greater speed than that? Maybe it is the bottleneck - jumbo frames on?

On 25/04/12 15:40, Chris Dickson wrote:
> Also use oflag=direct in both tests and perform them a few times,
> sometimes high speeds are the result of caching.
>
> On Wed, Apr 25, 2012 at 10:35 AM, Chris Dickson <chrisd1100 [at] gmail
> <mailto:chrisd1100 [at] gmail>> wrote:
>
> Try turning off disk-barrier and disk-flushes and see if that
> makes a difference.
>
>
> 2012/4/25 feng zheng <zf5984599 [at] gmail
> <mailto:zf5984599 [at] gmail>>
>
> hi, dear all:
>
> When I use drbd, I found the write performance very slow, against
> testing without drbd module.
>
> 1. the environment:
> -) CentOS 5.6
> -) 2.6.18 kernel
> -) drbd 8.4.1
> -) drbd.conf:
> resource r0
> {
> protocol B;
>
> net
> {
> max-buffers 8000;
> max-epoch-size 8000;
> sndbuf-size 512K;
> }
>
> disk
> {
> al-extents 3389;
> }
>
>
> on OSS211
> {
> device /dev/drbd0;
> disk /dev/sdb1;
> address *MailScanner warning: numerical links are often
> malicious:* 192.168.100.231:7788 <http://192.168.100.231:7788>;
> meta-disk internal;
> }
>
> on OSS213
> {
> device /dev/drbd0;
> disk /dev/sde1;
> address *MailScanner warning: numerical links are often
> malicious:* 192.168.100.213:7788 <http://192.168.100.213:7788>;
> meta-disk internal;
> }
>
> }
>
>
> 2. Test scenario:
> *) without drbd module,
> dd to write 1G stream into one disk, which formatted to ext3:
> [para]# !echo
> echo 3 > /proc/sys/vm/drop_caches
> [para]# !dd
> dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 10.9905 seconds, 95.4 MB/s
>
> *) with drbd module,
> dd 1G stream to the disk, which is ext3 too:
> [para]# cat /proc/drbd
> version: 8.4.1 (api:1/proto:86-100)
> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
> root [at] OSS21, 2012-04-16 21:38:36
> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B
> r-----
> ns:1260036 nr:0 dw:1260036 dr:297 al:330 bm:0 lo:0 pe:0
> ua:0 ap:0
> ep:1 wo:b oos:0
> [para]#
> [para]# dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 26.7392 seconds, 39.2 MB/s
> [para]# cat /proc/drbd
> version: 8.4.1 (api:1/proto:86-100)
> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
> root [at] OSS21, 2012-04-16 21:38:36
> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B
> r-----
>
> All the upper test writing disk are same. From the upper
> result, if i
> use DRBD to test, the performance
> is 39 MB/s; while if i do not use, the performance is about 95M/s.
>
> 3. My question is:
> -) this write performance decays so large is normal or not?
> I had read the following from the DRBD website:
> "15.1. Hardware considerations:
> .... A single, reasonably recent, SCSI or SAS disk will
> typically allow streaming writes of roughly 40MB/s to the
> single disk."
> But this is very slow.
>
> -)if this is not normal, how can i turn this? is the config file
> something not correct?
>
> thanks a lot
> BRs,
> feng
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists <mailto:drbd-user [at] lists>
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
>
>


zf5984599 at gmail

Apr 25, 2012, 6:24 PM

Post #6 of 10 (1792 views)
Permalink
Re: drbd write performance slow, per disk 40M/s by dd command [In reply to]

thanks, Chris.

I had turned off disk-barrier and disk-flushes in config file, but there is no difference.

And I use the oflag=direct to test, there is not faster than ever.

[para]# dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync oflag=direct
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 25.4056 seconds, 41.3 MB/s
[para]# dd if=/dev/zero of=test1 bs=1M count=1000 oflag=direct
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 27.502 seconds, 38.1 MB/s
[para]# echo 3 > /proc/sys/vm/drop_caches
[para]# dd if=/dev/zero of=test1 bs=1M count=1000 oflag=direct
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 25.6661 seconds, 40.9 MB/s
[para]# dd if=/dev/zero of=test1 bs=1M count=1000 oflag=direct
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 25.5987 seconds, 41.0 MB/s






From: Chris Dickson
Date: 2012-04-25 22:35
To: feng zheng
CC: drbd-user
Subject: Re: [DRBD-user] drbd write performance slow, per disk 40M/s by dd command
Try turning off disk-barrier and disk-flushes and see if that makes a difference.


2012/4/25 feng zheng <zf5984599 [at] gmail>

hi, dear all:

When I use drbd, I found the write performance very slow, against
testing without drbd module.

1. the environment:
-) CentOS 5.6
-) 2.6.18 kernel
-) drbd 8.4.1
-) drbd.conf:
resource r0
{
protocol B;

net
{
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 512K;
}

disk
{
al-extents 3389;
}


on OSS211
{
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.100.231:7788;
meta-disk internal;
}

on OSS213
{
device /dev/drbd0;
disk /dev/sde1;
address 192.168.100.213:7788;
meta-disk internal;
}

}


2. Test scenario:
*) without drbd module,
dd to write 1G stream into one disk, which formatted to ext3:
[para]# !echo
echo 3 > /proc/sys/vm/drop_caches
[para]# !dd
dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.9905 seconds, 95.4 MB/s

*) with drbd module,
dd 1G stream to the disk, which is ext3 too:
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----
ns:1260036 nr:0 dw:1260036 dr:297 al:330 bm:0 lo:0 pe:0 ua:0 ap:0
ep:1 wo:b oos:0
[para]#
[para]# dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 26.7392 seconds, 39.2 MB/s
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----

All the upper test writing disk are same. From the upper result, if i
use DRBD to test, the performance
is 39 MB/s; while if i do not use, the performance is about 95M/s.

3. My question is:
-) this write performance decays so large is normal or not?
I had read the following from the DRBD website:
"15.1. Hardware considerations:
.... A single, reasonably recent, SCSI or SAS disk will
typically allow streaming writes of roughly 40MB/s to the single disk."
But this is very slow.

-)if this is not normal, how can i turn this? is the config file
something not correct?

thanks a lot
BRs,
feng

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


zf5984599 at gmail

Apr 25, 2012, 7:06 PM

Post #7 of 10 (1804 views)
Permalink
Re: drbd write performance slow, per disk 40M/s by dd command [In reply to]

yes, the network is ok. I used netperf to test the network.

# netperf -H 192.168.100.231 -L 30
TCP STREAM TEST from 30 (0.0.0.30) port 0 AF_INET to 192.168.100.231 (192.168.100.231) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 16384 10.03 941.37

the network is ok.

then enable 'jumbo frames' on, set both machine MTU = 9000, but there is no difference.
the dd result is still about 40 MBps.





From: Trevor Hemsley
Date: 2012-04-25 23:13
To:
CC: drbd-user [at] lists
Subject: Re: [DRBD-user] drbd write performance slow, per disk 40M/s by dd command
Did you test your network connection to make sure that it can transfer at a greater speed than that? Maybe it is the bottleneck - jumbo frames on?

On 25/04/12 15:40, Chris Dickson wrote:
Also use oflag=direct in both tests and perform them a few times, sometimes high speeds are the result of caching.


On Wed, Apr 25, 2012 at 10:35 AM, Chris Dickson <chrisd1100 [at] gmail> wrote:

Try turning off disk-barrier and disk-flushes and see if that makes a difference.



2012/4/25 feng zheng <zf5984599 [at] gmail>

hi, dear all:

When I use drbd, I found the write performance very slow, against
testing without drbd module.

1. the environment:
-) CentOS 5.6
-) 2.6.18 kernel
-) drbd 8.4.1
-) drbd.conf:
resource r0
{
protocol B;

net
{
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 512K;
}

disk
{
al-extents 3389;
}


on OSS211
{
device /dev/drbd0;
disk /dev/sdb1;
address MailScanner warning: numerical links are often malicious: 192.168.100.231:7788;
meta-disk internal;
}

on OSS213
{
device /dev/drbd0;
disk /dev/sde1;
address MailScanner warning: numerical links are often malicious: 192.168.100.213:7788;
meta-disk internal;
}

}


2. Test scenario:
*) without drbd module,
dd to write 1G stream into one disk, which formatted to ext3:
[para]# !echo
echo 3 > /proc/sys/vm/drop_caches
[para]# !dd
dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.9905 seconds, 95.4 MB/s

*) with drbd module,
dd 1G stream to the disk, which is ext3 too:
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----
ns:1260036 nr:0 dw:1260036 dr:297 al:330 bm:0 lo:0 pe:0 ua:0 ap:0
ep:1 wo:b oos:0
[para]#
[para]# dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 26.7392 seconds, 39.2 MB/s
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----

All the upper test writing disk are same. From the upper result, if i
use DRBD to test, the performance
is 39 MB/s; while if i do not use, the performance is about 95M/s.

3. My question is:
-) this write performance decays so large is normal or not?
I had read the following from the DRBD website:
"15.1. Hardware considerations:
.... A single, reasonably recent, SCSI or SAS disk will
typically allow streaming writes of roughly 40MB/s to the single disk."
But this is very slow.

-)if this is not normal, how can i turn this? is the config file
something not correct?

thanks a lot
BRs,
feng

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


zf5984599 at gmail

Apr 25, 2012, 7:38 PM

Post #8 of 10 (1799 views)
Permalink
Re: drbd write performance slow, per disk 40M/s by dd command [In reply to]

I use the block size = 1024M to test, and found the write performance is about 100MBps.

# dd if=/dev/zero of=test1 bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 25.9577 seconds, 41.4 MB/s -------> this is the bs=1M result

# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/zero of=test1 bs=1024M count=1 oflag=direct
1+0 records in
1+0 records out

# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/zero of=test1 bs=1024M count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 10.714 seconds, 100 MB/s ---------> this is the bs=1024M result

I was wondering that is this a normal case? thanks, all

feng





From: Trevor Hemsley
Date: 2012-04-25 23:13
To:
CC: drbd-user [at] lists
Subject: Re: [DRBD-user] drbd write performance slow, per disk 40M/s by dd command
Did you test your network connection to make sure that it can transfer at a greater speed than that? Maybe it is the bottleneck - jumbo frames on?

On 25/04/12 15:40, Chris Dickson wrote:
Also use oflag=direct in both tests and perform them a few times, sometimes high speeds are the result of caching.


On Wed, Apr 25, 2012 at 10:35 AM, Chris Dickson <chrisd1100 [at] gmail> wrote:

Try turning off disk-barrier and disk-flushes and see if that makes a difference.



2012/4/25 feng zheng <zf5984599 [at] gmail>

hi, dear all:

When I use drbd, I found the write performance very slow, against
testing without drbd module.

1. the environment:
-) CentOS 5.6
-) 2.6.18 kernel
-) drbd 8.4.1
-) drbd.conf:
resource r0
{
protocol B;

net
{
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 512K;
}

disk
{
al-extents 3389;
}


on OSS211
{
device /dev/drbd0;
disk /dev/sdb1;
address MailScanner warning: numerical links are often malicious: 192.168.100.231:7788;
meta-disk internal;
}

on OSS213
{
device /dev/drbd0;
disk /dev/sde1;
address MailScanner warning: numerical links are often malicious: 192.168.100.213:7788;
meta-disk internal;
}

}


2. Test scenario:
*) without drbd module,
dd to write 1G stream into one disk, which formatted to ext3:
[para]# !echo
echo 3 > /proc/sys/vm/drop_caches
[para]# !dd
dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.9905 seconds, 95.4 MB/s

*) with drbd module,
dd 1G stream to the disk, which is ext3 too:
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----
ns:1260036 nr:0 dw:1260036 dr:297 al:330 bm:0 lo:0 pe:0 ua:0 ap:0
ep:1 wo:b oos:0
[para]#
[para]# dd if=/dev/zero of=test1 bs=1M count=1000 conv=fdatasync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 26.7392 seconds, 39.2 MB/s
[para]# cat /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] OSS21, 2012-04-16 21:38:36
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r-----

All the upper test writing disk are same. From the upper result, if i
use DRBD to test, the performance
is 39 MB/s; while if i do not use, the performance is about 95M/s.

3. My question is:
-) this write performance decays so large is normal or not?
I had read the following from the DRBD website:
"15.1. Hardware considerations:
.... A single, reasonably recent, SCSI or SAS disk will
typically allow streaming writes of roughly 40MB/s to the single disk."
But this is very slow.

-)if this is not normal, how can i turn this? is the config file
something not correct?

thanks a lot
BRs,
feng

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


ff at mpexnet

Apr 30, 2012, 7:38 AM

Post #9 of 10 (1732 views)
Permalink
Re: drbd write performance slow, per disk 40M/s by dd command [In reply to]

Hi,

On 04/26/2012 03:24 AM, zf5984599 wrote:
> I had turned off disk-barrier and disk-flushes in config file, but there
> is no difference.

did you make sure your tunings were applied correctly?
It's advisable to check using "drbdsetup 0 show" (if 0 is indeed the
minor number of the resource under test).

Your 100MBps result looks suspicious to me. Hum.

HTH,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


zf5984599 at gmail

May 3, 2012, 8:10 PM

Post #10 of 10 (1686 views)
Permalink
Re: drbd write performance slow, per disk 40M/s by dd command [In reply to]

hi,Felix, many thanks.

I used this cmd, here is my result:
machine 1:
resource r0 {
options {
}
net {
max-epoch-size 20000;
max-buffers 80000;
sndbuf-size 1048576; # bytes
}
_remote_host {
address ipv4 192.168.100.231:7788;
}
_this_host {
address ipv4 192.168.100.213:7788;
volume 0 {
device minor 0;
disk "/dev/sdd1";
meta-disk internal;
disk {
al-extents 3389;
}
}
}
}

machine 2:
resource r0 {
options {
}
net {
max-epoch-size 20000;
max-buffers 80000;
sndbuf-size 1048576; # bytes
}
_remote_host {
address ipv4 192.168.100.213:7788;
}
_this_host {
address ipv4 192.168.100.231:7788;
volume 0 {
device minor 0;
disk "/dev/sdc1";
meta-disk internal;
disk {
al-extents 3389;
}
}
}
}

and when I retest, the results of 3 sequential tests are:

# dd if=/dev/zero of=test3 bs=1M count=800 conv=fdatasync
800+0 records in
800+0 records out
838860800 bytes (839 MB) copied, 24.6033 seconds, 34.1 MB/s

# dd if=/dev/zero of=test3 bs=1M count=800 conv=fdatasync
800+0 records in
800+0 records out
838860800 bytes (839 MB) copied, 22.9287 seconds, 36.6 MB/s

# dd if=/dev/zero of=test3 bs=1M count=800 conv=fdatasync
800+0 records in
800+0 records out
838860800 bytes (839 MB) copied, 10.151 seconds, 82.6 MB/s

So, the first and second times write-performance is so slow,
but the 3rd performance is better ? what problem may induced this,
is my test method incorrect?

BRs,
feng









From: Felix Frank
Date: 2012-04-30 22:38
To: zf5984599
CC: drbd-user
Subject: Re: [DRBD-user] drbd write performance slow, per disk 40M/s by dd command
Hi,

On 04/26/2012 03:24 AM, zf5984599 wrote:
> I had turned off disk-barrier and disk-flushes in config file, but there
> is no difference.

did you make sure your tunings were applied correctly?
It's advisable to check using "drbdsetup 0 show" (if 0 is indeed the
minor number of the resource under test).

Your 100MBps result looks suspicious to me. Hum.

HTH,
Felix

DRBD users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.