Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DRBD: Users

a latency problem ( kernel:3.0.13 and drbd-8.3.12 )

 

 

DRBD users RSS feed   Index | Next | Previous | View Threaded


akos.csurai at ericsson

May 10, 2012, 9:00 AM

Post #1 of 5 (1321 views)
Permalink
a latency problem ( kernel:3.0.13 and drbd-8.3.12 )

Hello,

I have a backup time degradation problem.
In a nutshell (it looks a write latency performance):
- the old IO pair ( SLES10 + drbd 8.2.6 ) can finish the task in 10 seconds
- the new IO pair ( SLES11SP2 + drbd 8.3.12 ) can do it in 16 seconds only.

I can force the old 8.2.6 configuration to slow down if I comment
out
no-disk-flushes ;
no-md-flushes ;
from the disk section in the config file.

When I change the 8.3.12 configuration, it has no effect on speed.
I can prove this "slow down" with the 'standard' drbd latency measurement command:

dd if=/dev/zero of=/opt/mirror/perftest/sles.latency-bs512-count4096 bs=512 count=4096 oflag=direct

In this special case the new system is 4 times slower!

Questions:
A) - Is it a known issue for the 8.3.12 drbd ? ( no-disk-flushes no-md-flushes has no real effect )
B) - I have checked the sysctl and io-scheduler settings and tried to
tune the SLES IO performance with no success.
// I used :
http://www.suse.com/documentation/sles11/pdfdoc/book_sle_tuning/book_sle_tuning.pdf
// and alsohttp://www.drbd.org/users-guide/s-throughput-tuning.html
Is there any hint to try?

I must admit that the throughput test shows 20% advantage,
but I do not mind if it becomes worse, if I can tune up the latency.

Thanks your help in advanced:

BR,
Akos

Here is my config if anyone need it
-----------------------------------------------
drbd IO configuration on a SLES10 installation:
-----------------------------------------------
# drbdsetup /dev/drbd0 show
disk {
size 0s _is_default; # bytes
on-io-error detach;
fencing dont-care _is_default;
no-disk-flushes ;
no-md-flushes ;
max-bio-bvecs 0 _is_default;
}
net {
timeout 60 _is_default; # 1/10 seconds
max-epoch-size 2048 _is_default;
max-buffers 2048 _is_default;
unplug-watermark 128 _is_default;
connect-int 10 _is_default; # seconds
ping-int 10 _is_default; # seconds
sndbuf-size 131070 _is_default; # bytes
ko-count 0 _is_default;
after-sb-0pri discard-least-changes;
after-sb-1pri violently-as0p;
after-sb-2pri violently-as0p;
rr-conflict disconnect _is_default;
ping-timeout 5 _is_default; # 1/10 seconds
}
syncer {
rate 12288k; # bytes/second
after -1 _is_default;
al-extents 127 _is_default;
verify-alg "";
}
protocol C;
_this_host {
device "/dev/drbd0";
disk "/dev/part4";
meta-disk internal;
address 172.16.32.13:7788;
}
_remote_host {
address 172.16.32.9:7788;
}
---------------------------------------------------
drbd IO configuration on a SLES11_SP2 installation:
---------------------------------------------------
# drbdsetup /dev/drbd0 show
disk {
size 0s _is_default; # bytes
on-io-error detach;
fencing dont-care _is_default;
no-disk-flushes ;
no-md-flushes ;
max-bio-bvecs 0 _is_default;
disk-timeout 0 _is_default; # 1/10 seconds
}
net {
timeout 60 _is_default; # 1/10 seconds
max-epoch-size 2048 _is_default;
max-buffers 2048 _is_default;
unplug-watermark 128 _is_default;
connect-int 10 _is_default; # seconds
ping-int 10 _is_default; # seconds
sndbuf-size 0 _is_default; # bytes
rcvbuf-size 0 _is_default; # bytes
ko-count 0 _is_default;
after-sb-0pri discard-least-changes;
after-sb-1pri violently-as0p;
after-sb-2pri violently-as0p;
rr-conflict disconnect _is_default;
ping-timeout 5 _is_default; # 1/10 seconds
on-congestion block _is_default;
congestion-fill 0s _is_default; # byte
congestion-extents 127 _is_default;
}
syncer {
rate 12288k; # bytes/second
after -1 _is_default;
al-extents 127 _is_default;
on-no-data-accessible io-error _is_default;
c-plan-ahead 0 _is_default; # 1/10 seconds
c-delay-target 10 _is_default; # 1/10 seconds
c-fill-target 0s _is_default; # bytes
c-max-rate 102400k _is_default; # bytes/second
c-min-rate 4096k _is_default; # bytes/second
}
protocol C;
_this_host {
device minor 0;
disk "/dev/part4";
meta-disk internal;
address ipv4 172.16.0.13:7788;
}
_remote_host {
address ipv4 172.16.0.9:7788;
}




--
This communication is confidential and intended solely for the addressee(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you believe this message has been sent to you in error, please notify the sender by replying to this transmission and delete the message without disclosing it. Thank you.
E-mail including attachments is susceptible to data corruption, interception, unauthorized amendment, tampering and viruses, and we only send and receive emails on the basis that we are not liable for any such corruption, interception, amendment, tampering or viruses or any consequences thereof.

Ericsson Magyarország Kft., Székhely: 1097 Budapest, Könyves Kálmán krt. 11. B. épület. Nyilvántartó cégbíróság: Fővárosi Bíróság. Cégjegyzékszám: 01-09-070937

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


zweiss at scout

May 11, 2012, 11:21 AM

Post #2 of 5 (1221 views)
Permalink
Re: a latency problem ( kernel:3.0.13 and drbd-8.3.12 ) [In reply to]

On May 10, 2012, at 11:00 AM, Csurai Akos wrote:

> Hello,
>
> I have a backup time degradation problem.
> In a nutshell (it looks a write latency performance):
>

Hi,

I've also had problems with DRBD incurring (unreasonable, I would say) additional write latency on my systems. I've been watching this list since last July, and have seen a number of people report similar-sounding problems, but have yet to see any real explanation of why this happens or any solution to the problem.

The earliest version I've used was 8.3.7, so I can't really speak to whether or not 8.2.6 was any better in this regard. (I did try updating to 8.4.0, but that didn't seem to help and introduced other problems, so I returned to 8.3.x.)

For what it's worth, when I updated my servers from RHEL5 to RHEL6 (so a 2.6.18-based kernel to a 2.6.32-based one), it "felt" like things may have improved slightly, but not a whole lot (I say "felt" because I don't have any hard before/after numbers on it).


Zev Weiss

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


akos.csurai at ericsson

May 14, 2012, 12:57 AM

Post #3 of 5 (1229 views)
Permalink
Re: a latency problem ( kernel:3.0.13 and drbd-8.3.12 ) [In reply to]

Thanks your comment Zev,

We are about to elaborate an latency-intensive measurement scenario
with both default SLES configurations (SLES10 and SLES11SP2)
and we plan to share the result with this list.

Akos

p.s.

As I reckon 8.2.6 drbd can not be compiled with the 3.0.x kernel,
and I guess that 8.3.12 won't be compiled with 2.16.x kernel too
I hope we can find some kernel (or NFS or drbd) configuration option
that help a bit.

On 05/11/12 20:21, Zev Weiss wrote:
> On May 10, 2012, at 11:00 AM, Csurai Akos wrote:
>
>> Hello,
>>
>> I have a backup time degradation problem.
>> In a nutshell (it looks a write latency performance):
>>
> Hi,
>
> I've also had problems with DRBD incurring (unreasonable, I would say) additional write latency on my systems. I've been watching this list since last July, and have seen a number of people report similar-sounding problems, but have yet to see any real explanation of why this happens or any solution to the problem.
>
> The earliest version I've used was 8.3.7, so I can't really speak to whether or not 8.2.6 was any better in this regard. (I did try updating to 8.4.0, but that didn't seem to help and introduced other problems, so I returned to 8.3.x.)
>
> For what it's worth, when I updated my servers from RHEL5 to RHEL6 (so a 2.6.18-based kernel to a 2.6.32-based one), it "felt" like things may have improved slightly, but not a whole lot (I say "felt" because I don't have any hard before/after numbers on it).
>
>
> Zev Weiss
>
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
>


--
This communication is confidential and intended solely for the addressee(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you believe this message has been sent to you in error, please notify the sender by replying to this transmission and delete the message without disclosing it. Thank you.
E-mail including attachments is susceptible to data corruption, interception, unauthorized amendment, tampering and viruses, and we only send and receive emails on the basis that we are not liable for any such corruption, interception, amendment, tampering or viruses or any consequences thereof.

Ericsson Magyarország Kft., Székhely: 1097 Budapest, Könyves Kálmán krt. 11. B. épület. Nyilvántartó cégbíróság: Fővárosi Bíróság. Cégjegyzékszám: 01-09-070937

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


lars.ellenberg at linbit

May 15, 2012, 5:58 AM

Post #4 of 5 (1206 views)
Permalink
Re: a latency problem ( kernel:3.0.13 and drbd-8.3.12 ) [In reply to]

On Mon, May 14, 2012 at 09:57:10AM +0200, Csurai Akos wrote:
> Thanks your comment Zev,
>
> We are about to elaborate an latency-intensive measurement scenario
> with both default SLES configurations (SLES10 and SLES11SP2)
> and we plan to share the result with this list.
>
> Akos
>
> p.s.
>
> As I reckon 8.2.6 drbd can not be compiled with the 3.0.x kernel,
> and I guess that 8.3.12 won't be compiled with 2.16.x kernel too

8.3. (btw we are at 8.3.13 now) is supposed to work on all kernels
from sles9 (2.6.5 plus suse patches, I think) to current upstream.

You could post your full drbd config,
and some more details about your IO stack.

Or contact LINBIT or others who setup and tune DRBD setups everyday.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


lars.ellenberg at linbit

May 16, 2012, 5:44 AM

Post #5 of 5 (1202 views)
Permalink
Re: a latency problem ( kernel:3.0.13 and drbd-8.3.12 ) [In reply to]

On Thu, May 10, 2012 at 06:00:59PM +0200, Csurai Akos wrote:
> # drbdsetup /dev/drbd0 show
> disk {
> size 0s _is_default; # bytes
> on-io-error detach;
> fencing dont-care _is_default;
> no-disk-flushes ;
> no-md-flushes ;

You may want to add
no-disk-barrier;

> max-bio-bvecs 0 _is_default;
> }

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

DRBD users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.