Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DRBD: Users

Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6

 

 

DRBD users RSS feed   Index | Next | Previous | View Threaded


uwe.schuerkamp at nionex

Mar 12, 2012, 3:56 AM

Post #1 of 17 (2466 views)
Permalink
Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6

Hi folks,

I'm a DRBD newbie when it comes to administration, so please bear with
me a for a second while describing my problem:

- Setup: 2x COMPAQ ProLianz DL 385g7, Raid 6 over 8 900GB SAS disks on
internal p410i controller, initial RAID rebuild completed, 128 GB RAM,
2 x HexaCore CPU's

- OS: Debian Squeeze 6.0 64bit, 128GB RAM, DRBD 8.3.7 (from Repos)

- Dedicated gigabit link on eth1 for DRBD sync / replication, speed
around 100MB/sec using rsync or scp. The inferfaces are directly
connnected, there's no other network hardware involved, collision /
error stats on both ends seem fine and show no apparent problems.

show all output on drbd0:

# drbdsetup /dev/drbd0 show
disk {
size 0s _is_default; # bytes
on-io-error pass_on _is_default;
fencing dont-care _is_default;
max-bio-bvecs 0 _is_default;
}
net {
timeout 60 _is_default; # 1/10 seconds
max-epoch-size 2048 _is_default;
max-buffers 2048 _is_default;
unplug-watermark 128 _is_default;
connect-int 10 _is_default; # seconds
ping-int 10 _is_default; # seconds
sndbuf-size 0 _is_default; # bytes
rcvbuf-size 0 _is_default; # bytes
ko-count 0 _is_default;
cram-hmac-alg "sha1";
shared-secret "XXXXXXXXXXXXXXXXXXXXXXXXXXX"; (edited)
after-sb-0pri disconnect _is_default;
after-sb-1pri disconnect _is_default;
after-sb-2pri disconnect _is_default;
rr-conflict disconnect _is_default;
ping-timeout 5 _is_default; # 1/10 seconds
}
syncer {
rate 102400k; # bytes/second
after -1 _is_default;
al-extents 127 _is_default;
}
protocol C;
_this_host {
device minor 0;
disk "/dev/vg00/lvol0";
meta-disk internal;
address ipv4 192.168.168.200:7788;
}
_remote_host {
address ipv4 192.168.168.210:7788;
}


Some benchmarks on the mounted xfs filesystem when syncing is active:

# time dd oflag=direct if=/dev/zero of=/mnt/disk/speedtest bs=10M count=10
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 33.2383 s, 3.2 MB/s

Same fs, Syncing inactive:

# time dd oflag=direct if=/dev/zero of=/mnt/disk/speedtest bs=1024M count=8
8+0 records in
8+0 records out
8589934592 bytes (8.6 GB) copied, 27.3747 s, 314 MB/s

My Problem: I'm only getting 10MB/sec sustained throughput during
initial re-sync of the DRBD device (~ 5TB), no matter wether I mount
the xfs device or leave it unmounted. I've tried various resync rates
without success, RAID & interface speed seem fine on non-drbd
devices.

/proc/drbd shows: (sorry for the line length)
# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:48504579 nr:0 dw:9541797 dr:47365371 al:2346 bm:4890 lo:1
pe:2770 ua:1741 ap:0 ep:1 wo:b oos:3252837352

[>....................] sync'ed: 0.9% (3176596/3205188)M
finish: 57:25:48 speed: 15,604 (11,448) K/sec


I've asked around on the #drbd irc channel, and the friendly folks
over there pointed me to this mailing list to describe my problems
here in the hope someone more knowledgable can step in and help.

Please let me know if you need any other info and I'll provide what I
can. The system isn't in production yet, so load on both servers is
negligble.

All the best & thanks in advance for your comments,

Uwe

--
uwe.schuerkamp [at] nionex fon: [+49] 5242.91 - 4740, fax:-69 72
Hauptsitz: Avenwedder Str. 55, D-33311 Gütersloh, Germany
Registergericht Gütersloh HRB 4196, Geschäftsführer: H. Gosewehr, D. Suda
NIONEX --- Ein Unternehmen der Bertelsmann AG


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


misch at schwartzkopff

Mar 12, 2012, 5:25 AM

Post #2 of 17 (2398 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

> Hi folks,
>
> I'm a DRBD newbie when it comes to administration, so please bear with
> me a for a second while describing my problem:
>
(...)
>
> - Dedicated gigabit link on eth1 for DRBD sync / replication, speed
> around 100MB/sec using rsync or scp.
(...)
> # drbdsetup /dev/drbd0 show
(...)
> syncer {
> rate 102400k; # bytes/second
(...)
> }
(...)
> /proc/drbd shows: (sorry for the line length)
> # cat /proc/drbd
> version: 8.3.7 (api:88/proto:86-91)
> srcversion: EE47D8BF18AC166BE219757
> 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
> ns:48504579 nr:0 dw:9541797 dr:47365371 al:2346 bm:4890 lo:1
> pe:2770 ua:1741 ap:0 ep:1 wo:b oos:3252837352
>
> [>....................] sync'ed: 0.9% (3176596/3205188)M
> finish: 57:25:48 speed: 15,604 (11,448) K/sec
(...)


Hm,

somethings do not fit together. You configured a rate of 102M (Byte/s) and
when you ask the DRBD it tells you that your configured a rate of 11M (Byte/s)

Did your show us the /proc/drbd output of the above given config?

Greetings,

--
Dr. Michael Schwartzkopff
Guardinistr. 63
81375 München

Tel: (0163) 172 50 98
Fax: (089) 620 304 13
Attachments: signature.asc (0.19 KB)


uwe.schuerkamp at nionex

Mar 12, 2012, 6:17 AM

Post #3 of 17 (2391 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

On Mon, Mar 12, 2012 at 01:25:12PM +0100, Michael Schwartzkopff wrote:

> Hm,
>
> somethings do not fit together. You configured a rate of 102M (Byte/s) and
> when you ask the DRBD it tells you that your configured a rate of 11M (Byte/s)
>
> Did your show us the /proc/drbd output of the above given config?
>
Hello Michael,

here's the /proc/drbd output:

# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
r----
ns:2371132 nr:0 dw:0 dr:2378656 al:0 bm:144 lo:1 pe:2556 ua:1806
ap:0 ep:1 wo:d oos:3152530292
[>....................] sync'ed: 0.1% (3078640/3080948)M
finish: 59:10:08 speed: 14,628 (12,296) K/sec

I've been changing some disk params as recommended in the IRC (see
wo: entry), but sadly this has had no effect.

I've also tried using a different interface (one connected to a
gigabit switch), but sync speed was as low as before.

All the best & thanks for your comments,

Uwe

--
NIONEX --- Ein Unternehmen der Bertelsmann AG


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


misch at schwartzkopff

Mar 12, 2012, 6:28 AM

Post #4 of 17 (2393 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

> On Mon, Mar 12, 2012 at 01:25:12PM +0100, Michael Schwartzkopff wrote:
> > Hm,
> >
> > somethings do not fit together. You configured a rate of 102M (Byte/s)
> > and when you ask the DRBD it tells you that your configured a rate of
> > 11M (Byte/s)
> >
> > Did your show us the /proc/drbd output of the above given config?
>
> Hello Michael,
>
> here's the /proc/drbd output:
>
> # cat /proc/drbd
> version: 8.3.7 (api:88/proto:86-91)
> srcversion: EE47D8BF18AC166BE219757
> 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
> r----
> ns:2371132 nr:0 dw:0 dr:2378656 al:0 bm:144 lo:1 pe:2556 ua:1806
> ap:0 ep:1 wo:d oos:3152530292
> [>....................] sync'ed: 0.1% (3078640/3080948)M
> finish: 59:10:08 speed: 14,628 (12,296) K/sec

What about
drbdsetup /dev/drbd... show?

--
Dr. Michael Schwartzkopff
Guardinistr. 63
81375 München

Tel: (0163) 172 50 98
Fax: (089) 620 304 13
Attachments: signature.asc (0.19 KB)


uwe.schuerkamp at nionex

Mar 12, 2012, 6:57 AM

Post #5 of 17 (2388 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

On Mon, Mar 12, 2012 at 02:28:42PM +0100, Michael Schwartzkopff wrote:
> > On Mon, Mar 12, 2012 at 01:25:12PM +0100, Michael Schwartzkopff wrote:
> > > Hm,
> > >
> > > somethings do not fit together. You configured a rate of 102M (Byte/s)
> > > and when you ask the DRBD it tells you that your configured a rate of
> > > 11M (Byte/s)
> > >
> > > Did your show us the /proc/drbd output of the above given config?
> >
> > Hello Michael,
> >
> > here's the /proc/drbd output:
> >
> > # cat /proc/drbd
> > version: 8.3.7 (api:88/proto:86-91)
> > srcversion: EE47D8BF18AC166BE219757
> > 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
> > r----
> > ns:2371132 nr:0 dw:0 dr:2378656 al:0 bm:144 lo:1 pe:2556 ua:1806
> > ap:0 ep:1 wo:d oos:3152530292
> > [>....................] sync'ed: 0.1% (3078640/3080948)M
> > finish: 59:10:08 speed: 14,628 (12,296) K/sec
>
> What about
> drbdsetup /dev/drbd... show?
>
> --

Hello Michael,

I think I included this one with the first email, here's the current
setup:

# drbdsetup /dev/drbd0 show
disk {
size 0s _is_default; # bytes
on-io-error pass_on _is_default;
fencing dont-care _is_default;
no-disk-barrier ;
no-disk-flushes ;
no-md-flushes ;
max-bio-bvecs 0 _is_default;
}
net {
timeout 60 _is_default; # 1/10 seconds
max-epoch-size 2048 _is_default;
max-buffers 2048 _is_default;
unplug-watermark 128 _is_default;
connect-int 10 _is_default; # seconds
ping-int 10 _is_default; # seconds
sndbuf-size 0 _is_default; # bytes
rcvbuf-size 0 _is_default; # bytes
ko-count 0 _is_default;
cram-hmac-alg "sha1";
shared-secret "XXXXXXXXXXXXXXXXXXXX";
after-sb-0pri disconnect _is_default;
after-sb-1pri disconnect _is_default;
after-sb-2pri disconnect _is_default;
rr-conflict disconnect _is_default;
ping-timeout 5 _is_default; # 1/10 seconds
}
syncer {
rate 92160k; # bytes/second
after -1 _is_default;
al-extents 127 _is_default;
}
protocol C;
_this_host {
device minor 0;
disk "/dev/vg00/lvol0";
meta-disk internal;
address ipv4 192.168.168.200:7788;
}
_remote_host {
address ipv4 192.168.168.210:7788;
}


All the best, Uwe

--
uwe.schuerkamp [at] nionex fon: [+49] 5242.91 - 4740, fax:-69 72
Hauptsitz: Avenwedder Str. 55, D-33311 Gütersloh, Germany
Registergericht Gütersloh HRB 4196, Geschäftsführer: H. Gosewehr, D. Suda
NIONEX --- Ein Unternehmen der Bertelsmann AG


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


jsmith at argotec

Mar 12, 2012, 9:13 AM

Post #6 of 17 (2392 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

----- Original Message -----
> From: "Uwe Schuerkamp" <uwe.schuerkamp [at] nionex>
> To: "Michael Schwartzkopff" <misch [at] schwartzkopff>, drbd-user [at] lists
> Sent: Monday, March 12, 2012 9:57:29 AM
> Subject: Re: [DRBD-user] Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6
>
> On Mon, Mar 12, 2012 at 02:28:42PM +0100, Michael Schwartzkopff
> wrote:
> > > On Mon, Mar 12, 2012 at 01:25:12PM +0100, Michael Schwartzkopff
> > > wrote:
> > > > Hm,
> > > >
> > > > somethings do not fit together. You configured a rate of 102M
> > > > (Byte/s)
> > > > and when you ask the DRBD it tells you that your configured a
> > > > rate of
> > > > 11M (Byte/s)
> > > >
> > > > Did your show us the /proc/drbd output of the above given
> > > > config?
> > >
> > > Hello Michael,
> > >
> > > here's the /proc/drbd output:
> > >
> > > # cat /proc/drbd
> > > version: 8.3.7 (api:88/proto:86-91)
> > > srcversion: EE47D8BF18AC166BE219757
> > > 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
> > > r----
> > > ns:2371132 nr:0 dw:0 dr:2378656 al:0 bm:144 lo:1 pe:2556
> > > ua:1806
> > > ap:0 ep:1 wo:d oos:3152530292
> > > [>....................] sync'ed: 0.1% (3078640/3080948)M
> > > finish: 59:10:08 speed: 14,628 (12,296) K/sec
> >
> > What about
> > drbdsetup /dev/drbd... show?
> >
> > --
>
> Hello Michael,
>
> I think I included this one with the first email, here's the current
> setup:
>
> # drbdsetup /dev/drbd0 show
> disk {
> size 0s _is_default; # bytes
> on-io-error pass_on _is_default;
> fencing dont-care _is_default;
> no-disk-barrier ;
> no-disk-flushes ;
> no-md-flushes ;
> max-bio-bvecs 0 _is_default;
> }
> net {
> timeout 60 _is_default; # 1/10 seconds
> max-epoch-size 2048 _is_default;
> max-buffers 2048 _is_default;
> unplug-watermark 128 _is_default;
> connect-int 10 _is_default; # seconds
> ping-int 10 _is_default; # seconds
> sndbuf-size 0 _is_default; # bytes
> rcvbuf-size 0 _is_default; # bytes
> ko-count 0 _is_default;
> cram-hmac-alg "sha1";
> shared-secret "XXXXXXXXXXXXXXXXXXXX";

Does the rate change at all if you remove the cram-hmac-alg and shared-secret?

> after-sb-0pri disconnect _is_default;
> after-sb-1pri disconnect _is_default;
> after-sb-2pri disconnect _is_default;
> rr-conflict disconnect _is_default;
> ping-timeout 5 _is_default; # 1/10 seconds
> }
> syncer {
> rate 92160k; # bytes/second

Have you tried adjusting the syncer rate (temporarily) on the fly and see if it changes? i.e.:
drbdsetup /dev/drbd0 syncer -r 110M
drbdadm adjust 0

HTH
Jake

> after -1 _is_default;
> al-extents 127 _is_default;
> }
> protocol C;
> _this_host {
> device minor 0;
> disk "/dev/vg00/lvol0";
> meta-disk internal;
> address ipv4 192.168.168.200:7788;
> }
> _remote_host {
> address ipv4 192.168.168.210:7788;
> }

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


ff at mpexnet

Mar 12, 2012, 9:22 AM

Post #7 of 17 (2380 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

On 03/12/2012 05:13 PM, Jake Smith wrote:
> drbdsetup /dev/drbd0 syncer -r 110M

Good call.

> drbdadm adjust 0

Stressing this: Do the latter only *after* your test is done and you
want to rid yourself of the temporary adjustment.

Cheers,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


jsmith at argotec

Mar 12, 2012, 9:26 AM

Post #8 of 17 (2392 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

----- Original Message -----
> From: "Felix Frank" <ff [at] mpexnet>
> To: "Jake Smith" <jsmith [at] argotec>
> Cc: drbd-user [at] lists
> Sent: Monday, March 12, 2012 12:22:57 PM
> Subject: Re: [DRBD-user] Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6
>
> On 03/12/2012 05:13 PM, Jake Smith wrote:
> > drbdsetup /dev/drbd0 syncer -r 110M
>
> Good call.
>
> > drbdadm adjust 0
>
> Stressing this: Do the latter only *after* your test is done and you
> want to rid yourself of the temporary adjustment.


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
YES! Thank you Felix for catching that I didn't separate those commands (or even leave out the latter one!!)!!

Jake
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


uwe.schuerkamp at nionex

Mar 13, 2012, 2:04 AM

Post #9 of 17 (2375 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

On Mon, Mar 12, 2012 at 05:22:57PM +0100, Felix Frank wrote:
> On 03/12/2012 05:13 PM, Jake Smith wrote:
> > drbdsetup /dev/drbd0 syncer -r 110M
>
> Good call.
>
> > drbdadm adjust 0
>
> Stressing this: Do the latter only *after* your test is done and you
> want to rid yourself of the temporary adjustment.
>
> Cheers,
> Felix

Hi folks,

I've tried changing the syncer rate as you mentioned in the previous
post, but it hasn't changed the sync speed one bit (literally ;-).

It seems like I'm running out of options fast, the old hardware we're
intending to replace does 44MB/sec write speed in similar setup, but
on much inferior hardware, so I'm hoping to achieve at least the same
speeds once we'll migrate to the new servers.

For the record:

# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
r----
ns:873115000 nr:0 dw:2048 dr:873127461 al:1 bm:53289 lo:1 pe:3151
ua:1699 ap:0 ep:1 wo:d oos:2281801072
[====>...............] sync'ed: 25.6% (2228320/2992484)M
finish: 32:20:18 speed: 19,396 (12,256) K/sec


# drbdsetup /dev/drbd0 show
disk {
size 0s _is_default; # bytes
on-io-error pass_on _is_default;
fencing dont-care _is_default;
no-disk-barrier ;
no-disk-flushes ;
no-md-flushes ;
max-bio-bvecs 0 _is_default;
}
net {
timeout 60 _is_default; # 1/10 seconds
max-epoch-size 2048 _is_default;
max-buffers 2048 _is_default;
unplug-watermark 128 _is_default;
connect-int 10 _is_default; # seconds
ping-int 10 _is_default; # seconds
sndbuf-size 0 _is_default; # bytes
rcvbuf-size 0 _is_default; # bytes
ko-count 0 _is_default;
cram-hmac-alg "sha1";
shared-secret "XXXXX";
after-sb-0pri disconnect _is_default;
after-sb-1pri disconnect _is_default;
after-sb-2pri disconnect _is_default;
rr-conflict disconnect _is_default;
ping-timeout 5 _is_default; # 1/10 seconds
}
syncer {
rate 92160k; # bytes/second
after -1 _is_default;
al-extents 127 _is_default;
}
protocol C;
_this_host {
device minor 0;
disk "/dev/vg00/lvol0";
meta-disk internal;
address
ipv4 192.168.168.200:7788;
}
_remote_host {
address ipv4 192.168.168.210:7788;

All the best & thanks again for your help,

Uwe


--
uwe.schuerkamp [at] nionex fon: [+49] 5242.91 - 4740, fax:-69 72
Hauptsitz: Avenwedder Str. 55, D-33311 Gütersloh, Germany
Registergericht Gütersloh HRB 4196, Geschäftsführer: H. Gosewehr, D. Suda
NIONEX --- Ein Unternehmen der Bertelsmann AG


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


themsley at voiceflex

Mar 13, 2012, 2:07 AM

Post #10 of 17 (2367 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

Uwe Schuerkamp wrote:
> # cat /proc/drbd
> version: 8.3.7 (api:88/proto:86-91)
>

You could try upgrading from the rather ancient 8.3.7 that you currently
have to 8.3.12. There are quite a few bugs fixed between the two releases...


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


misch at schwartzkopff

Mar 13, 2012, 3:09 AM

Post #11 of 17 (2375 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

> On Mon, Mar 12, 2012 at 05:22:57PM +0100, Felix Frank wrote:
> > On 03/12/2012 05:13 PM, Jake Smith wrote:
> > > drbdsetup /dev/drbd0 syncer -r 110M
> >
> > Good call.
> >
> > > drbdadm adjust 0
> >
> > Stressing this: Do the latter only *after* your test is done and you
> > want to rid yourself of the temporary adjustment.
> >
> > Cheers,
> > Felix
>
> Hi folks,
>
> I've tried changing the syncer rate as you mentioned in the previous
> post, but it hasn't changed the sync speed one bit (literally ;-).
>
> It seems like I'm running out of options fast, the old hardware we're
> intending to replace does 44MB/sec write speed in similar setup, but
> on much inferior hardware, so I'm hoping to achieve at least the same
> speeds once we'll migrate to the new servers.
>
> For the record:
>
> # cat /proc/drbd
> version: 8.3.7 (api:88/proto:86-91)
> srcversion: EE47D8BF18AC166BE219757
> 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
> r----
> ns:873115000 nr:0 dw:2048 dr:873127461 al:1 bm:53289 lo:1 pe:3151
> ua:1699 ap:0 ep:1 wo:d oos:2281801072
> [====>...............] sync'ed: 25.6% (2228320/2992484)M
> finish: 32:20:18 speed: 19,396 (12,256) K/sec

Could you please show the /proc/drbd BEFORE and AFTER the drbdsetup command?
Perhaps you switch the sync speed several times.

--
Dr. Michael Schwartzkopff
Guardinistr. 63
81375 München

Tel: (0163) 172 50 98
Fax: (089) 620 304 13
Attachments: signature.asc (0.19 KB)


uwe.schuerkamp at nionex

Mar 13, 2012, 5:17 AM

Post #12 of 17 (2390 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

On Tue, Mar 13, 2012 at 11:09:05AM +0100, Michael Schwartzkopff wrote:

> > # cat /proc/drbd
> > version: 8.3.7 (api:88/proto:86-91)
> > srcversion: EE47D8BF18AC166BE219757
> > 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
> > r----
> > ns:873115000 nr:0 dw:2048 dr:873127461 al:1 bm:53289 lo:1 pe:3151
> > ua:1699 ap:0 ep:1 wo:d oos:2281801072
> > [====>...............] sync'ed: 25.6% (2228320/2992484)M
> > finish: 32:20:18 speed: 19,396 (12,256) K/sec
>
> Could you please show the /proc/drbd BEFORE and AFTER the drbdsetup command?
> Perhaps you switch the sync speed several times.
>


Here you go (snipped for brevity):
######################################################################
Current setting:

drbdsetup /dev/drbd0 show | grep rate
rate 30720k; # bytes/second

/proc/drbd on Slave node:

version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncTarget ro:Secondary/Secondary ds:Inconsistent/UpToDate C
r----
ns:0 nr:83468576 dw:83454884 dr:0 al:0 bm:5093 lo:3423 pe:57960
ua:3423 ap:0
ep:1 wo:b oos:2144478752
[>....................] sync'ed: 3.8% (2094216/2175716)M
finish: 46:07:34 speed: 12,912 (11,732) K/sec

######################################################################
Changing sync speed on master node:

# drbdsetup /dev/drbd0 syncer -r 110M
# drbdsetup /dev/drbd0 show | grep rate
rate 112640k; # bytes/second

/proc/drbd on client after waiting a couple of secs for stats to update:

version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncTarget ro:Secondary/Secondary ds:Inconsistent/UpToDate C
r----
ns:0 nr:84252128 dw:84237136 dr:0 al:0 bm:5141 lo:3748 pe:57829
ua:3748 ap:0
ep:1 wo:b oos:2143696500
[>....................] sync'ed: 3.8% (2093452/2175716)M
finish: 41:21:07 speed: 14,360 (11,732) K/sec

######################################################################

As for updating, I'd rather stay with the version from the Debian
repos if possible. The old system runs with a much older drbd version
and performance there seems fine for the moment.

All the best & thanks for your help,

Uwe

--
NIONEX --- Ein Unternehmen der Bertelsmann AG


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


ff at mpexnet

Mar 13, 2012, 5:27 AM

Post #13 of 17 (2367 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

On 03/13/2012 01:17 PM, Uwe Schuerkamp wrote:
> The old system runs with a much older drbd version
> and performance there seems fine for the moment.

Jake made a remark on cram-hmac-alg and shared-secret earlier that has
apparently been missed so far.

Are those present in your old setup? Does removing them make any difference?
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


uwe.schuerkamp at nionex

Mar 13, 2012, 5:55 AM

Post #14 of 17 (2383 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

Hi folks,

I've now enabled the write cache on the slave being synced to (using
HP's hpacucli tool), the syncing speed has picked up nicely to around
100MB/sec:

version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncTarget ro:Secondary/Secondary ds:Inconsistent/UpToDate C
r----
ns:0 nr:206092724 dw:206092524 dr:0 al:0 bm:12578 lo:51 pe:53950
ua:50 ap:0
ep:1 wo:b oos:2021841112
[>...................] sync'ed: 9.3% (1974452/2175716)M
finish: 3:47:22 speed: 148,028 (21,668) K/sec



All the best & thanks again for your help in this matter,

Uwe



--
uwe.schuerkamp [at] nionex fon: [+49] 5242.91 - 4740, fax:-69 72
Hauptsitz: Avenwedder Str. 55, D-33311 Gütersloh, Germany
Registergericht Gütersloh HRB 4196, Geschäftsführer: H. Gosewehr, D. Suda
NIONEX --- Ein Unternehmen der Bertelsmann AG


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


uwe.schuerkamp at nionex

Mar 13, 2012, 6:14 AM

Post #15 of 17 (2413 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

One more thing and then I'll leave it at this:

Disabling the dwc on the slave node does NOT slow down the sync, it's
still ticking along happily at 100MB/sec. Can anyone tell me if HP's
RAID6 controller trial licenses have some sort of bandwidth limit
imposed on them? It's hard to imagine, but I've seen worse in my
days. ;-)

Today we installed permanent RAID6 licenses on both boxes, that's why
I'm suspecting the above might actually be the case.

from hpacucli:

=> controller slot=0 show config detail

Smart Array P410i in Slot 0 (Embedded)
<snip snip>
Cache Status: OK
Accelerator Ratio: 25% Read / 75% Write
Drive Write Cache: Disabled
Total Cache Size: 1024 MB
No-Battery Write Cache: Disabled
Cache Backup Power Source: Capacitors
Battery/Capacitor Count: 1
Battery/Capacitor Status: OK
SATA NCQ Supported: True

And from proc/drbd:


version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncTarget ro:Secondary/Secondary ds:Inconsistent/UpToDate C
r----
ns:0 nr:104446148 dw:104446000 dr:0 al:0 bm:6374 lo:38 pe:58477
ua:37 ap:0 e
p:1 wo:b oos:1895045524
[>...................] sync'ed: 5.3% (1850628/1952628)M
finish: 4:04:42 speed: 129,052 (107,232) K/sec



All the best, Uwe




--
NIONEX --- Ein Unternehmen der Bertelsmann AG


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


arnold at arnoldarts

Mar 13, 2012, 1:01 PM

Post #16 of 17 (2382 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

On Tuesday 13 March 2012 14:14:37 Uwe Schuerkamp wrote:
> One more thing and then I'll leave it at this:
>
> Disabling the dwc on the slave node does NOT slow down the sync, it's
> still ticking along happily at 100MB/sec. Can anyone tell me if HP's
> RAID6 controller trial licenses have some sort of bandwidth limit
> imposed on them? It's hard to imagine, but I've seen worse in my
> days. ;-)
>
> Today we installed permanent RAID6 licenses on both boxes, that's why
> I'm suspecting the above might actually be the case.

There goes another reason why we use software-raid even when the controller
has its own raid-modi...

Do they also charge you a data-rescue license fee when your controller brakes
and you have to rescue the data from your disks with a replacement-controller?

Have fun,

Arnold
Attachments: signature.asc (0.19 KB)


uwe.schuerkamp at nionex

Mar 14, 2012, 1:27 AM

Post #17 of 17 (2362 views)
Permalink
Re: Please help with slow Replication / Syncing on debian squeeze 6.0, 8-disk RAID6 [In reply to]

On Tue, Mar 13, 2012 at 09:01:09PM +0100, Arnold Krille wrote:
>
> Do they also charge you a data-rescue license fee when your controller brakes
> and you have to rescue the data from your disks with a replacement-controller?
>

Hi Arnold,

it's "only" RAID6 / RAID60 functionality that's charged for extra, and
I'm not entirely sure the trial license that I used before was to
blame for the slow syncing speed as performance was great with DRBD
*inactive* on the slave node.

All the best,

Uwe


--
NIONEX --- Ein Unternehmen der Bertelsmann AG


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

DRBD users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.