Mailing List Archive: DRBD: Users

Performance regression with DRBD 8.3.12 and newer

Index | Next | Previous | View Threaded

lists-drbd at wspse

Jun 11, 2012, 9:35 AM

Post #1 of 11 (957 views)
Permalink

Performance regression with DRBD 8.3.12 and newer

Hi at all,

it seems to me that there is some kind of write performance regression
that started with DRBD 8.3.12, and is still present in 8.3.13 and 8.4.1.

First some details about the testing hardware:
1. two identical servers with an Intel Server S2600CP mainboard
2. each server has 2 physical Xeon E5-2620 CPUs and 64 GB Ram
3. the network connection is a patchcable between the on-board
Intel dual I350 Gigabit network card (so no active hardware/switches
between both nodes)
4. the servers system is on a separate SSD, for DRBD performance tests
I used a dedicated harddisk, there is no other load (I/O, CPU or
network) on the server while testing
5. harddisk is a 3 TB Seagate ST3000DM001-9YN166 (firmware CC4H)
connected to a Sata III port on the mainboard.
6. OS is a fully patched CentOS 6.2 with DRBD 8.3 packages from elrepo.

I started with the recent DRBD 8.3.13, trying to optimize the setup for
later production usage. On each harddisk on both sides I created a 3 TB
LVM partition (using gdisk, creating a GPT partition table), created a
volume group and two seperate logical volumes in it (one for usage
without DRBD and one for usage with DRBD to measure differences).

Both partitions have a size of 50G. One partition was put setup as DRBD
block device with more or less default values (protocol C, al-extents
3389 and sndbuf-size of 0 (auto)). Both (the real partition, as well as
the DRBD blockdevice) were formatted using ext4 with default values and
then mounted.

I started running bonnie++ 1.96 with different values on both
filesystems. Most of my later tests were made with
"bonnie++ -u root -d /mnt/drbd/ -r 8192 -b -n 0"
and looking at the sequential output block results there.

Writing on the physical volume I see a block-write rate of around
160-180 MBytes/s (which is to be expected).

I then checked the raw tcp network performance with iperf seeing a rate
of 992 MBit/s.

I then exported the filesystem on the non-drbd volume using NFS, running
bonnie++ on the NFS mount. I reached between 100 and 105 MBytes/s there.
That is without any further tuning and a reasonable value for a Gigabit
network.

Then the drbd volume was mounted and I run bonnie++ on it. I only got
values between 46 and 52 MBytes/s in several runs, which is only half of
the expected rate.

I spend a lot of time tuning the following parameters:
1. setting MTU from 1500 to 9000 on both sides
2. switched from CFQ to Deadline scheduler
3. increased net.ipv4.tcp_*mem values to 131072 131072 10485760
4. tried several settings for the deadline scheduler (disabled
front_merges, changed read/write-expire, etc)
5. set max-buffers, max-epoch-size and unplug-watermark to 8000
6. mounted my filesystem with barriers=0
7. tried bonnie++ with the secondard DRBD node disconnected
8. reformatted the filesystem with ext2 (this is the only case were I
saw an actually different value: the write performance dropped to 38
MByte/s)
9. added "no-tcp-cork" to DRBDs config
10. tries "use-bmbv" in DRBD
11. switched from protocol C to protocol A
12. finally removed the LVM layer by recreating the partitiontable on
both sides using parted (still GPT table) and a physical device for
DRBD (again 50 GB)

Not one of these tweaks changed my write performace, which left me
puzzled. I also removed the filesystem from the equation, running "dd"
on the drbd blockdevice with several blocksizes and different oflags
(direct, fsync, etc), but that also did not bring a write performance
above 50 MByte/s.

Finally I thought about trying DRBD 8.4.1, but decided to first try a
different version without changing the metadata. Therefore I downloaded
the oldest RHEL6 package (version 8.3.9) and redid my tests with an ext4
formatted drbddevice.

The result was actual different. Just by downgrading from 8.3.13 to
8.3.9 the write performance doubled and reached 100 MByte/s. That is the
value I expected on the first place and could nowhere archive with
8.3.13.

I then updated to 8.3.10 and then to 8.3.11. The results were still good,
reaching between 100 and 105 MByte/s.

Finally I installed DRBD 8.3.12: and back was the low write performance
(54 MByte/s). I cross checked with DRBD 8.4.1, but the performance was
still low (50 MByte/s).

As last check I mixed the versions:
1. 8.3.12 on the primary side and 8.3.11 on the secondary side gave me a
throughput of 53 MByte/s (still low)
2. 8.3.11 on the primary and 8.3.12 on the secondary side resulted in a
new value: 70 MByte/s (that is slightly higher than any value I could
archieve with both 8.3.12er versions, but still a big factor lower
than it should be).

In conclusion: all versions up to and including 8.3.11 give me a good
and to be expected write performace. Starting with 8.3.12 something did
break, leaving me with only half of the possible write throughput.

I should mention that the read performance was never affected and always
high.

I checked the changelog for 8.3.12, but nothing obviously struck me.
Also diffing the sourcetrees 8.3.11->8.3.12 I did not find any obvious.

So I want to ask here: anyone seeing that problem too, or maybe having a
clue what is going on here. As far as I see I already turned any knob
that should have an effect, but nothing gives me a good write
performance with 8.3.12 and later.

Maybe I missed something very obvious, but then I do not understand why
just downgrading the DRBD version doubles the write performance.

So, please advise :) I am happy to provide any additional details that
might be needed.

Regards,
Matthias

lists-drbd at wspse

Jun 11, 2012, 1:14 PM

Post #2 of 11 (884 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

On Mon, Jun 11, 2012 at 06:35:18PM +0200, Matthias Hensler wrote:
> [...]
> I checked the changelog for 8.3.12, but nothing obviously struck me.
> Also diffing the sourcetrees 8.3.11->8.3.12 I did not find any
> obvious.

Let me follow up on this myself. As suggested on IRC I tried to build
drbd from source, just to take the elrepo packages from the equation.

So I started with DRBD 8.3.13, and as expected I had a low performance.

Then I tried 8.3.11, and I also had a low performance (although 8.3.11
from elrepo worked fine).

That left me puzzled for a while, since I examined the elrepo packages
more closely. As it seemed, all working drbd versions where build on
2.6.32-71, while all broken versions where build on 2.6.32-220.

So, I installed the old el6 2.6.32-71 kernel (took me a while to find
it, since it was removed from nearly all archives) and its devel
package, booted into that kernel and build two new versions from source:
8.3.11 and 8.3.13. Then I booted back to 2.6.32-220.

First try with my selfcompiled 8.3.11 modules: everything is fine.
Second try with my selfcompiled 8.3.13 modules: still everything is
fine.

Indeed, the problem lies within the kernel version used to build the
drbd.ko module. I double checked by using all userland tools from 8.3.13
elrepo build together with my drbd.ko build on 2.6.32-71 (but run from
2.6.32-220).

Just to be clear: all tests were made with kernel 2.6.32-220, and the
userland version does not matter.

drbd.ko | 8.3.11 | 8.3.13
---------------------+--------+-------
build on 2.6.32-71 | good | good
build on 2.6.32-220 | bad | bad

So, how to debug this further? I would suspect looking at the symbols of
both modules might give a clue?

Regards,
Matthias

florian at hastexo

Jun 11, 2012, 1:31 PM

Post #3 of 11 (882 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

On 06/11/12 22:14, Matthias Hensler wrote:
> On Mon, Jun 11, 2012 at 06:35:18PM +0200, Matthias Hensler wrote:
>> [...]
>> I checked the changelog for 8.3.12, but nothing obviously struck me.
>> Also diffing the sourcetrees 8.3.11->8.3.12 I did not find any
>> obvious.
>
> Let me follow up on this myself. As suggested on IRC I tried to build
> drbd from source, just to take the elrepo packages from the equation.
>
> So I started with DRBD 8.3.13, and as expected I had a low performance.
>
> Then I tried 8.3.11, and I also had a low performance (although 8.3.11
> from elrepo worked fine).
>
> That left me puzzled for a while, since I examined the elrepo packages
> more closely. As it seemed, all working drbd versions where build on
> 2.6.32-71, while all broken versions where build on 2.6.32-220.
>
>
> So, I installed the old el6 2.6.32-71 kernel (took me a while to find
> it, since it was removed from nearly all archives) and its devel
> package, booted into that kernel and build two new versions from source:
> 8.3.11 and 8.3.13. Then I booted back to 2.6.32-220.
>
> First try with my selfcompiled 8.3.11 modules: everything is fine.
> Second try with my selfcompiled 8.3.13 modules: still everything is
> fine.
>
> Indeed, the problem lies within the kernel version used to build the
> drbd.ko module. I double checked by using all userland tools from 8.3.13
> elrepo build together with my drbd.ko build on 2.6.32-71 (but run from
> 2.6.32-220).
>
> Just to be clear: all tests were made with kernel 2.6.32-220, and the
> userland version does not matter.
>
> drbd.ko | 8.3.11 | 8.3.13
> ---------------------+--------+-------
> build on 2.6.32-71 | good | good
> build on 2.6.32-220 | bad | bad
>
>
> So, how to debug this further? I would suspect looking at the symbols of
> both modules might give a clue?

As a knee-jerk response based on a hunch -- you've been warned :) --,
this could be related to the BIO_RW_BARRIER vs. FLUSH/FUA dance that the
RHEL 6 kernel has been doing between the initial RHEL 6 release, and
more recent updates (when they've been backporting the "let's kill
barriers" upstream changes from post-2.6.32).

Try configuring your disk section with no-disk-barrier, no-disk-flushes
and no-md-flushes (in both configurations) and see if your kernel module
change still makes a difference.

Of course, in production you should only use those options if you have
no volatile caches involved in the I/O path.

Not sure if this is useful, but I sure hope it is. :)

Cheers,
Florian

--
Need help with High Availability?
http://www.hastexo.com/now
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

lists at alteeve

Jun 11, 2012, 1:54 PM

Post #4 of 11 (880 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

On 06/11/2012 04:31 PM, Florian Haas wrote:
> On 06/11/12 22:14, Matthias Hensler wrote:
>> On Mon, Jun 11, 2012 at 06:35:18PM +0200, Matthias Hensler wrote:
>>> [...]
>>> I checked the changelog for 8.3.12, but nothing obviously struck me.
>>> Also diffing the sourcetrees 8.3.11->8.3.12 I did not find any
>>> obvious.
>>
>> Let me follow up on this myself. As suggested on IRC I tried to build
>> drbd from source, just to take the elrepo packages from the equation.
>>
>> So I started with DRBD 8.3.13, and as expected I had a low performance.
>>
>> Then I tried 8.3.11, and I also had a low performance (although 8.3.11
>> from elrepo worked fine).
>>
>> That left me puzzled for a while, since I examined the elrepo packages
>> more closely. As it seemed, all working drbd versions where build on
>> 2.6.32-71, while all broken versions where build on 2.6.32-220.
>>
>>
>> So, I installed the old el6 2.6.32-71 kernel (took me a while to find
>> it, since it was removed from nearly all archives) and its devel
>> package, booted into that kernel and build two new versions from source:
>> 8.3.11 and 8.3.13. Then I booted back to 2.6.32-220.
>>
>> First try with my selfcompiled 8.3.11 modules: everything is fine.
>> Second try with my selfcompiled 8.3.13 modules: still everything is
>> fine.
>>
>> Indeed, the problem lies within the kernel version used to build the
>> drbd.ko module. I double checked by using all userland tools from 8.3.13
>> elrepo build together with my drbd.ko build on 2.6.32-71 (but run from
>> 2.6.32-220).
>>
>> Just to be clear: all tests were made with kernel 2.6.32-220, and the
>> userland version does not matter.
>>
>> drbd.ko | 8.3.11 | 8.3.13
>> ---------------------+--------+-------
>> build on 2.6.32-71 | good | good
>> build on 2.6.32-220 | bad | bad
>>
>>
>> So, how to debug this further? I would suspect looking at the symbols of
>> both modules might give a clue?
>
> As a knee-jerk response based on a hunch -- you've been warned :) --,
> this could be related to the BIO_RW_BARRIER vs. FLUSH/FUA dance that the
> RHEL 6 kernel has been doing between the initial RHEL 6 release, and
> more recent updates (when they've been backporting the "let's kill
> barriers" upstream changes from post-2.6.32).
>
> Try configuring your disk section with no-disk-barrier, no-disk-flushes
> and no-md-flushes (in both configurations) and see if your kernel module
> change still makes a difference.
>
> Of course, in production you should only use those options if you have
> no volatile caches involved in the I/O path.
>
> Not sure if this is useful, but I sure hope it is. :)
>
> Cheers,
> Florian
>

Oh! Please let me know if this works. :)

digimer

--
Digimer
Papers and Projects: https://alteeve.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

lists-drbd at wspse

Jun 11, 2012, 2:00 PM

Post #5 of 11 (884 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

On Mon, Jun 11, 2012 at 10:31:16PM +0200, Florian Haas wrote:
> On 06/11/12 22:14, Matthias Hensler wrote:
> > Indeed, the problem lies within the kernel version used to build the
> > drbd.ko module. I double checked by using all userland tools from 8.3.13
> > elrepo build together with my drbd.ko build on 2.6.32-71 (but run from
> > 2.6.32-220).
> >
> > Just to be clear: all tests were made with kernel 2.6.32-220, and the
> > userland version does not matter.
> >
> > drbd.ko | 8.3.11 | 8.3.13
> > ---------------------+--------+-------
> > build on 2.6.32-71 | good | good
> > build on 2.6.32-220 | bad | bad
> >
> >
> > So, how to debug this further? I would suspect looking at the symbols of
> > both modules might give a clue?
>
> As a knee-jerk response based on a hunch -- you've been warned :) --,
> this could be related to the BIO_RW_BARRIER vs. FLUSH/FUA dance that the
> RHEL 6 kernel has been doing between the initial RHEL 6 release, and
> more recent updates (when they've been backporting the "let's kill
> barriers" upstream changes from post-2.6.32).

OK.

> Try configuring your disk section with no-disk-barrier, no-disk-flushes
> and no-md-flushes (in both configurations) and see if your kernel module
> change still makes a difference.

Just did that:

Using the drbd.ko build on 2.6.32-71 shows minor increase in
performance (108,5 MByte/s, so some 5% more or so).

Using the drbd.ko build on 2.6.32-220.17.1 now finally brings the
expected performance (same as with the 2.6.32-71 built).

> Of course, in production you should only use those options if you have
> no volatile caches involved in the I/O path.

Yes, that is clear. I did not plan to disable barriers, as the
bottleneck in my setup should be clearly the network.

> Not sure if this is useful, but I sure hope it is. :)

Well, what does that mean: are the modules build on 2.6.32-71 broken in
a way that they do not use barriers (and therefore dangerous to use), or
is everything fine with the 2.6.32-71 builds and just building on a
newer kernel produces broken modules?

Regards,
Matthias

sthomas at optionshouse

Jun 11, 2012, 2:15 PM

Post #6 of 11 (887 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

On 06/11/2012 03:31 PM, Florian Haas wrote:

> Try configuring your disk section with no-disk-barrier,
> no-disk-flushes and no-md-flushes (in both configurations) and see if
> your kernel module change still makes a difference.

FWIW, we saw exactly this behavior on newer 3.2.x kernels as well. With
XFS and our RAID controllers being capacitor-backed, we disabled
barriers and flushes, and that brought performance back to expected levels.

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas [at] optionshouse

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

lists-drbd at wspse

Jun 11, 2012, 2:29 PM

Post #7 of 11 (877 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

On Mon, Jun 11, 2012 at 11:00:09PM +0200, Matthias Hensler wrote:
> On Mon, Jun 11, 2012 at 10:31:16PM +0200, Florian Haas wrote:
> > On 06/11/12 22:14, Matthias Hensler wrote:
> > > Indeed, the problem lies within the kernel version used to build the
> > > drbd.ko module. I double checked by using all userland tools from 8.3.13
> > > elrepo build together with my drbd.ko build on 2.6.32-71 (but run from
> > > 2.6.32-220).
> > >
> > > Just to be clear: all tests were made with kernel 2.6.32-220, and the
> > > userland version does not matter.
> > >
> > > drbd.ko | 8.3.11 | 8.3.13
> > > ---------------------+--------+-------
> > > build on 2.6.32-71 | good | good
> > > build on 2.6.32-220 | bad | bad
> > >
> > >
> > > So, how to debug this further? I would suspect looking at the symbols of
> > > both modules might give a clue?
> >
> > As a knee-jerk response based on a hunch -- you've been warned :) --,
> > this could be related to the BIO_RW_BARRIER vs. FLUSH/FUA dance that the
> > RHEL 6 kernel has been doing between the initial RHEL 6 release, and
> > more recent updates (when they've been backporting the "let's kill
> > barriers" upstream changes from post-2.6.32).
>
> OK.
>
> > Try configuring your disk section with no-disk-barrier, no-disk-flushes
> > and no-md-flushes (in both configurations) and see if your kernel module
> > change still makes a difference.
>
> Just did that:
>
> Using the drbd.ko build on 2.6.32-71 shows minor increase in
> performance (108,5 MByte/s, so some 5% more or so).
>
> Using the drbd.ko build on 2.6.32-220.17.1 now finally brings the
> expected performance (same as with the 2.6.32-71 built).
>
> > Of course, in production you should only use those options if you have
> > no volatile caches involved in the I/O path.
>
> Yes, that is clear. I did not plan to disable barriers, as the
> bottleneck in my setup should be clearly the network.
>
> > Not sure if this is useful, but I sure hope it is. :)
>
> Well, what does that mean: are the modules build on 2.6.32-71 broken in
> a way that they do not use barriers (and therefore dangerous to use), or
> is everything fine with the 2.6.32-71 builds and just building on a
> newer kernel produces broken modules?

Let me extend this: when using the 2.6.32-71 build it seems that
barriers are still used (if switching back to my config with barriers),
at least /proc/drbd shows "wo:b" and kernel log says "Method to ensure
write ordering: barrier".

To summerize:

drbd.ko 8.3.13 | with barriers | with no barriers
| /proc: wo:b | /proc: wo:d
---------------------+---------------+-----------------
build on 2.6.32-71 | 100-105 MB/s | 108-109 MB/s
build on 2.6.32-220 | 45-55 MB/s | 106-108 MB/s

So, it looks that the 71er build still uses barriers and brings good
performance, while the 220er build seems to only perform if barriers are
disabled.

Regards,
Matthias

florian at hastexo

Jun 11, 2012, 2:42 PM

Post #8 of 11 (876 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

On 06/11/12 23:00, Matthias Hensler wrote:
> On Mon, Jun 11, 2012 at 10:31:16PM +0200, Florian Haas wrote:
>> On 06/11/12 22:14, Matthias Hensler wrote:
>>> Indeed, the problem lies within the kernel version used to build the
>>> drbd.ko module. I double checked by using all userland tools from 8.3.13
>>> elrepo build together with my drbd.ko build on 2.6.32-71 (but run from
>>> 2.6.32-220).
>>>
>>> Just to be clear: all tests were made with kernel 2.6.32-220, and the
>>> userland version does not matter.
>>>
>>> drbd.ko | 8.3.11 | 8.3.13
>>> ---------------------+--------+-------
>>> build on 2.6.32-71 | good | good
>>> build on 2.6.32-220 | bad | bad
>>>
>>>
>>> So, how to debug this further? I would suspect looking at the symbols of
>>> both modules might give a clue?
>>
>> As a knee-jerk response based on a hunch -- you've been warned :) --,
>> this could be related to the BIO_RW_BARRIER vs. FLUSH/FUA dance that the
>> RHEL 6 kernel has been doing between the initial RHEL 6 release, and
>> more recent updates (when they've been backporting the "let's kill
>> barriers" upstream changes from post-2.6.32).
>
> OK.
>
>> Try configuring your disk section with no-disk-barrier, no-disk-flushes
>> and no-md-flushes (in both configurations) and see if your kernel module
>> change still makes a difference.
>
> Just did that:
>
> Using the drbd.ko build on 2.6.32-71 shows minor increase in
> performance (108,5 MByte/s, so some 5% more or so).

Yeah, that's what you tend to top out at, throughput wise, over a GbE
network.

> Using the drbd.ko build on 2.6.32-220.17.1 now finally brings the
> expected performance (same as with the 2.6.32-71 built).
>
>> Of course, in production you should only use those options if you have
>> no volatile caches involved in the I/O path.
>
> Yes, that is clear. I did not plan to disable barriers, as the
> bottleneck in my setup should be clearly the network.

Slightly misguided conclusions, or incorrect assumptions, or both. :)
What you should be doing is get a good cache with a BBU or
capacitor-backed flash, and then disable barriers and flushes. That's
all in the User's Guide.

>> Not sure if this is useful, but I sure hope it is. :)
>
> Well, what does that mean: are the modules build on 2.6.32-71 broken in
> a way that they do not use barriers (and therefore dangerous to use), or
> is everything fine with the 2.6.32-71 builds and just building on a
> newer kernel produces broken modules?

For whether the newly produced modules are "broken", and if so in what
way and how to prevent any such breakage, I'll defer to the devs. As for
the alleged performance regression that was also reported in this thread
I'm curious to find out what this means for upstream kernel users, as
(IIUC) the current 8.3.13 codebase just resynced with upstream for the
3.5 kernel merge window.

I'm fairly confident Lars will weigh in on this, so let's see what he says.

Cheers,
Florian

--
Need help with High Availability?
http://www.hastexo.com/now

Attachments:

signature.asc (0.88 KB)

lars.ellenberg at linbit

Jun 12, 2012, 3:18 AM

Post #9 of 11 (879 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

On Mon, Jun 11, 2012 at 11:42:35PM +0200, Florian Haas wrote:
> On 06/11/12 23:00, Matthias Hensler wrote:
> > On Mon, Jun 11, 2012 at 10:31:16PM +0200, Florian Haas wrote:
> >> On 06/11/12 22:14, Matthias Hensler wrote:
> >>> Indeed, the problem lies within the kernel version used to build the
> >>> drbd.ko module. I double checked by using all userland tools from 8.3.13
> >>> elrepo build together with my drbd.ko build on 2.6.32-71 (but run from
> >>> 2.6.32-220).
> >>>
> >>> Just to be clear: all tests were made with kernel 2.6.32-220, and the
> >>> userland version does not matter.
> >>>
> >>> drbd.ko | 8.3.11 | 8.3.13
> >>> ---------------------+--------+-------
> >>> build on 2.6.32-71 | good | good
> >>> build on 2.6.32-220 | bad | bad
> >>>
> >>>
> >>> So, how to debug this further? I would suspect looking at the symbols of
> >>> both modules might give a clue?
> >>
> >> As a knee-jerk response based on a hunch -- you've been warned :) --,
> >> this could be related to the BIO_RW_BARRIER vs. FLUSH/FUA dance that the
> >> RHEL 6 kernel has been doing between the initial RHEL 6 release, and
> >> more recent updates (when they've been backporting the "let's kill
> >> barriers" upstream changes from post-2.6.32).
> >
> > OK.
> >
> >> Try configuring your disk section with no-disk-barrier, no-disk-flushes
> >> and no-md-flushes (in both configurations) and see if your kernel module
> >> change still makes a difference.
> >
> > Just did that:
> >
> > Using the drbd.ko build on 2.6.32-71 shows minor increase in
> > performance (108,5 MByte/s, so some 5% more or so).
>
> Yeah, that's what you tend to top out at, throughput wise, over a GbE
> network.
>
> > Using the drbd.ko build on 2.6.32-220.17.1 now finally brings the
> > expected performance (same as with the 2.6.32-71 built).
> >
> >> Of course, in production you should only use those options if you have
> >> no volatile caches involved in the I/O path.
> >
> > Yes, that is clear. I did not plan to disable barriers, as the
> > bottleneck in my setup should be clearly the network.
>
> Slightly misguided conclusions, or incorrect assumptions, or both. :)
> What you should be doing is get a good cache with a BBU or
> capacitor-backed flash, and then disable barriers and flushes. That's
> all in the User's Guide.
>
> >> Not sure if this is useful, but I sure hope it is. :)
> >
> > Well, what does that mean: are the modules build on 2.6.32-71 broken in
> > a way that they do not use barriers (and therefore dangerous to use), or
> > is everything fine with the 2.6.32-71 builds and just building on a
> > newer kernel produces broken modules?
>
> For whether the newly produced modules are "broken", and if so in what
> way and how to prevent any such breakage, I'll defer to the devs. As for
> the alleged performance regression that was also reported in this thread
> I'm curious to find out what this means for upstream kernel users, as
> (IIUC) the current 8.3.13 codebase just resynced with upstream for the
> 3.5 kernel merge window.
>
> I'm fairly confident Lars will weigh in on this, so let's see what he says.

All said already. But if you insist...

The RHEL kernel changed in a very peculiar way between RHEL 6.0 and RHEL 6.1:
They kept kABI (in theory), but changed the meaning of certain bitflags
in the bio.

So what used to cause a "BARRIER"
now causes it to fail more or less noisily.

And what used to be unused bits now cause FUA and/or FLUSH requests.

Which bitflags are used, and their meaning,
is hardcoded at compiletime, based on the kernel headers used.

If you use a drbd.ko, built against the old headers, on a new kernel,
all weak-modules and modversion and whatnot magic would think it is kABI
compatible, modprobe will find and load it,
but in fact the bio bitflag meanings have changed.

Explicit barriers/flushes done internally by DRBD simply would not
be possible, regardless of drbd configuration.

The performance degradation you experience is simply that your config
did not disable these, and if compiled against the right headers,
they are used and effective.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

dirk.schulz at kinzesberg

Jun 13, 2012, 11:31 AM

Post #10 of 11 (850 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

Sorry, but I am not deeply into kernel details, so it is not easy to
follow the thread and filter a result.
> The performance degradation you experience is simply that your config
> did not disable these, and if compiled against the right headers, they
> are used and effective.
What exactly do I have to do to avoid the perf degradation? Do I have to
compile a modified kernel and compile drbd.ko against that? If yes, what
changes to the kernel config?

Thanks,

Dirk

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

lars.ellenberg at linbit

Jun 14, 2012, 1:52 AM

Post #11 of 11 (849 views)
Permalink

Re: Performance regression with DRBD 8.3.12 and newer [In reply to]

On Wed, Jun 13, 2012 at 08:31:05PM +0200, Dirk wrote:
> Sorry, but I am not deeply into kernel details, so it is not easy to
> follow the thread and filter a result.
> >The performance degradation you experience is simply that your
> >config did not disable these, and if compiled against the right
> >headers, they are used and effective.
> What exactly do I have to do to avoid the perf degradation? Do I
> have to compile a modified kernel and compile drbd.ko against that?
> If yes, what changes to the kernel config?

* there is no performance degradation in DRBD
* what you perceive as performance degradation in DRBD is
the difference in performance between
barriers and flushes being used vs. being not supported
* to "get back" your performance,
explicitly disable barriers and flushes in drbd.

Cheers,

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads