Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DRBD: Users

DRBD very slow because of 4 KiB IO request size

 

 

DRBD users RSS feed   Index | Next | Previous | View Threaded


sebastian.riemer at profitbricks

Jul 9, 2012, 9:04 AM

Post #1 of 2 (2234 views)
Permalink
DRBD very slow because of 4 KiB IO request size

Hi all,

we at ProfitBricks do virtual data centers and we've got a customer who
has built-up a DRBD 8.3.11 system inside his virtual data center.
Storage is connected via InfiniBand and SCSI RDMA Protocol (SRP),
presented as local storage in the VMs (VirtIO blk). Like with iSCSI and
iSER such connections have trouble with 4k synchronous reads. The
connection between VMs is done via VirtIO net but is also IB-based on
the physical layer.

The IO request size detection worked - the customer can do up to 128 KiB
IO requests (not more because of the 128 KiB hashing functions) on
regular data. But everything which is related to sync or resync is at
only 4 KiB IO request size, verified by "blktrace". It is not a matter
of the syncer rate limit. He has put it to 1000 MB/s. He can be lucky
that he has 40 MB/s, because synchronous 4 KiB reads on 1 Gbit/s
Ethernet with iSCSI are at only 10 MB/s.

Why does DRBD syncs and resyncs in only slow 4 KiB IO requests?

Btw.: The IO request size detection also doesn't work above MD RAID
devices. In this configuration it always remains at 4 KiB both syncer
stuff and regular data, although the connection to the other host is
established. This IO request stuff is seriously broken IMHO! A regular
HDD can handle IO request sizes up to 512 KiB. Why not DRBD?

You limit this stuff to 4 KiB even if there is no other host. The
request size detection is only performed when there is a connection to
the other host and set back to 4 KiB if the connection gets lost. This
kills the performance completely.

What about an option to disable this IO request size detection and
setting the limit directly to the limits from the devices below (use
kernel 3.3 function "blk_set_stacking_limits()") ???

We've done this by patching the kernel code and it works. But seriously,
a 128 KiB limit above a 512 KiB chunk default MD RAID 0 doesn't make
sense. This kills all performance! So forget about using DRBD above MD
RAID 0, for now!

Cheers,
Sebastian

--
Sebastian Riemer
Linux Kernel Developer

ProfitBricks GmbH
Greifswalder Str. 207
10405 Berlin, Germany

Tel.: +49 - 30 - 60 98 56 991 - 303
Fax: +49 - 30 - 51 64 09 22
Email: sebastian.riemer [at] profitbricks
Web: http://www.profitbricks.com/

Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Andreas Gauger, Achim Weiss

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


sebastian.riemer at profitbricks

Sep 3, 2012, 7:08 AM

Post #2 of 2 (725 views)
Permalink
Re: DRBD very slow because of 4 KiB IO request size [In reply to]

On 09.07.2012 18:04, Sebastian Riemer wrote:
> Hi all,
>
> we at ProfitBricks do virtual data centers and we've got a customer who
> has built-up a DRBD 8.3.11 system inside his virtual data center.
> Storage is connected via InfiniBand and SCSI RDMA Protocol (SRP),
> presented as local storage in the VMs (VirtIO blk). Like with iSCSI and
> iSER such connections have trouble with 4k synchronous reads. The
> connection between VMs is done via VirtIO net but is also IB-based on
> the physical layer.
>
> The IO request size detection worked - the customer can do up to 128 KiB
> IO requests (not more because of the 128 KiB hashing functions) on
> regular data. But everything which is related to sync or resync is at
> only 4 KiB IO request size, verified by "blktrace". It is not a matter
> of the syncer rate limit. He has put it to 1000 MB/s. He can be lucky
> that he has 40 MB/s, because synchronous 4 KiB reads on 1 Gbit/s
> Ethernet with iSCSI are at only 10 MB/s.
>
> Why does DRBD syncs and resyncs in only slow 4 KiB IO requests?

Found the reason and the fix I guess:

http://git.drbd.org/gitweb.cgi?p=drbd-8.3.git;a=commit;h=3a2911f0c7ab80d061a5a6878c95f6d731c98554

It will be in 8.3.14.

Cheers,
Sebastian
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

DRBD users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.