sebastian.riemer at profitbricks
Jul 9, 2012, 9:04 AM
Post #1 of 2
DRBD very slow because of 4 KiB IO request size
we at ProfitBricks do virtual data centers and we've got a customer who
has built-up a DRBD 8.3.11 system inside his virtual data center.
Storage is connected via InfiniBand and SCSI RDMA Protocol (SRP),
presented as local storage in the VMs (VirtIO blk). Like with iSCSI and
iSER such connections have trouble with 4k synchronous reads. The
connection between VMs is done via VirtIO net but is also IB-based on
the physical layer.
The IO request size detection worked - the customer can do up to 128 KiB
IO requests (not more because of the 128 KiB hashing functions) on
regular data. But everything which is related to sync or resync is at
only 4 KiB IO request size, verified by "blktrace". It is not a matter
of the syncer rate limit. He has put it to 1000 MB/s. He can be lucky
that he has 40 MB/s, because synchronous 4 KiB reads on 1 Gbit/s
Ethernet with iSCSI are at only 10 MB/s.
Why does DRBD syncs and resyncs in only slow 4 KiB IO requests?
Btw.: The IO request size detection also doesn't work above MD RAID
devices. In this configuration it always remains at 4 KiB both syncer
stuff and regular data, although the connection to the other host is
established. This IO request stuff is seriously broken IMHO! A regular
HDD can handle IO request sizes up to 512 KiB. Why not DRBD?
You limit this stuff to 4 KiB even if there is no other host. The
request size detection is only performed when there is a connection to
the other host and set back to 4 KiB if the connection gets lost. This
kills the performance completely.
What about an option to disable this IO request size detection and
setting the limit directly to the limits from the devices below (use
kernel 3.3 function "blk_set_stacking_limits()") ???
We've done this by patching the kernel code and it works. But seriously,
a 128 KiB limit above a 512 KiB chunk default MD RAID 0 doesn't make
sense. This kills all performance! So forget about using DRBD above MD
RAID 0, for now!
Linux Kernel Developer
Greifswalder Str. 207
10405 Berlin, Germany
Tel.: +49 - 30 - 60 98 56 991 - 303
Fax: +49 - 30 - 51 64 09 22
Email: sebastian.riemer [at] profitbricks
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Andreas Gauger, Achim Weiss
drbd-user mailing list
drbd-user [at] lists