Mailing List Archive: DRBD: Users

Slow Reads on VM - Xenserver and DRBD

Index | Next | Previous | View Threaded

stril at gmx

Jul 23, 2012, 8:52 AM

Post #1 of 11 (1711 views)
Permalink

Slow Reads on VM - Xenserver and DRBD

Hi!

I just started with a setup of DRBD and Xenserver 6.0.2

Of course, my read-performance inside the VMs is very bad. The storage system is a Raid 10 with 8x Intel 320 SSD on an LSI-Raid-Controller.

Testing the Read-Performance with hdparm on Dom-0 shows:

/dev/drbd1:
Timing buffered disk reads: 1432 MB in 3.00 seconds = 477.18 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device

If I start the same test in a VM, I get:
Timing buffered disk reads: 100 MB in 2.56 seconds = 39.10 MB/sec

There is just one single VM on that system.

Can you help me? Windows VMs have shown the same issue. The problem does also consist, if the second node is disconnected...

Thank you and best wishes.
Phil
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

chibi at gol

Jul 23, 2012, 9:14 PM

Post #2 of 11 (1646 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

On Mon, 23 Jul 2012 17:52:21 +0200 Phil Stricker wrote:

> Hi!
>
> I just started with a setup of DRBD and Xenserver 6.0.2
>
> Of course, my read-performance inside the VMs is very bad.

Of course why? Did you expect a bad read performance and if so, what made
you believe so?

>The storage
> system is a Raid 10 with 8x Intel 320 SSD on an LSI-Raid-Controller.
>
What LSI controller? I can get better performance than your 477MB/s with a
software RAID 10 and just 5 plain 2TB Hitachi SATA drives...

What is the speed of the raw backing device?
For reads, it should be same as the DRBD device...

> Testing the Read-Performance with hdparm on Dom-0 shows:
>
> /dev/drbd1:
> Timing buffered disk reads: 1432 MB in 3.00 seconds = 477.18 MB/sec
>
You should be able to get much better numbers than that, also depending on
your controller a RAID6 using 8 disks can be faster (in all aspects!) than
a RAID 10 with 8 drives. Simply put, more active data disks to read from
and write to. Of course the controller needs to have a fast enough CPU to
do the checksum crunching.

> HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate
> ioctl for device
>
> If I start the same test in a VM, I get:
> Timing buffered disk reads: 100 MB in 2.56 seconds = 39.10 MB/sec
>
>
> There is just one single VM on that system.
>
>
> Can you help me? Windows VMs have shown the same issue.
That clearly looks like some Xen deficiency or configuration issue, can't
help you with that, sorry.

> The problem does
> also consist, if the second node is disconnected...
>
Reads are ALWAYS local and should not be affected by the presence of
replication or not.

Regards,

Christian
--
Christian Balzer Network/Systems Engineer
chibi [at] gol Global OnLine Japan/Fusion Communications
http://www.gol.com/
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jul 24, 2012, 12:40 AM

Post #3 of 11 (1646 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

On 07/24/2012 06:14 AM, Christian Balzer wrote:
>> > Can you help me? Windows VMs have shown the same issue.
> That clearly looks like some Xen deficiency or configuration issue, can't
> help you with that, sorry.
>

That reminds me - I've once heard a talk on just why Xen has I/O
performance issues. It's because of the way syscalls are handled among
DomU kernel, Dom0 kernel and hypervisor.

KVM has advantages there. There's even an experimental hypervisor(?)
called Xenner that allows running Xen domains on a KVM host, with better
syscall handling than Xen itself.

Note that such deficiencies (and/or advantages) usually apply to certain
workloads only. I advise to do more than simple throughput tests. If you
*do* measure throughput, I believe that you should transfer more data
than fits a 3 second window.

Finally: hdparm? Isn't that limited to the old IDE kernel drivers? Are
you using those? If so - why?

Cheers,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

chibi at gol

Jul 24, 2012, 2:11 AM

Post #4 of 11 (1642 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

On Tue, 24 Jul 2012 09:40:17 +0200 Felix Frank wrote:

> On 07/24/2012 06:14 AM, Christian Balzer wrote:
> >> > Can you help me? Windows VMs have shown the same issue.
> > That clearly looks like some Xen deficiency or configuration issue,
> > can't help you with that, sorry.
> >
>
> That reminds me - I've once heard a talk on just why Xen has I/O
> performance issues. It's because of the way syscalls are handled among
> DomU kernel, Dom0 kernel and hypervisor.
>
Yeah, I've read the same thing and am leaning towards KVM for a fully
virtualized system, though Vserver and (in the future) LXC work for 90% of
the requirements I have.

[...]
>
> Finally: hdparm? Isn't that limited to the old IDE kernel drivers? Are
> you using those? If so - why?
>
While I certainly wouldn't recommend hdparm as generic storage
and filesystem performance measuring tool (bonnie++ comes to mind), it does
the "right" thing for read tests, more or less.
---
# hdparm -t /dev/drbd0

/dev/drbd0:
Timing buffered disk reads: 2114 MB in 3.01 seconds = 702.38 MB/sec
---
That clocks in about 15% (100MB/s) slower than my bonnie tests, but it's
at least in the same ballpark. So the numbers he's seeing are
unfortunately correct AFAICT.

Regards,

Christian
--
Christian Balzer Network/Systems Engineer
chibi [at] gol Global OnLine Japan/Fusion Communications
http://www.gol.com/
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jul 24, 2012, 2:17 AM

Post #5 of 11 (1639 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

On 07/24/2012 11:11 AM, Christian Balzer wrote:
> Yeah, I've read the same thing and am leaning towards KVM for a fully
> virtualized system, though Vserver and (in the future) LXC work for 90% of
> the requirements I have.

Seconded. Linux vServer + DRBD make for a robust HA setup at very good
performance. A vserver RA for pacemaker is floating through the web.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

downstreamz at gmx

Jul 24, 2012, 2:46 AM

Post #6 of 11 (1632 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

Hi!

I think, I am seing two different Problems of which one is isolated:

- Slow performance in a VM
The issue seams to be related to different OSs: In a Debian-VM, I am getting nearly full speed of the drbd-block-device , in a CentOS 5.8 VM, I can only see 10% of that speed.

- Slow overall performance of the raid-array:
I cannot reach more than 450 MB/s on the array (LSI 9260-4i, 8x Intel 320 160 GB SSD). I see 450 MB/s in a Raid10 setting and I see 450MB/s in a Raid0 test setting. That is crazy!

Did you ever see that behaviour?

Best wishes,
Phil

> -----Original Message-----
> From: ff [at] mpexnet [mailto:drbd-user-bounces [at] lists] On
> Behalf Of Felix Frank
> Sent: Tuesday, July 24, 2012 11:17 AM
> To: Christian Balzer
> Cc: drbd-user [at] lists
> Subject: Re: [DRBD-user] Slow Reads on VM - Xenserver and DRBD
>
> On 07/24/2012 11:11 AM, Christian Balzer wrote:
> > Yeah, I've read the same thing and am leaning towards KVM for a fully
> > virtualized system, though Vserver and (in the future) LXC work for
> > 90% of the requirements I have.
>
> Seconded. Linux vServer + DRBD make for a robust HA setup at very good
> performance. A vserver RA for pacemaker is floating through the web.
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

stril at gmx

Jul 24, 2012, 5:08 AM

Post #7 of 11 (1636 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

Hi!

One Problem is solved:
The extremely bad read-performance in the VM is gone away after the downgrade from DRBD 8.4.1 to 8.3.13!!

Now, I can see "normal" values for write _and_ read operations!

There is just one problem left:
The setup "xenserver <-> LSI9260 <-> 8x SSD" is too slow! I can just get about 450 MB/s - no matter, if i set it up as an Raid 10 or Raid 0!

Did you ever see something like that?

Thank you
Phil
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

chibi at gol

Jul 24, 2012, 5:56 PM

Post #8 of 11 (1648 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

On Tue, 24 Jul 2012 14:08:29 +0200 Phil Stricker wrote:

> Hi!
>
> One Problem is solved:
> The extremely bad read-performance in the VM is gone away after the
> downgrade from DRBD 8.4.1 to 8.3.13!!
>
Ah yes, 8.4... ^o^

> There is just one problem left:
> The setup "xenserver <-> LSI9260 <-> 8x SSD" is too slow! I can just get
> about 450 MB/s - no matter, if i set it up as an Raid 10 or Raid 0!
>
This is xenserver directly talking to the RAID device, not DRBD?

> Did you ever see something like that?
Not using xenserver, but that now really looks like a xenserver limitation.

This is what I get on an Adaptec 51645 with an 8 disk RAID10 of 2TB SATA
Hitachi drives (bonnie++):
---
3.2, 8disk R10, ext4, 1G journal
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
mb13 64G 529030 55 270813 33 1072430 63 642.0 50
---

So the writes on this setup with inferior drives are faster than your reads
and reads are twice as fast...

Regards,

Christian
--
Christian Balzer Network/Systems Engineer
chibi [at] gol Global OnLine Japan/Fusion Communications
http://www.gol.com/
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

chibi at gol

Jul 25, 2012, 6:41 PM

Post #9 of 11 (1624 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

On Tue, 24 Jul 2012 11:46:09 +0200 Phil Stricker wrote:

>
> - Slow overall performance of the raid-array:
> I cannot reach more than 450 MB/s on the array (LSI 9260-4i, 8x Intel
> 320 160 GB SSD). I see 450 MB/s in a Raid10 setting and I see 450MB/s in
> a Raid0 test setting. That is crazy!
>
a) The controller you're listing has only 4 ports. Typo or are you using 2
of them?

b) Boot into a Live CD (the Debian one has bonnie++ in it) and run tests
from there to exclude any Xenserver influences.

Regards,

Christian
--
Christian Balzer Network/Systems Engineer
chibi [at] gol Global OnLine Japan/Fusion Communications
http://www.gol.com/
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jul 26, 2012, 12:43 AM

Post #10 of 11 (1625 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

On 07/24/2012 11:46 AM, Phil Stricker wrote:
> - Slow performance in a VM
> The issue seams to be related to different OSs: In a Debian-VM, I am getting nearly full speed of the drbd-block-device , in a CentOS 5.8 VM, I can only see 10% of that speed.

If you're using stock kernels in your VMs, try
- swapping them
- hand-configuring them
- building your own, new-ish and newest versions etc.

I've seen performance issues related to defaults configured into
kernels, even on physical machines. Maybe you have that kind of culprit.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

andreas at hastexo

Jul 26, 2012, 1:01 AM

Post #11 of 11 (1620 views)
Permalink

Re: Slow Reads on VM - Xenserver and DRBD [In reply to]

On 07/24/2012 11:46 AM, Phil Stricker wrote:
> Hi!
>
> I think, I am seing two different Problems of which one is isolated:
>
> - Slow performance in a VM
> The issue seams to be related to different OSs: In a Debian-VM, I am getting nearly full speed of the drbd-block-device , in a CentOS 5.8 VM, I can only see 10% of that speed.
>

You are using paravirtualized drivers in the VM?

> - Slow overall performance of the raid-array:
> I cannot reach more than 450 MB/s on the array (LSI 9260-4i, 8x Intel 320 160 GB SSD). I see 450 MB/s in a Raid10 setting and I see 450MB/s in a Raid0 test setting. That is crazy!
>

You already shared your drbd configuration? ... and the controller has a
non-volatile cache?

Regards,
Andreas

--
Need help with DRBD?
http://www.hastexo.com/now

> Did you ever see that behaviour?
>
> Best wishes,
> Phil
>
>
>
>> -----Original Message-----
>> From: ff [at] mpexnet [mailto:drbd-user-bounces [at] lists] On
>> Behalf Of Felix Frank
>> Sent: Tuesday, July 24, 2012 11:17 AM
>> To: Christian Balzer
>> Cc: drbd-user [at] lists
>> Subject: Re: [DRBD-user] Slow Reads on VM - Xenserver and DRBD
>>
>> On 07/24/2012 11:11 AM, Christian Balzer wrote:
>>> Yeah, I've read the same thing and am leaning towards KVM for a fully
>>> virtualized system, though Vserver and (in the future) LXC work for
>>> 90% of the requirements I have.
>>
>> Seconded. Linux vServer + DRBD make for a robust HA setup at very good
>> performance. A vserver RA for pacemaker is floating through the web.
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user [at] lists
>> http://lists.linbit.com/mailman/listinfo/drbd-user
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
>

Attachments:

signature.asc (0.22 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads