Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DRBD: Users

Read performance goes really low while writing.

 

 

DRBD users RSS feed   Index | Next | Previous | View Threaded


ildefonso.camargo at gmail

Feb 1, 2012, 5:16 PM

Post #1 of 5 (814 views)
Permalink
Read performance goes really low while writing.

Greetings,

I'm doing some experimentation with DRBD and relatively low speed
links (~30Mbps), with latency around 34ms. I have this:

1. DRBD version: 8.3.7 (api:88/proto:86-91)
2. Several virtual machine images on the DRBD volume.
3. deadline scheduler by default.
4. LVM on top of DRBD, and ext4 on top of this LVM.

I did this:

1. I started 3 KVM VMs (mainly idle).
2. I started to copy a big file (50GB) into DRBD partition.
3. I started to use the VMs, mainly read-only operations, with some
small writes.

At some point, the VMs just froze while trying to read. So, I went to
physical machine, and tried a "ls" on a directory inside the DRBD
volume, that I hasn't saw before (so it was not cached)... ls command
also froze.... access to cached data seems to be unaffected, then, I
stopped the cp process (ctrl-z), and waited for network traffic to
stop (patiently watching bwm). After network (replication) traffic
stopped: all unfroze.

Now, my understanding is that DRBD reads are local, and thus: I can't
understand why writing would freeze the reads....

Any ideas?

Thanks in advance,

Ildefonso Camargo
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


lars.ellenberg at linbit

Feb 2, 2012, 5:53 AM

Post #2 of 5 (766 views)
Permalink
Re: Read performance goes really low while writing. [In reply to]

On Wed, Feb 01, 2012 at 08:46:54PM -0430, Jose Ildefonso Camargo Tolosa wrote:
> Greetings,
>
> I'm doing some experimentation with DRBD and relatively low speed
> links (~30Mbps), with latency around 34ms. I have this:
>
> 1. DRBD version: 8.3.7 (api:88/proto:86-91)
> 2. Several virtual machine images on the DRBD volume.
> 3. deadline scheduler by default.
> 4. LVM on top of DRBD, and ext4 on top of this LVM.
>
> I did this:
>
> 1. I started 3 KVM VMs (mainly idle).
> 2. I started to copy a big file (50GB) into DRBD partition.
> 3. I started to use the VMs, mainly read-only operations, with some
> small writes.
>
> At some point, the VMs just froze while trying to read. So, I went to
> physical machine, and tried a "ls" on a directory inside the DRBD
> volume, that I hasn't saw before (so it was not cached)... ls command
> also froze.... access to cached data seems to be unaffected, then, I
> stopped the cp process (ctrl-z), and waited for network traffic to
> stop (patiently watching bwm). After network (replication) traffic
> stopped: all unfroze.
>
> Now, my understanding is that DRBD reads are local, and thus: I can't
> understand why writing would freeze the reads....
>
> Any ideas?

Try the same without DRBD, I would expect similar effects.
I suspect your are simply overloading your IO subsystem.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


ildefonso.camargo at gmail

Feb 2, 2012, 8:05 AM

Post #3 of 5 (768 views)
Permalink
Re: Read performance goes really low while writing. [In reply to]

On Thu, Feb 2, 2012 at 9:23 AM, Lars Ellenberg
<lars.ellenberg [at] linbit> wrote:
> On Wed, Feb 01, 2012 at 08:46:54PM -0430, Jose Ildefonso Camargo Tolosa wrote:
>> Greetings,
>>
>> I'm doing some experimentation with DRBD and relatively low speed
>> links (~30Mbps), with latency around 34ms.  I have this:
>>
>> 1. DRBD version: 8.3.7 (api:88/proto:86-91)
>> 2. Several virtual machine images on the DRBD volume.
>> 3. deadline scheduler by default.
>> 4. LVM on top of DRBD, and ext4 on top of this LVM.
>>
>> I did this:
>>
>> 1. I started 3 KVM VMs (mainly idle).
>> 2. I started to copy a big file (50GB) into DRBD partition.
>> 3. I started to use the VMs, mainly read-only operations, with some
>> small writes.
>>
>> At some point, the VMs just froze while trying to read.  So, I went to
>> physical machine, and tried a "ls" on a directory inside the DRBD
>> volume, that I hasn't saw before (so it was not cached)... ls command
>> also froze.... access to cached data seems to be unaffected, then, I
>> stopped the cp process (ctrl-z), and waited for network traffic to
>> stop (patiently watching bwm).  After network (replication) traffic
>> stopped: all unfroze.
>>
>> Now, my understanding is that DRBD reads are local, and thus: I can't
>> understand why writing would freeze the reads....
>>
>> Any ideas?
>
> Try the same without DRBD, I would expect similar effects.
> I suspect your are simply overloading your IO subsystem.

I already did that, before posting to the list. Also, note that the
problem doesn't happen if I disconnect the secondary.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


brian at linbit

Feb 2, 2012, 8:14 AM

Post #4 of 5 (766 views)
Permalink
Re: Read performance goes really low while writing. [In reply to]

On 02/01/2012 05:16 PM, Jose Ildefonso Camargo Tolosa wrote:
> Greetings,
>
> I'm doing some experimentation with DRBD and relatively low speed
> links (~30Mbps), with latency around 34ms. I have this:

^^^ I think that's where your problem is. Your buffers are probably
filling up causing the slow down, which is also why you don't see the
issue while DRBD isn't connected. Even with Protocol A your buffer is
only so big. In situations like this we recommend people use DRBD
Proxy. If you're interested in giving it a try let us (LINBIT) know.
We use it internally to replicate data from Beaverton, OR to Vienna,
Austria and it works quite well.


>
> 1. DRBD version: 8.3.7 (api:88/proto:86-91)
> 2. Several virtual machine images on the DRBD volume.
> 3. deadline scheduler by default.
> 4. LVM on top of DRBD, and ext4 on top of this LVM.
>
> I did this:
>
> 1. I started 3 KVM VMs (mainly idle).
> 2. I started to copy a big file (50GB) into DRBD partition.
> 3. I started to use the VMs, mainly read-only operations, with some
> small writes.
>
> At some point, the VMs just froze while trying to read. So, I went to
> physical machine, and tried a "ls" on a directory inside the DRBD
> volume, that I hasn't saw before (so it was not cached)... ls command
> also froze.... access to cached data seems to be unaffected, then, I
> stopped the cp process (ctrl-z), and waited for network traffic to
> stop (patiently watching bwm). After network (replication) traffic
> stopped: all unfroze.
>
> Now, my understanding is that DRBD reads are local, and thus: I can't
> understand why writing would freeze the reads....
>
> Any ideas?
>
> Thanks in advance,
>
> Ildefonso Camargo
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user

--

: Brian Hellman
: LINBIT | "Your Way to High Availability"
: 1-503-573-1262 x202 | 1-877-4-LINBIT
: Web: http://www.linbit.com
:
: Twitter: http://www.linbit.com/en/twitter
: Facebook: http://www.linbit.com/en/facebook

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


ildefonso.camargo at gmail

Feb 3, 2012, 7:47 PM

Post #5 of 5 (768 views)
Permalink
Re: Read performance goes really low while writing. [In reply to]

Greetings,

It looks like an update to DRBD 8.4.1 (with update to kernel 3.x)
solved at least the issue with slow reads while writing.

Just wanted to share this with the list.

Ildefonso.

On Thu, Feb 2, 2012 at 2:34 PM, Jose Ildefonso Camargo Tolosa
<ildefonso.camargo [at] gmail> wrote:
> On Thu, Feb 2, 2012 at 11:44 AM, Brian R. Hellman <brian [at] linbit> wrote:
>>
>>
>> On 02/01/2012 05:16 PM, Jose Ildefonso Camargo Tolosa wrote:
>>> Greetings,
>>>
>>> I'm doing some experimentation with DRBD and relatively low speed
>>> links (~30Mbps), with latency around 34ms.  I have this:
>>
>> ^^^ I think that's where your problem is. Your buffers are probably
>> filling up causing the slow down, which is also why you don't see the
>> issue while DRBD isn't connected. Even with Protocol A your buffer is
>> only so big.  In situations like this we recommend people use DRBD
>> Proxy.  If you're interested in giving it a try let us (LINBIT) know.
>> We use it internally to replicate data from Beaverton, OR to Vienna,
>> Austria and it works quite well.
>
> Yeah, maybe, but the question still remains: why writing puts reads on
> hold? why if DRBD is supposed to read locally, it doesn't allow reads
> while writing?
>
>>
>>
>>>
>>> 1. DRBD version: 8.3.7 (api:88/proto:86-91)
>>> 2. Several virtual machine images on the DRBD volume.
>>> 3. deadline scheduler by default.
>>> 4. LVM on top of DRBD, and ext4 on top of this LVM.
>>>
>>> I did this:
>>>
>>> 1. I started 3 KVM VMs (mainly idle).
>>> 2. I started to copy a big file (50GB) into DRBD partition.
>>> 3. I started to use the VMs, mainly read-only operations, with some
>>> small writes.
>>>
>>> At some point, the VMs just froze while trying to read.  So, I went to
>>> physical machine, and tried a "ls" on a directory inside the DRBD
>>> volume, that I hasn't saw before (so it was not cached)... ls command
>>> also froze.... access to cached data seems to be unaffected, then, I
>>> stopped the cp process (ctrl-z), and waited for network traffic to
>>> stop (patiently watching bwm).  After network (replication) traffic
>>> stopped: all unfroze.
>>>
>>> Now, my understanding is that DRBD reads are local, and thus: I can't
>>> understand why writing would freeze the reads....
>>>
>>> Any ideas?
>>>
>>> Thanks in advance,
>>>
>>> Ildefonso Camargo
>>> _______________________________________________
>>> drbd-user mailing list
>>> drbd-user [at] lists
>>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>
>> --
>>
>> : Brian Hellman
>> : LINBIT | "Your Way to High Availability"
>> : 1-503-573-1262 x202 | 1-877-4-LINBIT
>> : Web: http://www.linbit.com
>> :
>> : Twitter: http://www.linbit.com/en/twitter
>> : Facebook: http://www.linbit.com/en/facebook
>>
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user [at] lists
>> http://lists.linbit.com/mailman/listinfo/drbd-user
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

DRBD users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.