Mailing List Archive: DRBD: Users

replication link stability

Index | Next | Previous | View Threaded

raveenpl at gmail

Feb 16, 2012, 6:18 AM

Post #1 of 7 (514 views)
Permalink

replication link stability

Hi,

I have two servers with dedicated replication link (WiFI - 48Mbit/s). My
clients are connecting via other network link (100Mbit/s) to SAMBA service.

When they are copying files then replication link is very unstable - I can
see a lot of NetworkFailure statuses. Finally, they are disconnected from
SAMBA share (what is strange).

I think that dedicated network link is ok, because when I manually copy
files between two samba locations then copying is fine.

What is the reason of this behaviour? Any solution?

Below you can find my net and syncer section from drbd.conf:

net {
max-buffers 2048;
max-epoch-size 2048;
unplug-watermark 128;
sndbuf-size 0;
}

syncer {
c-min-rate 10M;
rate 10M;
al-extents 127;
}

Best regards
Piotr Kandziora

arnold at arnoldarts

Feb 16, 2012, 7:30 AM

Post #2 of 7 (497 views)
Permalink

Re: replication link stability [In reply to]

On Thursday 16 February 2012 15:18:05 Piotr Kandziora wrote:
> Hi,
>
> I have two servers with dedicated replication link (WiFI - 48Mbit/s). My
> clients are connecting via other network link (100Mbit/s) to SAMBA service.

Rofl!

Sorry, spend the 10€ per server to get them dedicated 100MBit cards. It will
do you a great favour.
But unless you spend 20€ per server for Gigabit cards, it will still be slow
as hell...

The wifi is not only bad for your throughput: 48MBit/s is low compared to
120MBytes/s for direct disk access. Its actually half of that because wifi is
half-duplex by design.
100MBit will be a bit better as you then have ~12MByte/s per direction.
Only Gigabit will really keep up with the throughput of the disk.

Additionally wifi has the ethernet-latency plus the wireless-latency plus the
latency of half-duplex and collision-detection.

It is a) no wonder your users will be complaining about the poor performance
(mine are still complaining while I have a dedicated GB-link) and b) its no
wonder drbd complains and stops the sync...

Have fun,

Arnold

Attachments:

signature.asc (0.19 KB)

linux at alteeve

Feb 16, 2012, 7:43 AM

Post #3 of 7 (490 views)
Permalink

Re: replication link stability [In reply to]

On 02/16/2012 09:18 AM, Piotr Kandziora wrote:
> Hi,
>
> I have two servers with dedicated replication link (WiFI - 48Mbit/s). My
> clients are connecting via other network link (100Mbit/s) to SAMBA service.
>
> When they are copying files then replication link is very unstable - I
> can see a lot of NetworkFailure statuses. Finally, they are disconnected
> from SAMBA share (what is strange).
>
> I think that dedicated network link is ok, because when I manually copy
> files between two samba locations then copying is fine.
>
> What is the reason of this behaviour? Any solution?
>
> Below you can find my net and syncer section from drbd.conf:
>
> net {
> max-buffers 2048;
> max-epoch-size 2048;
> unplug-watermark 128;
> sndbuf-size 0;
> }
>
> syncer {
> c-min-rate 10M;
> rate 10M;
> al-extents 127;
> }
>
>
> Best regards
> Piotr Kandziora

Hi Piotr,

Samba (and similar protocols) are generally pretty forgiving of
network issues, so it's not a surprise that they work where DRBD does
not. DRBD is designed for high speed/low latency environments, and has a
fair bit of attention paid to fault detection and recovery. If it sees a
network fault, even a short one, it has to assume the link has failed
and recovery is needed. This does not lend itself well to use on
slow/error prone networks.

Is it possible to use a wired connection for the DRBD replication? If
not, I would suggest looking at DRBD Proxy instead, as it is designed to
run over slow/unstable links, but it is asynchronous, so be advised that
consistency is not always guaranteed.

--
Digimer
E-Mail: digimer [at] alteeve
Papers and Projects: https://alteeve.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

raveenpl at gmail

Feb 16, 2012, 8:02 AM

Post #4 of 7 (492 views)
Permalink

Re: replication link stability [In reply to]

Digimer,

Thanks for your answer. DRBD Proxy could be a solution, but it is not
possible for today.

I am thinking of trying DRBD with protocol A, but currently I am not able
to test on my environment as it is in production. I will have maintenance
window on the weekend. Do you think it can help? If yes, may you suggest
values for tcp send buffer?

Thanks in advance.

Best regards
Piotr Kandziora

On Thu, Feb 16, 2012 at 4:43 PM, Digimer <linux [at] alteeve> wrote:

> Hi Piotr,
>
> Samba (and similar protocols) are generally pretty forgiving of
> network issues, so it's not a surprise that they work where DRBD does
> not. DRBD is designed for high speed/low latency environments, and has a
> fair bit of attention paid to fault detection and recovery. If it sees a
> network fault, even a short one, it has to assume the link has failed
> and recovery is needed. This does not lend itself well to use on
> slow/error prone networks.
>
> Is it possible to use a wired connection for the DRBD replication? If
> not, I would suggest looking at DRBD Proxy instead, as it is designed to
> run over slow/unstable links, but it is asynchronous, so be advised that
> consistency is not always guaranteed.
>

linux at alteeve

Feb 16, 2012, 8:37 AM

Post #5 of 7 (488 views)
Permalink

Re: replication link stability [In reply to]

On 02/16/2012 11:02 AM, Piotr Kandziora wrote:
> Digimer,
>
>
> Thanks for your answer. DRBD Proxy could be a solution, but it is not
> possible for today.
>
> I am thinking of trying DRBD with protocol A, but currently I am not
> able to test on my environment as it is in production. I will have
> maintenance window on the weekend. Do you think it can help? If yes, may
> you suggest values for tcp send buffer?
>
> Thanks in advance.
>
> Best regards
> Piotr Kandziora

I am not the person to advise on tuning. For that, I would either ask on
#drbd or call Linbit and ask for tuning support.

Protocol A is not ideal, as it prevents dual-primary use (which may not
be an issue) and is asynchronous, which means consistency on the
secondary node is not always guaranteed. I would not recommend it, but
it is an option is you are willing to assume the risk.

By far, the ideal solution is to use a wired connection.

--
Digimer
E-Mail: digimer [at] alteeve
Papers and Projects: https://alteeve.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ls at numerigraphe

Feb 17, 2012, 1:08 AM

Post #6 of 7 (483 views)
Permalink

Re: replication link stability [In reply to]

> (...) I would suggest looking at DRBD Proxy instead, as it is designed to
> run over slow/unstable links, but it is asynchronous, so be advised that
> consistency is not always guaranteed.
>
For what I know of drbd-proxy, consistency is in fact guaranteed, though
the secondary node may of course be out of date.
However, the proxy is probably more expensive than buying a pair of GigE
adapters and having a specialist install the cable.

We've been running DRBD for a year over wimax (60Mb/s full duplex)
without a proxy and it was slow and reacted very badly to the natural
unreliability of the link.
We ended running disconnected most of the time, and synchronizing in
off-peak time but even that flawless.
A few years ago the docs used to advice against using even a switch
between the nodes...
So get as good a network as you can for DRBD.

Lionel Sausin.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

lars.ellenberg at linbit

Feb 17, 2012, 2:51 AM

Post #7 of 7 (481 views)
Permalink

Re: replication link stability [In reply to]

On Thu, Feb 16, 2012 at 11:37:48AM -0500, Digimer wrote:
> On 02/16/2012 11:02 AM, Piotr Kandziora wrote:
> > Digimer,
> >
> >
> > Thanks for your answer. DRBD Proxy could be a solution, but it is not
> > possible for today.
> >
> > I am thinking of trying DRBD with protocol A, but currently I am not
> > able to test on my environment as it is in production. I will have
> > maintenance window on the weekend. Do you think it can help? If yes, may
> > you suggest values for tcp send buffer?
> >
> > Thanks in advance.
> >
> > Best regards
> > Piotr Kandziora
>
> I am not the person to advise on tuning. For that, I would either ask on
> #drbd or call Linbit and ask for tuning support.
>
> Protocol A is not ideal, as it prevents dual-primary use (which may not
> be an issue) and is asynchronous, which means consistency on the
> secondary node is not always guaranteed. I would not recommend it, but
> it is an option is you are willing to assume the risk.

That's not about consistency.
The Secondary is consistent with protocol A as well.

In case of Primary crash, some of the data that was in-flight
to the Secondary will just not be there.
But what is there is still consistent.

It is however a very *bad* idea if you run NFS or SAMBA clients,
or iSCSI or anything that is not restarted with the failover anyways:
they assume that, if the storage said stuff was written, it was written.
If then, after a failover with async replication, the last few (amount that
fits in the socket buffers) is missing,
they will all break in the most "interesting" ways.

Async mirroring is intended to be used for desaster recovery purposes.
After a "desaster" and failover, any clients would need to be rebooted, or
unmounted, remounted or otherwise made aware of the fact that the data just
"jumped back in time" a few (milli) seconds.

> By far, the ideal solution is to use a wired connection.

Accessing a storage with higher bandwidth than the storage has internally
is a bad idea in general. So yes, make sure your DRBD internally has *at least*
the bandwidth and link quality you want to see from the clients accessing it.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads