Mailing List Archive: DRBD: Users

DRBD - one half of Proxmox cluster miscommunicating

1 2

View All

Index | Next | Previous | View Threaded

james.gibbon at virgin

Jul 30, 2012, 1:06 PM

Post #1 of 30 (2680 views)
Permalink

DRBD - one half of Proxmox cluster miscommunicating

Hi,

I moved my Proxmox cluster - consisting essentially of two
physical servers, two Cisco NAS units where the (KVM) VM images
live and two switches, to a new data centre where they now have
new IP addresses.

I reconfigured basic networking on the two servers, updated the
IP addresses in the Proxmox config and rebooted the boxes, master
node first.

The storage is set up as /dev/drbdvg0 and /dev/drbdvg1. I didn't
install this myself and I'm not that familiar with DRBD or indeed
iSCSI. Both are used to store KVM guest virtual machine images,
seen by both servers.

Everything looked fine, until I attempted to start a VM on the
second (slave) node. It took ages to start, hanging for thirty
seconds at a time. It was clearly miscommunicating with the NAS.

All of the images, including those set up on the second node,
will run fine on the first (and that's what I'm doing for now).

So the first (master) box has excellent access to the NAS, while
the second (slave) has trouble reading from it.

On the first box, /proc/drbd looks like this:

version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----
ns:0 nr:0 dw:27568823 dr:156762105 al:309656 bm:309639 lo:0 pe:0 ua:0
ap:0 ep:1 wo:b oos:10184632
1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----
ns:0 nr:0 dw:2451648 dr:14918745 al:1244 bm:1211 lo:0 pe:0 ua:0 ap:0
ep:1 wo:b oos:1152564

And on the second, troublesome box:

version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----
ns:0 nr:0 dw:0 dr:1705944 al:0 bm:107 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:954596
1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----
ns:0 nr:0 dw:0 dr:1821288 al:0 bm:107 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:520192

So it looks like at some level they aren't talking to each other
- I don't see the usual "UpToDate/UpToDate".

I'm also seeing lots of messages like this on the second node:

connection1:0: ping timeout of 5 secs expired, recv timeout 5,
last rx 4329026692, last ping 4329027942, now 4329029192
connection1:0: detected conn error (1011)

Can anyone suggest what might have gone wrong here? A cabling
issue maybe? Or how to fix it? I'm particular anxious to avoid
losing updates to the images as seen by the first node if they
manage to sync up - don't want to lose or corrupt the VM images!

I inherited this setup and I'm not that familiar with DRBD, though
keen to learn. Very grateful for any advice.

Thanks,
James

ff at mpexnet

Jul 31, 2012, 12:32 AM

Post #2 of 30 (2575 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Hi,

On 07/30/2012 10:06 PM, JAMES GIBBON wrote:
> version: 8.3.7 (api:88/proto:86-91)
> srcversion: EE47D8BF18AC166BE219757
> 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----
> ns:0 nr:0 dw:27568823 dr:156762105 al:309656 bm:309639 lo:0 pe:0
> ua:0 ap:0 ep:1 wo:b oos:10184632
> 1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----
> ns:0 nr:0 dw:2451648 dr:14918745 al:1244 bm:1211 lo:0 pe:0 ua:0 ap:0
> ep:1 wo:b oos:1152564
>
> And on the second, troublesome box:
>
> version: 8.3.7 (api:88/proto:86-91)
> srcversion: EE47D8BF18AC166BE219757
> 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----
> ns:0 nr:0 dw:0 dr:1705944 al:0 bm:107 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
> oos:954596
> 1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----
> ns:0 nr:0 dw:0 dr:1821288 al:0 bm:107 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
> oos:520192
>
>
> So it looks like at some level they aren't talking to each other
> - I don't see the usual "UpToDate/UpToDate".

you could say that ;)

Judging from your log excerpt, there might be a connectivity issue, but
this could very well be a pure split brain that needs resolving. See
http://www.drbd.org/users-guide/s-resolve-split-brain.html and note that
you will likely loose whatever has been written to your "troubled" node.
You may want to copy precious data if any has been written.

What we'd need to see is your drbd configuration. Also the connection
states of both nodes' respective NICs. Finally: Have you tried just
issuing "drbdadm connect all" on the second node?

HTH,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

james.gibbon at virgin

Jul 31, 2012, 1:51 AM

Post #3 of 30 (2571 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On Tue, 31 Jul 2012 09:32:48 +0200
Felix Frank <ff [at] mpexnet> wrote:

>
> Judging from your log excerpt, there might be a connectivity
> issue, but this could very well be a pure split brain that
> needs resolving. See
> http://www.drbd.org/users-guide/s-resolve-split-brain.html and
> note that you will likely loose whatever has been written to
> your "troubled" node. You may want to copy precious data if any
> has been written.
>
> What we'd need to see is your drbd configuration. Also the
> connection states of both nodes' respective NICs. Finally: Have
> you tried just issuing "drbdadm connect all" on the second node?
>

Hi Felix,

Many thanks for your reply.

At the moment, no virtual machines are running on the second,
"troubled" server. I cannot afford to lose data from the master's
view of the storage, which is exactly as I would wish it, but
the slave isn't doing anything - so if they are out of sync, I'm
happy for the slave to adopt the same view of the world as the
master.

Aaaaaahhh.. I can see what I've done. Thanks for asking about the
NICs. The IP address connecting to the storage on the master is
10.0.1.1. The IP on the slave should be 10.0.1.2, but in fact at
the moment it is also 10.0.1.1. This is because when I changed
the server public IP addresses I copied over the interfaces file
from the master to the slave. I edited the public IP address, but
not the private IP address used to communicate with the NAS units.

So it looks like it's my fault.

OK. So my question now is - how can I fix this, without losing
data from the master? Tempting though it is to simply correct the
file and reboot, I think it better to solicit a more experienced
opinion first.

Another consideration is that if resyncing the disks will consume
a lot of resources on the master, I'll need that to happen out of
hours to avoid impacting production systems running on it.

Thanks again, and I'd be very grateful for some help. My main
concern is that I can't afford for the master's version to be disrupted.

Cheers
James

dirk at proactive

Jul 31, 2012, 1:57 AM

Post #4 of 30 (2591 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Op 31-7-2012 10:51, JAMES GIBBON schreef:
> On Tue, 31 Jul 2012 09:32:48 +0200
> Felix Frank <ff [at] mpexnet <mailto:ff [at] mpexnet>> wrote:
>
> >
> > Judging from your log excerpt, there might be a connectivity
> > issue, but this could very well be a pure split brain that
> > needs resolving. See
> > http://www.drbd.org/users-guide/s-resolve-split-brain.html and
> > note that you will likely loose whatever has been written to
> > your "troubled" node. You may want to copy precious data if any
> > has been written.
> >
> > What we'd need to see is your drbd configuration. Also the
> > connection states of both nodes' respective NICs. Finally: Have
> > you tried just issuing "drbdadm connect all" on the second node?
> >
>
> Hi Felix,
>
> Many thanks for your reply.
>
> At the moment, no virtual machines are running on the second,
> "troubled" server. I cannot afford to lose data from the master's
> view of the storage, which is exactly as I would wish it, but
> the slave isn't doing anything - so if they are out of sync, I'm
> happy for the slave to adopt the same view of the world as the
> master.
>
> Aaaaaahhh.. I can see what I've done. Thanks for asking about the
> NICs. The IP address connecting to the storage on the master is
> 10.0.1.1. The IP on the slave should be 10.0.1.2, but in fact at
> the moment it is also 10.0.1.1. This is because when I changed
> the server public IP addresses I copied over the interfaces file
> from the master to the slave. I edited the public IP address, but
> not the private IP address used to communicate with the NAS units.
>
> So it looks like it's my fault.
>
> OK. So my question now is - how can I fix this, without losing
> data from the master? Tempting though it is to simply correct the
> file and reboot, I think it better to solicit a more experienced
> opinion first.
>
> Another consideration is that if resyncing the disks will consume
> a lot of resources on the master, I'll need that to happen out of
> hours to avoid impacting production systems running on it.
>
> Thanks again, and I'd be very grateful for some help. My main
> concern is that I can't afford for the master's version to be disrupted.
>

Since the problem is on the slave, I wouldn't worry to much. Fix the IP
on the slave and issue the 'drdbadm connect all' on the slave. If it has
been the slave all the time, it should connect and synchronise without
complaining.

And... just to be sure: Make a backup of the data on the master node...

Cheers,

Dirk

ff at mpexnet

Jul 31, 2012, 2:09 AM

Post #5 of 30 (2573 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On 07/31/2012 10:57 AM, Dirk Bonenkamp - ProActive wrote:
> split brain that
>> needs resolving. See
>> http://www.drbd.org/users-guide/s-resolve-split-brain.html and

It's all laid out in the above link. You *are* in split brain.

This probably won't mess with performance. Be sure your syncer rate does
not exceed 30% of your maximum network throughput (i.e., for 2 resources
you should limit to 15% per resource).

Also be *very* sure you discard the data on the correct (broken) node.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

james.gibbon at virgin

Jul 31, 2012, 2:16 AM

Post #6 of 30 (2578 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On Tue, 31 Jul 2012 10:57:47 +0200
Dirk Bonenkamp - ProActive <dirk [at] proactive> wrote:

>
> Since the problem is on the slave, I wouldn't worry to much.
> Fix the IP on the slave and issue the 'drdbadm connect all' on
> the slave. If it has been the slave all the time, it should
> connect and synchronise without complaining.
>
> And... just to be sure: Make a backup of the data on the master
> node...
>

Thanks, Dirk!

Unfortunately I can't back up the master node's data. It consists of
running VM disk images, and most of them can't be shut down in the
foreseeable future.

The "troublesome" node is configured to be the official slave, so it
sounds as though it won't affect the master's view of the data.

I'm guessing the sync operation will take quite a long time and cause
a performance hit on those VMs, so I'll have to do it out of hours.

James

ff at mpexnet

Jul 31, 2012, 2:33 AM

Post #7 of 30 (2564 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Hi,

On 07/31/2012 11:27 AM, JAMES GIBBON wrote:
> Ah, thanks Felix.
>
> So - I think I need to do, on the broken node:
>
> # drbdadm disconnect all
> # drbdadm secondary all
> # drbadm connect --discard-my-data all

without checking back with the Guide, this *looks* sound to me.
Disconnecting does not appear to be necessary, your node disconnected on
its own.

> Since the "survivor" node appears to be in "WFConnection" state
> according to its /proc/drbd, I don't need to do anything from there.

Correct.

> Is there a risk that it will attempt to connect without doing
> "--discard-my-data" when the correct IP address comes up?

As long as it's Standalone, no, not a real danger at all. If if you
*would* attempt such a connection, DRBD would severe the link upon split
brain detection.

What's bugging me a bit is the relatively high number of "out of sync"
blocks as reported by your second node. Are you absolutely certain that
no VM has been started on this node since you've lost connectivity? For
that matter, how certain are you about the exact time your nodes started
diverging?

You may want to check your logs. If possible, you may want to copy your
second node's data before syncing, just to be safe.

Good luck!
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

james.gibbon at virgin

Jul 31, 2012, 3:07 AM

Post #8 of 30 (2571 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On 31 July 2012 10:33, Felix Frank <ff [at] mpexnet> wrote:

> What's bugging me a bit is the relatively high number of "out
> of sync" blocks as reported by your second node. Are you
> absolutely certain that no VM has been started on this node
> since you've lost connectivity? For that matter, how certain
> are you about the exact time your nodes started diverging?
>
> You may want to check your logs. If possible, you may want to
> copy your second node's data before syncing, just to be safe.
>

Hi Felix,

I did start one or two of the VMs on the second node initially. That's
how I discovered that things had gone wrong. They did start but
gave horrendous errors and hung for half a minute at a time.

But for sure, there aren't any running now on the broken node. The
first node is running all of the VMs, quite smoothly fortunately.

So it's safe to assume that that was the point that the nodes started
to diverge, I believe. Certainly the second node has had the wrong IP
address on the NIC facing the storage since before I attempted to
start any of the VMs.

Not sure I even can copy the second node's data - it's not
communicating with the storage at all. But I only need the first node's
data.

Thanks again,
James

ff at mpexnet

Jul 31, 2012, 3:12 AM

Post #9 of 30 (2589 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On 07/31/2012 12:07 PM, JAMES GIBBON wrote:
> But for sure, there aren't any running now on the broken node. The
> first node is running all of the VMs, quite smoothly fortunately.

Good, so it's probably safe to trash the other node's data.

> Not sure I even can copy the second node's data - it's not
> communicating with the storage at all. But I only need the first node's
> data.

This too makes me edgy. The "storage"? In most DRBD setups, the data
resides on the local disk, and DRBD takes care of syncing local disks of
peers.

We have yet to see your drbd config.

Thanks,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

james.gibbon at virgin

Jul 31, 2012, 4:10 AM

Post #10 of 30 (2571 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Hi Felix,

On Tue, 31 Jul 2012 12:12:30 +0200
Felix Frank <ff [at] mpexnet> wrote:
>
> > Not sure I even can copy the second node's data - it's not
> > communicating with the storage at all. But I only need the
> > first node's data.
>
> This too makes me edgy. The "storage"? In most DRBD setups, the
> data resides on the local disk, and DRBD takes care of syncing
> local disks of peers.
>

OK. The VM images for the cluster are stored in two Cisco NAS
units. Both the physical servers are connected to these using
iSCSI, through two Cisco gigabit switches. So for example a
particular VM disk image might be visible, from both servers,
as /dev/mapper/drbdvg0-vm--104--disk--1.

Each of the servers has a private IP address used to read and
write to the storage, in additional to their regular IP address
used for ssh and other communication over the Internet.

On the first (master) server the private IP address, on the NIC
that's hooked up to the NAS units via the switches, is 10.0.1.1.
On the second it should be 10.0.1.2. But because I misconfigured
it, it's also 10.0.1.1 at the moment, which is where I assume
the split brain is derived from. The IP conflict is stopping the
slave from communicating properly with the NAS units, over iSCSI.

> We have yet to see your drbd config.
>

OK, thanks for taking a look at this:

/etc/drbd.conf is the same on both servers and looks like this:

include "drbd.d/global_common.conf";
include "drbd.d/*.res"

.. global_common.conf looks like:

global { usage-count no; }
common { syncer { rate 30M; } }

Both have an identical .res file in drbd.d/ with a lot more
information, which I've pasted here:

http://www.pastedump.com/paste/2365

Thanks again.

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jul 31, 2012, 4:24 AM

Post #11 of 30 (2576 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Hi,

On 07/31/2012 01:10 PM, James Gibbon wrote:
> OK. The VM images for the cluster are stored in two Cisco NAS
> units. Both the physical servers are connected to these using
> iSCSI, through two Cisco gigabit switches. So for example a
> particular VM disk image might be visible, from both servers,
> as /dev/mapper/drbdvg0-vm--104--disk--1.

I see. Quite unusual I'd say, to have two drbd nodes that each use a
NAS as backing device. But it looks sound, judging from the config.
Thanks for that.

Out of curiosity: Do you gain *any* advantage from using NAS in this
setup instead of local disks in your drbd nodes?
I'd like to point out that the drbd latency cost per write in this setup
is probably (RTT between nodes) + (RTT between secondary and NAS), which
may be small overhead, but potentially sub-optimal nontheless.

Cheers,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

james.gibbon at virgin

Jul 31, 2012, 4:43 AM

Post #12 of 30 (2564 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Hi Felix,

On Tue, 31 Jul 2012 13:24:38 +0200
Felix Frank <ff [at] mpexnet> wrote:
>
> I see. Quite unusual I'd say, to have two drbd nodes that each
> use a NAS as backing device. But it looks sound, judging from
> the config. Thanks for that.
>

Thanks a lot for taking a look! I take it the strategy we've
already discussed for restoring sanity applies just as well to
this setup, then.

> Out of curiosity: Do you gain *any* advantage from using NAS in
> this setup instead of local disks in your drbd nodes?
> I'd like to point out that the drbd latency cost per write in
> this setup is probably (RTT between nodes) + (RTT between
> secondary and NAS), which may be small overhead, but
> potentially sub-optimal nontheless.
>

That's an interesting point. This is the only time I've ever worked
with DRBD, and I inherited these servers - so I can't compare. My
personal preference would certainly be to use local disk, if only
to circumvent the need for additional cabling and switches.

James

--
James Gibbon <james.gibbon [at] virgin>
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

james.gibbon at virgin

Jul 31, 2012, 5:49 AM

Post #13 of 30 (2562 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Can someone tell me what the "become-primary-on-both" part means? I'm
fairly anxious to ensure that the second node doesn't attempt to become
primary when its restarted as its data will be out of date..

startup {
wfc-timeout 15;
degr-wfc-timeout 60;
become-primary-on both;
}

http://www.pastedump.com/paste/2365
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jul 31, 2012, 6:03 AM

Post #14 of 30 (2583 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On 07/31/2012 02:49 PM, James Gibbon wrote:
> Can someone tell me what the "become-primary-on-both" part means? I'm
> fairly anxious to ensure that the second node doesn't attempt to become
> primary when its restarted as its data will be out of date..

It does what you assume. The initscript will promote either node upon
booting. This will cause split brain if connection isn't established first.
Please note that there are other ways to start DRBD besides the
initscript, notably pacemaker.

I only just noticed this: You do have a dual-primary setup. I shall
assume this is for live migration purposes and/or mixing active LVs
among peers (i.e. in one VG, each peer has a couple of active LVs. I
believe you need cLVM for that).

You'll have to ask your predecessors about the specifics I'm afraid ;)

HTH,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

jsmith at argotec

Jul 31, 2012, 6:07 AM

Post #15 of 30 (2576 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

----- Original Message -----
> From: "James Gibbon" <james.gibbon [at] virgin>
> To: "James Gibbon" <james.gibbon [at] virgin>
> Cc: drbd-user [at] lists
> Sent: Tuesday, July 31, 2012 8:49:52 AM
> Subject: Re: [DRBD-user] DRBD - one half of Proxmox cluster miscommunicating
>
>
> Can someone tell me what the "become-primary-on-both" part means? I'm
> fairly anxious to ensure that the second node doesn't attempt to
> become
> primary when its restarted as its data will be out of date..
>
>
> startup {
> wfc-timeout 15;
> degr-wfc-timeout 60;
> become-primary-on both;
> }
>
>
> http://www.pastedump.com/paste/2365

Just what it sounds like - both servers will be allowed to be primary at the same time - "dual primary"

Short version - when secondary comes up it will make sure it is uptodate the it will also be able to become primary at the same time. However you also have settings that control what happens when the two nodes are not uptodate (these settings are right out of the DRBD users guide for automatic split brain recovery in a dual primary setup - links after):

net {
cram-hmac-alg sha1;
shared-secret "abcxxx";
allow-two-primaries;
after-sb-0pri discard-zero-changes;

## sb-1pri says to discard data on the secondary and resync if there has only been the one primary since the last uptodate status
after-sb-1pri discard-secondary;

## sb-2pri says to disconnect from each other if they have both been primary since the last uptodate - I believe this is where you are at currently. They have both been primary when they were not connected and uptodate so DRBD is disconnecting them so that you don't overwrite possibly good data. Manual recovery required
after-sb-2pri disconnect;
}

More here:
http://www.drbd.org/users-guide-8.3/s-enable-dual-primary.html
http://www.drbd.org/users-guide-8.3/s-configure-split-brain-behavior.html

HTH

Jake

> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
>
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

james.gibbon at virgin

Jul 31, 2012, 6:23 AM

Post #16 of 30 (2573 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On Tue, 31 Jul 2012 15:03:22 +0200
Felix Frank <ff [at] mpexnet> wrote:

> On 07/31/2012 02:49 PM, James Gibbon wrote:
> > Can someone tell me what the "become-primary-on-both" part
> > means? I'm fairly anxious to ensure that the second node
> > doesn't attempt to become primary when its restarted as its
> > data will be out of date..
>
> It does what you assume. The initscript will promote either
> node upon booting. This will cause split brain if connection
> isn't established first. Please note that there are other ways
> to start DRBD besides the initscript, notably pacemaker.
>

OK - thanks again. Since the master is already up and running,
I'm hoping that the broken secondary box isn't going to get
promoted - is that a reasonable assumption?

Sorry if I'm being anally careful about this. The consequences
of losing the current sane view of the data would be quite serious
unfortunately.

> I only just noticed this: You do have a dual-primary setup. I
> shall assume this is for live migration purposes and/or mixing
> active LVs among peers (i.e. in one VG, each peer has a couple
> of active LVs. I believe you need cLVM for that).
>

Yes, it is possible to migrate a VM from one server to the other
while it's running - a very nice feature.

So given that it's a dual primary setup, is the strategy of doing

1. fix network config on server2 and reboot it
2. server2 # drbdadm disconnect all
3. server2 # drbdadm secondary all
4. server2 # drbadm connect --discard-my-data all

Still going to work?

Thanks ..
James

--
James Gibbon <james.gibbon [at] virgin>
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jul 31, 2012, 6:24 AM

Post #17 of 30 (2568 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Hi,

On 07/31/2012 03:07 PM, Jake Smith wrote:
> ## sb-1pri says to discard data on the secondary and resync if there has only been the one primary since the last uptodate status

uhm, no. It's a policy for "what to do if there is only 1 primary at the
time of connecting".
If there has been only 1 primary during disconnection, there is no split
brain.

Being such...

> after-sb-1pri discard-secondary;

this is an extremely dangerous setting.

If for any reason, at the time of reconnection the node with the stale
data is primary, you loose your current data.

Best,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jul 31, 2012, 6:27 AM

Post #18 of 30 (2563 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Hi,

On 07/31/2012 03:23 PM, James Gibbon wrote:
> OK - thanks again. Since the master is already up and running,
> I'm hoping that the broken secondary box isn't going to get
> promoted - is that a reasonable assumption?

it's already promoted. Hence split brain. It's not an issue though.

It's easy to mix up "in primary role" and "source for resync". The two
are largely independend. In your setup, you need not even demote your
split brain victim.

> 1. fix network config on server2 and reboot it
> 2. server2 # drbdadm disconnect all
> 3. server2 # drbdadm secondary all
> 4. server2 # drbadm connect --discard-my-data all

Actually:
1. configure the DRBD IP correctly
2. drbadm connect --discard-my-data all

Should do the trick.

Again, the single most important thing is to be 100% certain you're
discarding the right dataset (i.e., the one on your "second" node).

HTH,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

james.gibbon at virgin

Jul 31, 2012, 7:23 AM

Post #19 of 30 (2562 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On Tue, 31 Jul 2012 15:27:52 +0200
Felix Frank <ff [at] mpexnet> wrote:

> Hi,
>
> On 07/31/2012 03:23 PM, James Gibbon wrote:
> > OK - thanks again. Since the master is already up and running,
> > I'm hoping that the broken secondary box isn't going to get
> > promoted - is that a reasonable assumption?
>
> it's already promoted. Hence split brain. It's not an issue
> though.
>
> It's easy to mix up "in primary role" and "source for resync".
> The two are largely independend. In your setup, you need not
> even demote your split brain victim.
>

Thanks again for your reply, Felix. I'm still a bit confused though,
I'm sorry to say. In your earlier reply to Jake, you said:

>
> > after-sb-1pri discard-secondary;
>
> this is an extremely dangerous setting.
>
> If for any reason, at the time of reconnection the node with
> the stale data is primary, you loose your current data.
>

My config DOES have this setting. In your earlier comment you say
that it's not an issue that the secondary box might get promoted
- but here you say that if it is a primary when it reconnects, the
current data will be lost..

Sorry to remain puzzled, appreciate the help;
James

--
James Gibbon <james.gibbon [at] virgin>
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jul 31, 2012, 8:04 AM

Post #20 of 30 (2566 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Hi,

On 07/31/2012 04:23 PM, James Gibbon wrote:
>
> Thanks again for your reply, Felix. I'm still a bit confused though,

understandably so I guess ;-)

> I'm sorry to say. In your earlier reply to Jake, you said:
>
>
>> >
>> >
>> > this is an extremely dangerous setting.
>> >
>> > If for any reason, at the time of reconnection the node with
>> > the stale data is primary, you loose your current data.
>> >
>
> My config DOES have this setting. In your earlier comment you say
> that it's not an issue that the secondary box might get promoted
> - but here you say that if it is a primary when it reconnects, the
> current data will be lost..
>
> Sorry to remain puzzled, appreciate the help;
> James

I stand by this assessment. You got "lucky" insofar that both nodes were
primary when they saw each other again. There is no autorecovery from that.
If for some freak reason your "good" node would have been in a demoted
state at such a time, the stale node would have killed its data.

Please note: If you would have been even more "lucky", your bad node
would have been secondary and your policy would have done the right
thing. The question you have to ask yourself boils down to "how lucky do
you feel in the long run?" ;)

Again: It's paramount to be absolutely sure which dataset is discarded.
The above setting makes the decision a fair bit more arbitrary.

Cheers,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

james.gibbon at virgin

Jul 31, 2012, 8:19 AM

Post #21 of 30 (2559 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Hello,

On Tue, 31 Jul 2012 17:04:02 +0200
Felix Frank <ff [at] mpexnet> wrote:

> I stand by this assessment. You got "lucky" insofar that both
> nodes were primary when they saw each other again. There is no
> autorecovery from that. If for some freak reason your "good"
> node would have been in a demoted state at such a time, the
> stale node would have killed its data.
>

Ah I see .. so because the "good" node is in a primary state at
the moment, it's not at risk when I bring up the interface on
the "bad" node. I only get bitten by that if the first node
running the VMs becomes secondary. And there's no reason to
suppose that will happen. Is that right?

> Please note: If you would have been even more "lucky", your bad
> node would have been secondary and your policy would have done
> the right thing. The question you have to ask yourself boils
> down to "how lucky do you feel in the long run?" ;)
>
> Again: It's paramount to be absolutely sure which dataset is
> discarded. The above setting makes the decision a fair bit more
> arbitrary.
>

Well - all of the VMs are running on the first node and everything
is fine and up to date on those. So I'm sure I need to discard the
second server's data and sync it up from the first.

Many thanks
James

--
James Gibbon <james.gibbon [at] virgin>
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

ff at mpexnet

Jul 31, 2012, 8:21 AM

Post #22 of 30 (2567 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On 07/31/2012 05:19 PM, James Gibbon wrote:
> Is that right?

Yes.

I'd still consider losing the split brain resolution option.
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

jsmith at argotec

Jul 31, 2012, 8:41 AM

Post #23 of 30 (2574 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

Felix - thank you for correcting that!

Jake

-------- Original Message --------
From: Felix Frank
Sent: Tue, 31/07/2012 09:25 AM
To: Jake Smith
CC: James Gibbon ; drbd-user [at] lists
Subject: Re: [DRBD-user] DRBD - one half of Proxmox cluster miscommunicating

james.gibbon at virgin

Jul 31, 2012, 8:54 AM

Post #24 of 30 (2564 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

On Tue, 31 Jul 2012 17:21:14 +0200
Felix Frank <ff [at] mpexnet> wrote:

> On 07/31/2012 05:19 PM, James Gibbon wrote:
> > Is that right?
>
> Yes.
>
> I'd still consider losing the split brain resolution option.
>

Many thanks, Felix. This has all been a useful learning experience
as well as a problem resolution resource. I'm going to run the fix
shortly.

One last question: why does /proc/drbd on the broken node show as
"UpToDate" in /proc/drbd? Is that because it's a primary, and not
able to communicate with the other side, therefore it's entitled to
consider itself current?

But provided it doesn't attempt to impose its version of "UpToDate"
on the other half, I'm safe.

James

--
James Gibbon <james.gibbon [at] virgin>
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

jg at jamesgibbon

Jul 31, 2012, 10:39 AM

Post #25 of 30 (2549 views)
Permalink

Re: DRBD - one half of Proxmox cluster miscommunicating [In reply to]

OK. This didn't quite go to plan.

I assigned the proper IP address to the DRDB NIC on the secondary
successfully.

But:

Firstly my version of drbdadm doesn't support "--discard-my-data"

# "drbdadm connect --discard-my-data all"
drbdadm: unrecognized option `--discard-my-data'
try 'drbdadm help'

A bit of Googling suggested that this might help:

# drbdadm -- --discard-my-data connect all

- allegedly to pass the option straight through to drbdsetup.

But that gave an error - complaining that:

0: Failure: (123) --discard-my-data not allowed when primary.

So I then tried:

# drbdadm secondary all
0: State change failed: (-12) Device is held open by someone
Command 'drbdsetup 0 secondary' terminated with exit code 11
1: State change failed: (-12) Device is held open by someone
Command 'drbdsetup 1 secondary' terminated with exit code 11
pves2:/etc/network#

Google then suggested:

# vgchange -an <volume group>

.. so I ran that on drbdvg and drbdvg1.

Then "drbdadm secondary all" worked successfully, following
which, "drbdadm connect --discard-my-data all" was also
happy to run.

I watched /proc/drbd while the mirror synced up - it displays
nice little progress bars like this:

# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:1253940 nr:0 dw:41622456 dr:181039187 al:519664 bm:519738 lo:18 pe:223 ua:60 ap:8 ep:1 wo:b oos:10332000
[=>..................] sync'ed: 10.8% (10088/11300)M
finish: 0:17:43 speed: 9,672 (12,680) K/sec
1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:1296696 nr:0 dw:4851300 dr:17169105 al:1699 bm:1946 lo:0 pe:120 ua:0 ap:8 ep:1 wo:b oos:742860
[===========>........] sync'ed: 63.6% (742860/2034536)K
finish: 0:00:46 speed: 16,108 (13,180) K/sec

.. and it completed successfully. Comfortingly, the VMs on the first node
continued to work properly.

Anyway .. since it was now in Primary/Secondary and the logical volumes were
unavailable, I rebooted the second box.

And finally,

# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
0: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
ns:11689952 nr:0 dw:41746996 dr:191403115 al:522263 bm:522764 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
ns:2073056 nr:0 dw:4888628 dr:17917209 al:1706 bm:2088 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
#

Deep joy.

Many thanks for the help, and I hope this thread will prove useful to
some other victim of their career choice at some point in the future.

James
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads