Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DRBD: Users

DRBD+Pacemaker: Won't promote with only one node

 

 

DRBD users RSS feed   Index | Next | Previous | View Threaded


seligman at nevis

Jan 4, 2012, 12:10 PM

Post #1 of 10 (1574 views)
Permalink
DRBD+Pacemaker: Won't promote with only one node

I'll give the technical details in a moment, but I thought I'd start with a
description of the problem.

I have a two-node active/passive cluster, with DRBD controlled by Pacemaker. I
upgraded to DRBD 8.4.x about six months ago (probably too soon); everything was
fine. Then last week we did some power-outage tests on our cluster.

Each node in the cluster is attached to its own uninterruptible power supply;
the STONITH mechanism is to turn off the other node's UPS. In the event of an
extended power outage (this happens 2-3 times a year at my site), it's likely
that one node will STONITH the other when the other node's UPS runs out of power
and shuts it down. This means that when power comes back on, only one node will
come back up, since the STONITHed UPS won't turn on again without manual
intervention.

The problem is that with only one node, Pacemaker+DRBD won't promote the DRBD
resource to primary; it just sits there at secondary and won't start up any
DRBD-dependent resources. Only when the second node comes back up will Pacemaker
assign one of them the primary role. I've confirmed this by shutting down
corosync on both nodes, then bringing it up again on just one of them.

I'm pretty sure that this is due to a mistake I"ve made in made in my DRBD
configuration when I fiddled with it during the 8.4.x upgrade. I've attached the
files. Can one of you kind folks spot the error?

Technical details:

Two-node configuration: hypatia and orestes
OS: Scientific Linux 5.5, kernel 2.6.18-238.19.1.el5xen
Packages:
drbd-8.4.1-1
corosync-1.2.7-1.1.el5
pacemaker-1.0.12-1.el5.centos
openais-1.1.3-1.6.el5

Attached: global_common.conf, nevis-admin.res

--
Bill Seligman | Phone: (914) 591-2823
Nevis Labs, Columbia Univ | mailto://seligman [at] nevis
PO Box 137 |
Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/
Attachments: global_common.conf (0.60 KB)
  nevis-admin.res (0.88 KB)
  smime.p7s (4.39 KB)


dbarker at visioncomm

Jan 4, 2012, 12:58 PM

Post #2 of 10 (1515 views)
Permalink
Re: DRBD+Pacemaker: Won't promote with only one node [In reply to]

I'd say the error is in the STONITH method. You evidently are giving the UPS a "SHUTDOWN" command when you should be giving it a "SLEEP" or "SUSPEND" command (Whatever your UPS Vendor's idea of power off the outlets only until the mains come on and have charged the batteries to above 5% or whatever. With the APC family and a Network Card, there are very fine controls over this sort of action. The APIs published are fairly primitive. I had to write SNMP routines to make my APC do what I wanted, when I wanted. The doc was like pulling teeth to find. If you have APC equipment, I can share. If not, what do you have? What controls does it publish?

Dan

-----Original Message-----
From: drbd-user-bounces [at] lists [mailto:drbd-user-bounces [at] lists] On Behalf Of William Seligman
Sent: Wednesday, January 04, 2012 3:10 PM
To: drbd-user [at] lists
Subject: [DRBD-user] DRBD+Pacemaker: Won't promote with only one node

I'll give the technical details in a moment, but I thought I'd start with a description of the problem.

I have a two-node active/passive cluster, with DRBD controlled by Pacemaker. I upgraded to DRBD 8.4.x about six months ago (probably too soon); everything was fine. Then last week we did some power-outage tests on our cluster.

Each node in the cluster is attached to its own uninterruptible power supply; the STONITH mechanism is to turn off the other node's UPS. In the event of an extended power outage (this happens 2-3 times a year at my site), it's likely that one node will STONITH the other when the other node's UPS runs out of power and shuts it down. This means that when power comes back on, only one node will come back up, since the STONITHed UPS won't turn on again without manual intervention.

The problem is that with only one node, Pacemaker+DRBD won't promote the DRBD resource to primary; it just sits there at secondary and won't start up any DRBD-dependent resources. Only when the second node comes back up will Pacemaker assign one of them the primary role. I've confirmed this by shutting down corosync on both nodes, then bringing it up again on just one of them.

I'm pretty sure that this is due to a mistake I"ve made in made in my DRBD configuration when I fiddled with it during the 8.4.x upgrade. I've attached the files. Can one of you kind folks spot the error?

Technical details:

Two-node configuration: hypatia and orestes
OS: Scientific Linux 5.5, kernel 2.6.18-238.19.1.el5xen
Packages:
drbd-8.4.1-1
corosync-1.2.7-1.1.el5
pacemaker-1.0.12-1.el5.centos
openais-1.1.3-1.6.el5

Attached: global_common.conf, nevis-admin.res

--
Bill Seligman | Phone: (914) 591-2823
Nevis Labs, Columbia Univ | mailto://seligman [at] nevis
PO Box 137 |
Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


df.cluster at gmail

Jan 5, 2012, 12:34 AM

Post #3 of 10 (1510 views)
Permalink
Re: DRBD+Pacemaker: Won't promote with only one node [In reply to]

Hi,

On Wed, Jan 4, 2012 at 10:10 PM, William Seligman
<seligman [at] nevis> wrote:
> I'll give the technical details in a moment, but I thought I'd start with a
> description of the problem.
>
> I have a two-node active/passive cluster, with DRBD controlled by Pacemaker. I
> upgraded to DRBD 8.4.x about six months ago (probably too soon); everything was
> fine. Then last week we did some power-outage tests on our cluster.
>
> Each node in the cluster is attached to its own uninterruptible power supply;
> the STONITH mechanism is to turn off the other node's UPS. In the event of an
> extended power outage (this happens 2-3 times a year at my site), it's likely
> that one node will STONITH the other when the other node's UPS runs out of power
> and shuts it down. This means that when power comes back on, only one node will
> come back up, since the STONITHed UPS won't turn on again without manual
> intervention.
>
> The problem is that with only one node, Pacemaker+DRBD won't promote the DRBD
> resource to primary; it just sits there at secondary and won't start up any
> DRBD-dependent resources. Only when the second node comes back up will Pacemaker
> assign one of them the primary role. I've confirmed this by shutting down
> corosync on both nodes, then bringing it up again on just one of them.
>

Could you also post your Pacemaker configuration?

Also you might want to check
http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/#id890288
for no-quorum-policy, in two-node clusters, losing one node means you
don't have quorum, and unless you something else as a quorum device,
then the policy is set to stop.

HTH,
Dan

> I'm pretty sure that this is due to a mistake I"ve made in made in my DRBD
> configuration when I fiddled with it during the 8.4.x upgrade. I've attached the
> files. Can one of you kind folks spot the error?
>
> Technical details:
>
> Two-node configuration: hypatia and orestes
> OS: Scientific Linux 5.5, kernel 2.6.18-238.19.1.el5xen
> Packages:
> drbd-8.4.1-1
> corosync-1.2.7-1.1.el5
> pacemaker-1.0.12-1.el5.centos
> openais-1.1.3-1.6.el5
>
> Attached: global_common.conf, nevis-admin.res
>
> --
> Bill Seligman             | Phone: (914) 591-2823
> Nevis Labs, Columbia Univ | mailto://seligman [at] nevis
> PO Box 137                |
> Irvington NY 10533 USA    | http://www.nevis.columbia.edu/~seligman/
>
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
>



--
Dan Frincu
CCNA, RHCE
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


seligman at nevis

Jan 5, 2012, 9:06 AM

Post #4 of 10 (1502 views)
Permalink
Re: DRBD+Pacemaker: Won't promote with only one node [In reply to]

> Message: 1
> Date: Wed, 4 Jan 2012 15:58:09 -0500
> From: "Dan Barker" <dbarker [at] visioncomm>
> Subject: Re: [DRBD-user] DRBD+Pacemaker: Won't promote with only one
> node
> To: <drbd-user [at] lists>
> Message-ID: <01a901cccb23$90815270$b183f750$@visioncomm.net>
> Content-Type: text/plain; charset="UTF-8"
>
> I'd say the error is in the STONITH method. You evidently are giving the UPS a "SHUTDOWN" command when you should be giving it a "SLEEP" or "SUSPEND" command (Whatever your UPS Vendor's idea of power off the outlets only until the mains come on and have charged the batteries to above 5% or whatever. With the APC family and a Network Card, there are very fine controls over this sort of action. The APIs published are fairly primitive. I had to write SNMP routines to make my APC do what I wanted, when I wanted. The doc was like pulling teeth to find. If you have APC equipment, I can share. If not, what do you have? What controls does it publish?

I'm using APC SMART-UPSes, and issuing the SHUTDOWN command as you suspected. I
wouldn't mind seeing the SNMP write-up you've got on the obscure APC API.

However, I believe what I've got is what I want to do. Suppose one node STONITHs
another for reasons that have nothing to do with a power outage. I don't want
the STONITHed UPS to come back on for any reason. I'm concerned about the
(admittedly unlikely) possibility that a node is STONITHed because it goes
wonky, and then there's a power outage. Upon power recovery I'd get the wonky
node trying to rejoin the cluster again.

So I think I've got STONITH set up satisfactorily. I just need help figuring out
why a single node's DRBD resource is not being promoted to primary after a restart.

On 1/4/12 3:10 PM, William Seligman wrote:
> I'll give the technical details in a moment, but I thought I'd start with a
> description of the problem.
>
> I have a two-node active/passive cluster, with DRBD controlled by Pacemaker. I
> upgraded to DRBD 8.4.x about six months ago (probably too soon); everything was
> fine. Then last week we did some power-outage tests on our cluster.
>
> Each node in the cluster is attached to its own uninterruptible power supply;
> the STONITH mechanism is to turn off the other node's UPS. In the event of an
> extended power outage (this happens 2-3 times a year at my site), it's likely
> that one node will STONITH the other when the other node's UPS runs out of power
> and shuts it down. This means that when power comes back on, only one node will
> come back up, since the STONITHed UPS won't turn on again without manual
> intervention.
>
> The problem is that with only one node, Pacemaker+DRBD won't promote the DRBD
> resource to primary; it just sits there at secondary and won't start up any
> DRBD-dependent resources. Only when the second node comes back up will Pacemaker
> assign one of them the primary role. I've confirmed this by shutting down
> corosync on both nodes, then bringing it up again on just one of them.
>
> I'm pretty sure that this is due to a mistake I"ve made in made in my DRBD
> configuration when I fiddled with it during the 8.4.x upgrade. I've attached the
> files. Can one of you kind folks spot the error?
>
> Technical details:
>
> Two-node configuration: hypatia and orestes
> OS: Scientific Linux 5.5, kernel 2.6.18-238.19.1.el5xen
> Packages:
> drbd-8.4.1-1
> corosync-1.2.7-1.1.el5
> pacemaker-1.0.12-1.el5.centos
> openais-1.1.3-1.6.el5


--
Bill Seligman | Phone: (914) 591-2823
Nevis Labs, Columbia Univ | mailto://seligman [at] nevis
PO Box 137 |
Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/
Attachments: smime.p7s (4.39 KB)


dbarker at visioncomm

Jan 5, 2012, 9:32 AM

Post #5 of 10 (1534 views)
Permalink
Re: DRBD+Pacemaker: Won't promote with only one node [In reply to]

I see your point. I was only thinking about the power failure scenario. Maybe you could shut down differently depending on the reason for the STONITH?

The problem I had, was that the APC-provided shutdown process was pretty dumb, I had two hours of battery and it takes about 45 minutes to gracefully shut everything down. So, my program monitors the battery remaining, and when it gets to 45 minutes, starts shutting down VMs. If the power comes back on, it restarts the VMs it's stopped. The APC version once begun would continue the shutdown even if the power came back on. I finally implemented this using:

string UPSOff = ".1.3.6.1.4.1.318.1.1.1.6.2.1.0 i 3" ;
string UPSSleep = ".1.3.6.1.4.1.318.1.1.1.6.2.3.0 i 3" ;
read the Utility voltage to see if the power has come back on (.1.3.6.1.4.1.318.1.1.1.3.2.1.0)
Battery remaining (.1.3.6.1.4.1.318.1.1.1.2.2.3.0)

I needed to use UPSSleep. I had been using USPOff. I use Battery Remaining and Utility voltage to control if to abandon/reverse my suspends. After I’ve suspended all the VMs, shutdown the ESXi hosts and stopped the SAN, a UPSSleep will only go lights out until the utility power comes back on. OID codes were found at:

http://support.ipmonitor.com/mibs/POWERNET-MIB/oids.aspx (says it's unmaintained, but worked for my SUA3000XL / SUA48XLBP.

(I don't see any DRBD in my reply. I guess we've drifted off topic<g>)

Dan

-----Original Message-----
From: drbd-user-bounces [at] lists [mailto:drbd-user-bounces [at] lists] On Behalf Of William Seligman
Sent: Thursday, January 05, 2012 12:06 PM
To: drbd-user [at] lists
Subject: Re: [DRBD-user] DRBD+Pacemaker: Won't promote with only one node

> Message: 1
> Date: Wed, 4 Jan 2012 15:58:09 -0500
> From: "Dan Barker" <dbarker [at] visioncomm>
> Subject: Re: [DRBD-user] DRBD+Pacemaker: Won't promote with only one
> node
> To: <drbd-user [at] lists>
> Message-ID: <01a901cccb23$90815270$b183f750$@visioncomm.net>
> Content-Type: text/plain; charset="UTF-8"
>
> I'd say the error is in the STONITH method. You evidently are giving the UPS a "SHUTDOWN" command when you should be giving it a "SLEEP" or "SUSPEND" command (Whatever your UPS Vendor's idea of power off the outlets only until the mains come on and have charged the batteries to above 5% or whatever. With the APC family and a Network Card, there are very fine controls over this sort of action. The APIs published are fairly primitive. I had to write SNMP routines to make my APC do what I wanted, when I wanted. The doc was like pulling teeth to find. If you have APC equipment, I can share. If not, what do you have? What controls does it publish?

I'm using APC SMART-UPSes, and issuing the SHUTDOWN command as you suspected. I wouldn't mind seeing the SNMP write-up you've got on the obscure APC API.

However, I believe what I've got is what I want to do. Suppose one node STONITHs another for reasons that have nothing to do with a power outage. I don't want the STONITHed UPS to come back on for any reason. I'm concerned about the (admittedly unlikely) possibility that a node is STONITHed because it goes wonky, and then there's a power outage. Upon power recovery I'd get the wonky node trying to rejoin the cluster again.

So I think I've got STONITH set up satisfactorily. I just need help figuring out why a single node's DRBD resource is not being promoted to primary after a restart.

On 1/4/12 3:10 PM, William Seligman wrote:
> I'll give the technical details in a moment, but I thought I'd start
> with a description of the problem.
>
> I have a two-node active/passive cluster, with DRBD controlled by
> Pacemaker. I upgraded to DRBD 8.4.x about six months ago (probably too
> soon); everything was fine. Then last week we did some power-outage tests on our cluster.
>
> Each node in the cluster is attached to its own uninterruptible power
> supply; the STONITH mechanism is to turn off the other node's UPS. In
> the event of an extended power outage (this happens 2-3 times a year
> at my site), it's likely that one node will STONITH the other when the
> other node's UPS runs out of power and shuts it down. This means that
> when power comes back on, only one node will come back up, since the
> STONITHed UPS won't turn on again without manual intervention.
>
> The problem is that with only one node, Pacemaker+DRBD won't promote
> the DRBD resource to primary; it just sits there at secondary and
> won't start up any DRBD-dependent resources. Only when the second node
> comes back up will Pacemaker assign one of them the primary role. I've
> confirmed this by shutting down corosync on both nodes, then bringing it up again on just one of them.
>
> I'm pretty sure that this is due to a mistake I"ve made in made in my
> DRBD configuration when I fiddled with it during the 8.4.x upgrade.
> I've attached the files. Can one of you kind folks spot the error?
>
> Technical details:
>
> Two-node configuration: hypatia and orestes
> OS: Scientific Linux 5.5, kernel 2.6.18-238.19.1.el5xen
> Packages:
> drbd-8.4.1-1
> corosync-1.2.7-1.1.el5
> pacemaker-1.0.12-1.el5.centos
> openais-1.1.3-1.6.el5


--
Bill Seligman | Phone: (914) 591-2823
Nevis Labs, Columbia Univ | mailto://seligman [at] nevis
PO Box 137 |
Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/


_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


seligman at nevis

Jan 5, 2012, 9:36 AM

Post #6 of 10 (1498 views)
Permalink
Re: DRBD+Pacemaker: Won't promote with only one node [In reply to]

> Message: 3
> Date: Thu, 5 Jan 2012 10:34:28 +0200
> From: Dan Frincu <df.cluster [at] gmail>
> Subject: Re: [DRBD-user] DRBD+Pacemaker: Won't promote with only one
> node
> To: drbd-user [at] lists
> Message-ID:
> <CADQRkwiTL-rPDx_4JUDuzsVnCe-ED4UihJNeV7gaVgdSnR5cZw [at] mail>
> Content-Type: text/plain; charset=UTF-8
>
> Hi,
>
> On Wed, Jan 4, 2012 at 10:10 PM, William Seligman
> <seligman [at] nevis> wrote:
>> I'll give the technical details in a moment, but I thought I'd start with a
>> description of the problem.
>>
>> I have a two-node active/passive cluster, with DRBD controlled by Pacemaker. I
>> upgraded to DRBD 8.4.x about six months ago (probably too soon); everything was
>> fine. Then last week we did some power-outage tests on our cluster.
>>
>> Each node in the cluster is attached to its own uninterruptible power supply;
>> the STONITH mechanism is to turn off the other node's UPS. In the event of an
>> extended power outage (this happens 2-3 times a year at my site), it's likely
>> that one node will STONITH the other when the other node's UPS runs out of power
>> and shuts it down. This means that when power comes back on, only one node will
>> come back up, since the STONITHed UPS won't turn on again without manual
>> intervention.
>>
>> The problem is that with only one node, Pacemaker+DRBD won't promote the DRBD
>> resource to primary; it just sits there at secondary and won't start up any
>> DRBD-dependent resources. Only when the second node comes back up will Pacemaker
>> assign one of them the primary role. I've confirmed this by shutting down
>> corosync on both nodes, then bringing it up again on just one of them.
>>
>
> Could you also post your Pacemaker configuration?

Sure. I didn't do this before, since the configuration is complex. I also don't
know which would be more comprehensible, so I've attached both cib.xml and the
result of "crm configure show". I should mention that I'm a lazy bum, so I use
crm-gui to configure corosync; that's why these files look more baroque than usual.

To keep everything in one place, I've also attached the DRBD configuration files
again.

> Also you might want to check
> http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/#id890288
> for no-quorum-policy, in two-node clusters, losing one node means you
> don't have quorum, and unless you something else as a quorum device,
> then the policy is set to stop.

As you'll see, I already have this in my setup:

crm configure property no-quorum-policy=ignore

Another thing I left out: The cluster configuration was working "once upon a
time." The only significant change I made in recent months was upgrading to DRBD
8.4.0 (then to 8.4.1), then changing my DRBD configuration to add fencing plus
split-brain recovery. I don't think any of the changes I've made to the corosync
configuration would have any bearing on DRBD resource promotion when only one
node is available... but I could be wrong.

>> I'm pretty sure that this is due to a mistake I"ve made in made in my DRBD
>> configuration when I fiddled with it during the 8.4.x upgrade. I've attached the
>> files. Can one of you kind folks spot the error?
>>
>> Technical details:
>>
>> Two-node configuration: hypatia and orestes
>> OS: Scientific Linux 5.5, kernel 2.6.18-238.19.1.el5xen
>> Packages:
>> drbd-8.4.1-1
>> corosync-1.2.7-1.1.el5
>> pacemaker-1.0.12-1.el5.centos
>> openais-1.1.3-1.6.el5
>>

Attached: cib.xml, crm-configure-show.txt, global_common.conf, nevis-admin.res

--
Bill Seligman | Phone: (914) 591-2823
Nevis Labs, Columbia Univ | mailto://seligman [at] nevis
PO Box 137 |
Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/
Attachments: cib.xml (44.1 KB)
  crm-configure-show.txt (15.5 KB)
  global_common.conf (0.60 KB)
  nevis-admin.res (0.88 KB)
  smime.p7s (4.39 KB)


florian at hastexo

Jan 5, 2012, 2:01 PM

Post #7 of 10 (1497 views)
Permalink
Re: DRBD+Pacemaker: Won't promote with only one node [In reply to]

On Thu, Jan 5, 2012 at 6:36 PM, William Seligman
<seligman [at] nevis> wrote:
> Sure. I didn't do this before, since the configuration is complex. I also don't
> know which would be more comprehensible, so I've attached both cib.xml and the
> result of "crm configure show". I should mention that I'm a lazy bum, so I use
> crm-gui to configure corosync; that's why these files look more baroque than usual.

In the CIB you posted, both nodes are in the "UNCLEAN (offline)"
state. In that state, nothing gets promoted, nothing gets started. Are
you sure you posted the right CIB dump?

Cheers,
Florian

--
Need help with High Availability?
http://www.hastexo.com/now
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


seligman at nevis

Jan 6, 2012, 8:54 AM

Post #8 of 10 (1536 views)
Permalink
Re: DRBD+Pacemaker: Won't promote with only one node [In reply to]

> Message: 1
> Date: Thu, 5 Jan 2012 23:01:24 +0100
> From: Florian Haas <florian [at] hastexo>
> Subject: Re: [DRBD-user] DRBD+Pacemaker: Won't promote with only one
> node
> To: drbd-user [at] lists
> Message-ID:
> <CAPUexz9jgpq0V49eZmwDH3cThrGEJ7VopXCh1_27yjX-6JC+fQ [at] mail>
> Content-Type: text/plain; charset=UTF-8
>
> On Thu, Jan 5, 2012 at 6:36 PM, William Seligman
> <seligman [at] nevis> wrote:
>> Sure. I didn't do this before, since the configuration is complex. I also don't
>> know which would be more comprehensible, so I've attached both cib.xml and the
>> result of "crm configure show". I should mention that I'm a lazy bum, so I use
>> crm-gui to configure corosync; that's why these files look more baroque than usual.

> In the CIB you posted, both nodes are in the "UNCLEAN (offline)"
> state. In that state, nothing gets promoted, nothing gets started. Are
> you sure you posted the right CIB dump?

That caused me to panic! I rushed to check that my cluster was running.
It's OK; an occasional panic is good once in a while.

I assume you're talking about the following lines:

<nodes>
<node id="orestes.nevis.columbia.edu" uname="orestes.nevis.columbia.edu"
type="normal">
<instance_attributes id="nodes-orestes.nevis.columbia.edu">
<nvpair id="nodes-orestes.nevis.columbia.edu-standby" name="standby"
value="off"/>
</instance_attributes>
</node>
<node id="hypatia.nevis.columbia.edu" uname="hypatia.nevis.columbia.edu"
type="normal">
<instance_attributes id="nodes-hypatia.nevis.columbia.edu">
<nvpair id="nodes-hypatia.nevis.columbia.edu-standby" name="standby"
value="off"/>
</instance_attributes>
</node>
</nodes>

For both nodes, the attribute "standby" is set to "off", which evidently means
it's not in standby mode, so it's online. According to a web page I found at
home last night, but can't find at work today, "standby:off" gets added to a
node's attributes if it was ever marked "UNCLEAN (offline)" and brought online
again; this has indeed happened with both my nodes.

To confirm, the first few lines from running "crm status" are:

============
Last updated: Fri Jan 6 11:29:23 2012
Stack: openais
Current DC: hypatia.nevis.columbia.edu - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
25 Resources configured.
============

Online: [ orestes.nevis.columbia.edu hypatia.nevis.columbia.edu ]

Master/Slave Set: Admin
Masters: [ hypatia.nevis.columbia.edu ]
Slaves: [ orestes.nevis.columbia.edu ]


Getting back to my question, I hunted through my logs and saw that during time
of "one-node; no promotion" crm reported the above status as:

Master/Slave Set: Admin
Slaves: [ hypatia.nevis.columbia.edu ]
Stopped: [ AdminDrbd:1 ]

... and it simply stayed like that. Since all the other cluster resources depend
on Admin:promote, nothing else would happen. The relevant drbd messages in the
log file appear to be:

Dec 30 19:24:26 hypatia kernel: events: mcg drbd: 1
Dec 30 19:24:26 hypatia kernel: drbd: initialized. Version: 8.4.1
(api:1/proto:86-100)
Dec 30 19:24:26 hypatia kernel: drbd: GIT-hash:
91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] hypatia, 2011-12-21 13:42:51
Dec 30 19:24:26 hypatia kernel: drbd: registered as block device major 147
Dec 30 19:24:26 hypatia lrmd: [5800]: info: RA output: (AdminDrbd:0:start:stdout)
Dec 30 19:24:26 hypatia kernel: igb: eth1 NIC Link is Down
Dec 30 19:24:27 hypatia kernel: d-con admin: Starting worker thread (from
drbdsetup [7213])
Dec 30 19:24:27 hypatia kernel: block drbd1: disk( Diskless -> Attaching )
Dec 30 19:24:27 hypatia kernel: d-con admin: Method to ensure write ordering:
barrier
Dec 30 19:24:27 hypatia kernel: block drbd1: max BIO size = 130560
Dec 30 19:24:27 hypatia kernel: block drbd1: drbd_bm_resize called with capacity
== 1953460176
Dec 30 19:24:27 hypatia kernel: block drbd1: resync bitmap: bits=244182522
words=3815352 pages=7452
Dec 30 19:24:27 hypatia kernel: block drbd1: size = 931 GB (976730088 KB)
Dec 30 19:24:27 hypatia lrmd: [5800]: info: RA output:
(AdminDrbd:0:start:stderr) Marked additional 1028 MB as out-of-sync based on AL.
Dec 30 19:24:27 hypatia crmd: [5803]: info: process_lrm_event: LRM operation
WorkDrbd:0_start_0 (call=49, rc=0, cib-update=56, confirmed=true) ok
Dec 30 19:24:28 hypatia kernel: block drbd1: bitmap READ of 7452 pages took 131
jiffies
Dec 30 19:24:28 hypatia kernel: block drbd1: recounting of set bits took
additional 21 jiffies
Dec 30 19:24:28 hypatia kernel: block drbd1: 1028 MB (263168 bits) marked
out-of-sync by on disk bit-map.
Dec 30 19:24:28 hypatia kernel: block drbd1: disk( Attaching -> Consistent )
Dec 30 19:24:28 hypatia kernel: block drbd1: attached to UUIDs
A82875A514F576EB:0000000000000000:5283A3879DE4DED1:5282A3879DE4DED1
Dec 30 19:24:28 hypatia lrmd: [5800]: info: RA output: (AdminDrbd:0:start:stdout)
Dec 30 19:24:28 hypatia kernel: d-con admin: conn( StandAlone -> Unconnected )
Dec 30 19:24:28 hypatia kernel: d-con admin: Starting receiver thread (from
drbd_w_admin [7214])
Dec 30 19:24:28 hypatia kernel: d-con admin: receiver (re)started
Dec 30 19:24:28 hypatia kernel: d-con admin: conn( Unconnected -> WFConnection )
Dec 30 19:24:28 hypatia lrmd: [5800]: info: RA output: (AdminDrbd:0:start:stdout)


I noticed today a post on Linux-HA that appears to be the same problem as mine:

<http://www.gossamer-threads.com/lists/drbd/users/19943>

Unfortunately no resolution to the problem was posted then.

Any thoughts?
--
Bill Seligman | Phone: (914) 591-2823
Nevis Labs, Columbia Univ | mailto://seligman [at] nevis
PO Box 137 |
Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/
Attachments: smime.p7s (4.39 KB)


seligman at nevis

Jan 11, 2012, 11:34 AM

Post #9 of 10 (1451 views)
Permalink
Re: DRBD+Pacemaker: Won't promote with only one node [In reply to]

I didn't get a response up-thread, so please forgive my sin of top-posting and
ask a more restricted question:

I've re-posted a section of my log file below. What I suspect happened is this:

- I have two servers running DRBD+Pacemaker, hypatia (master) and orestes (slave).
- Master crashes due to a power outage. It's STONITHed.
- Slave becomes master, then it too crashes due to power outage.
- I bring up both systems, but things are confused and there are multiple
STONITHs, even a case when both systems STONITH each other.
- I go more slowly and just bring up hypatia.
- hypatia starts Pacemaker which starts up DRBD.
- What I think the log says is that hypatia DRBD sees that some of its sectors
are out-of-sync. It waits for the slave to back to sync them before it will
allow the resource to be promoted to master.

Is this scenario consistent with these log entries, or am I way off course?

Dec 30 19:24:26 hypatia kernel: events: mcg drbd: 1
Dec 30 19:24:26 hypatia kernel: drbd: initialized. Version: 8.4.1
(api:1/proto:86-100)
Dec 30 19:24:26 hypatia kernel: drbd: GIT-hash:
91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
root [at] hypatia, 2011-12-21 13:42:51
Dec 30 19:24:26 hypatia kernel: drbd: registered as block device major 147
Dec 30 19:24:26 hypatia lrmd: [5800]: info: RA output: (AdminDrbd:0:start:stdout)
Dec 30 19:24:26 hypatia kernel: igb: eth1 NIC Link is Down
Dec 30 19:24:27 hypatia kernel: d-con admin: Starting worker thread (from
drbdsetup [7213])
Dec 30 19:24:27 hypatia kernel: block drbd1: disk( Diskless -> Attaching )
Dec 30 19:24:27 hypatia kernel: d-con admin: Method to ensure write ordering:
barrier
Dec 30 19:24:27 hypatia kernel: block drbd1: max BIO size = 130560
Dec 30 19:24:27 hypatia kernel: block drbd1: drbd_bm_resize called with capacity
== 1953460176
Dec 30 19:24:27 hypatia kernel: block drbd1: resync bitmap: bits=244182522
words=3815352 pages=7452
Dec 30 19:24:27 hypatia kernel: block drbd1: size = 931 GB (976730088 KB)
Dec 30 19:24:27 hypatia lrmd: [5800]: info: RA output:
(AdminDrbd:0:start:stderr) Marked additional 1028 MB as out-of-sync based on AL.
Dec 30 19:24:27 hypatia crmd: [5803]: info: process_lrm_event: LRM operation
WorkDrbd:0_start_0 (call=49, rc=0, cib-update=56, confirmed=true) ok
Dec 30 19:24:28 hypatia kernel: block drbd1: bitmap READ of 7452 pages took 131
jiffies
Dec 30 19:24:28 hypatia kernel: block drbd1: recounting of set bits took
additional 21 jiffies
Dec 30 19:24:28 hypatia kernel: block drbd1: 1028 MB (263168 bits) marked
out-of-sync by on disk bit-map.
Dec 30 19:24:28 hypatia kernel: block drbd1: disk( Attaching -> Consistent )
Dec 30 19:24:28 hypatia kernel: block drbd1: attached to UUIDs
A82875A514F576EB:0000000000000000:5283A3879DE4DED1:5282A3879DE4DED1
Dec 30 19:24:28 hypatia lrmd: [5800]: info: RA output: (AdminDrbd:0:start:stdout)
Dec 30 19:24:28 hypatia kernel: d-con admin: conn( StandAlone -> Unconnected )
Dec 30 19:24:28 hypatia kernel: d-con admin: Starting receiver thread (from
drbd_w_admin [7214])
Dec 30 19:24:28 hypatia kernel: d-con admin: receiver (re)started
Dec 30 19:24:28 hypatia kernel: d-con admin: conn( Unconnected -> WFConnection )
Dec 30 19:24:28 hypatia lrmd: [5800]: info: RA output: (AdminDrbd:0:start:stdout)


On 1/6/12 11:54 AM, William Seligman wrote:
>> Message: 1
>> Date: Thu, 5 Jan 2012 23:01:24 +0100
>> From: Florian Haas <florian [at] hastexo>
>> Subject: Re: [DRBD-user] DRBD+Pacemaker: Won't promote with only one
>> node
>> To: drbd-user [at] lists
>> Message-ID:
>> <CAPUexz9jgpq0V49eZmwDH3cThrGEJ7VopXCh1_27yjX-6JC+fQ [at] mail>
>> Content-Type: text/plain; charset=UTF-8
>>
>> On Thu, Jan 5, 2012 at 6:36 PM, William Seligman
>> <seligman [at] nevis> wrote:
>>> Sure. I didn't do this before, since the configuration is complex. I also don't
>>> know which would be more comprehensible, so I've attached both cib.xml and the
>>> result of "crm configure show". I should mention that I'm a lazy bum, so I use
>>> crm-gui to configure corosync; that's why these files look more baroque than usual.
>
>> In the CIB you posted, both nodes are in the "UNCLEAN (offline)"
>> state. In that state, nothing gets promoted, nothing gets started. Are
>> you sure you posted the right CIB dump?
>
> That caused me to panic! I rushed to check that my cluster was running.
> It's OK; an occasional panic is good once in a while.
>
> I assume you're talking about the following lines:
>
> <nodes>
> <node id="orestes.nevis.columbia.edu" uname="orestes.nevis.columbia.edu"
> type="normal">
> <instance_attributes id="nodes-orestes.nevis.columbia.edu">
> <nvpair id="nodes-orestes.nevis.columbia.edu-standby" name="standby"
> value="off"/>
> </instance_attributes>
> </node>
> <node id="hypatia.nevis.columbia.edu" uname="hypatia.nevis.columbia.edu"
> type="normal">
> <instance_attributes id="nodes-hypatia.nevis.columbia.edu">
> <nvpair id="nodes-hypatia.nevis.columbia.edu-standby" name="standby"
> value="off"/>
> </instance_attributes>
> </node>
> </nodes>
>
> For both nodes, the attribute "standby" is set to "off", which evidently means
> it's not in standby mode, so it's online. According to a web page I found at
> home last night, but can't find at work today, "standby:off" gets added to a
> node's attributes if it was ever marked "UNCLEAN (offline)" and brought online
> again; this has indeed happened with both my nodes.
>
> To confirm, the first few lines from running "crm status" are:
>
> ============
> Last updated: Fri Jan 6 11:29:23 2012
> Stack: openais
> Current DC: hypatia.nevis.columbia.edu - partition with quorum
> Version: 1.0.12-unknown
> 2 Nodes configured, 2 expected votes
> 25 Resources configured.
> ============
>
> Online: [ orestes.nevis.columbia.edu hypatia.nevis.columbia.edu ]
>
> Master/Slave Set: Admin
> Masters: [ hypatia.nevis.columbia.edu ]
> Slaves: [ orestes.nevis.columbia.edu ]
>
>
> Getting back to my question, I hunted through my logs and saw that during time
> of "one-node; no promotion" crm reported the above status as:
>
> Master/Slave Set: Admin
> Slaves: [ hypatia.nevis.columbia.edu ]
> Stopped: [ AdminDrbd:1 ]
>
> ... and it simply stayed like that. Since all the other cluster resources depend
> on Admin:promote, nothing else would happen. The relevant drbd messages in the
> log file appear to be:
>
> Dec 30 19:24:26 hypatia kernel: events: mcg drbd: 1
> Dec 30 19:24:26 hypatia kernel: drbd: initialized. Version: 8.4.1
> (api:1/proto:86-100)
> Dec 30 19:24:26 hypatia kernel: drbd: GIT-hash:
> 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by
> root [at] hypatia, 2011-12-21 13:42:51
> Dec 30 19:24:26 hypatia kernel: drbd: registered as block device major 147
> Dec 30 19:24:26 hypatia lrmd: [5800]: info: RA output: (AdminDrbd:0:start:stdout)
> Dec 30 19:24:26 hypatia kernel: igb: eth1 NIC Link is Down
> Dec 30 19:24:27 hypatia kernel: d-con admin: Starting worker thread (from
> drbdsetup [7213])
> Dec 30 19:24:27 hypatia kernel: block drbd1: disk( Diskless -> Attaching )
> Dec 30 19:24:27 hypatia kernel: d-con admin: Method to ensure write ordering:
> barrier
> Dec 30 19:24:27 hypatia kernel: block drbd1: max BIO size = 130560
> Dec 30 19:24:27 hypatia kernel: block drbd1: drbd_bm_resize called with capacity
> == 1953460176
> Dec 30 19:24:27 hypatia kernel: block drbd1: resync bitmap: bits=244182522
> words=3815352 pages=7452
> Dec 30 19:24:27 hypatia kernel: block drbd1: size = 931 GB (976730088 KB)
> Dec 30 19:24:27 hypatia lrmd: [5800]: info: RA output:
> (AdminDrbd:0:start:stderr) Marked additional 1028 MB as out-of-sync based on AL.
> Dec 30 19:24:27 hypatia crmd: [5803]: info: process_lrm_event: LRM operation
> WorkDrbd:0_start_0 (call=49, rc=0, cib-update=56, confirmed=true) ok
> Dec 30 19:24:28 hypatia kernel: block drbd1: bitmap READ of 7452 pages took 131
> jiffies
> Dec 30 19:24:28 hypatia kernel: block drbd1: recounting of set bits took
> additional 21 jiffies
> Dec 30 19:24:28 hypatia kernel: block drbd1: 1028 MB (263168 bits) marked
> out-of-sync by on disk bit-map.
> Dec 30 19:24:28 hypatia kernel: block drbd1: disk( Attaching -> Consistent )
> Dec 30 19:24:28 hypatia kernel: block drbd1: attached to UUIDs
> A82875A514F576EB:0000000000000000:5283A3879DE4DED1:5282A3879DE4DED1
> Dec 30 19:24:28 hypatia lrmd: [5800]: info: RA output: (AdminDrbd:0:start:stdout)
> Dec 30 19:24:28 hypatia kernel: d-con admin: conn( StandAlone -> Unconnected )
> Dec 30 19:24:28 hypatia kernel: d-con admin: Starting receiver thread (from
> drbd_w_admin [7214])
> Dec 30 19:24:28 hypatia kernel: d-con admin: receiver (re)started
> Dec 30 19:24:28 hypatia kernel: d-con admin: conn( Unconnected -> WFConnection )
> Dec 30 19:24:28 hypatia lrmd: [5800]: info: RA output: (AdminDrbd:0:start:stdout)
>
>
> I noticed today a post on Linux-HA that appears to be the same problem as mine:
>
> <http://www.gossamer-threads.com/lists/drbd/users/19943>
>
> Unfortunately no resolution to the problem was posted then.
>
> Any thoughts?

--
Bill Seligman | Phone: (914) 591-2823
Nevis Labs, Columbia Univ | mailto://seligman [at] nevis
PO Box 137 |
Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/
Attachments: smime.p7s (4.39 KB)


ff at mpexnet

Jan 12, 2012, 12:53 AM

Post #10 of 10 (1437 views)
Permalink
Re: DRBD+Pacemaker: Won't promote with only one node [In reply to]

Hi,

On 01/11/2012 08:34 PM, William Seligman wrote:
> Marked additional 1028 MB as out-of-sync based on AL.

uhuh, yes, your local node would like to sync some data back from its
peer because it had marked about a gigabyte of data as "hot" in the
activity log.

> Dec 30 19:24:28 hypatia kernel: block drbd1: disk( Attaching -> Consistent )

Therefor, you don't have access to up-to-date data yet.

I suppose that by now you've split-brained badly. Question to the room:
Is Pacemaker handling this situation poorly when STONITH is configured?

What you probably want to do is to put pacemaker into maintenance mode,
bring up the peer node and resolve your split-brain or whatever is
causing your trouble. Disable maintenance mode once you're all set.

HTH,
Felix
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

DRBD users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.