andrew at beekhof
Apr 12, 2012, 6:47 PM
Views: 651
Permalink
|
On Thu, Apr 12, 2012 at 5:26 PM, Lars Ellenberg
<lars.ellenberg [at] linbit> wrote:
> On Wed, Apr 11, 2012 at 08:22:59AM +1000, Andrew Beekhof wrote:
>> It looks like the drbd RA is calling crm_master during the monitor action.
>> That wouldn't seem like a good idea as the value isn't counted until
>> the resource is started and if the transition is interrupted (as it is
>> here) then the PE won't try to promote it (because the value didn't
>> change).
>
> I did not get the last part.
> Why would it not be promoted,
> even though it has positive master score?
Because we don't know that we need to run the PE again - because the
only changes in the PE were things we expected.
See:
https://github.com/beekhof/pacemaker/commit/65f1a22a4b66581159d8b747dbd49fa5e2ef34e1
This "only" becomes and issue when the transition is interrupted
between the non-recurring monitor and the start, which I guess was
rare enough that we hadn't noticed it for 4 years :-(
>
>> Has the drbd RA always done this?
>
> Yes.
>
> When else should we call crm_master?
I guess the only situation you shouldn't is during a non-recurring
monitor if you're about to return 7.
Which I'll concede isn't exactly obvious.
>
> Preference changes: we may lose a local disk,
> we may have been outdated or inconsistent,
> then sync up, etc.
>
>> On Sat, Mar 31, 2012 at 2:56 AM, William Seligman
>> <seligman [at] nevis> wrote:
>> > On 3/30/12 1:13 AM, Andrew Beekhof wrote:
>> >> On Fri, Mar 30, 2012 at 2:57 AM, William Seligman
>> >> <seligman [at] nevis> wrote:
>> >>> On 3/29/12 3:19 AM, Andrew Beekhof wrote:
>> >>>> On Wed, Mar 28, 2012 at 9:12 AM, William Seligman
>> >>>> <seligman [at] nevis> wrote:
>> >>>>> The basics: Dual-primary cman+pacemaker+drbd cluster running on RHEL6.2; spec
>> >>>>> files and versions below.
>> >>>>>
>> >>>>> Problem: If I restart both nodes at the same time, or even just start pacemaker
>> >>>>> on both nodes at the same time, the drbd ms resource starts, but both nodes stay
>> >>>>> in slave mode. They'll both stay in slave mode until one of the following occurs:
>> >>>>>
>> >>>>> - I manually type "crm resource cleanup <ms-resource-name>"
>> >>>>>
>> >>>>> - 15 minutes elapse. Then the "PEngine Recheck Timer" is fired, and the ms
>> >>>>> resources are promoted.
>> >>>>>
>> >>>>> The key resource definitions:
>> >>>>>
>> >>>>> primitive AdminDrbd ocf:linbit:drbd \
>> >>>>> � � � �params drbd_resource="admin" \
>> >>>>> � � � �op monitor interval="59s" role="Master" timeout="30s" \
>> >>>>> � � � �op monitor interval="60s" role="Slave" timeout="30s" \
>> >>>>> � � � �op stop interval="0" timeout="100" \
>> >>>>> � � � �op start interval="0" timeout="240" \
>> >>>>> � � � �meta target-role="Master"
>> >>>>> ms AdminClone AdminDrbd \
>> >>>>> � � � �meta master-max="2" master-node-max="1" clone-max="2" \
>> >>>>> � � � �clone-node-max="1" notify="true" interleave="true"
>> >>>>> # The lengthy definition of "FilesystemGroup" is in the crm pastebin below
>> >>>>> clone FilesystemClone FilesystemGroup \
>> >>>>> � � � �meta interleave="true" target-role="Started"
>> >>>>> colocation Filesystem_With_Admin inf: FilesystemClone AdminClone:Master
>> >>>>> order Admin_Before_Filesystem inf: AdminClone:promote FilesystemClone:start
>> >>>>>
>> >>>>> Note that I stuck in "target-role" options to try to solve the problem; no effect.
>> >>>>>
>> >>>>> When I look in /var/log/messages, I see no error messages or indications why the
>> >>>>> promotion should be delayed. The 'admin' drbd resource is reported as UpToDate
>> >>>>> on both nodes. There are no error messages when I force the issue with:
>> >>>>>
>> >>>>> crm resource cleanup AdminClone
>> >>>>>
>> >>>>> It's as if pacemaker, at start, needs some kind of "kick" after the drbd
>> >>>>> resource is ready to be promoted.
>> >>>>>
>> >>>>> This is not just an abstract case for me. At my site, it's not uncommon for
>> >>>>> there to be lengthy power outages that will bring down the cluster. Both systems
>> >>>>> will come up when power is restored, and I need for cluster services to be
>> >>>>> available shortly afterward, not 15 minutes later.
>> >>>>>
>> >>>>> Any ideas?
>> >>>>
>> >>>> Not without any logs
>> >>>
>> >>> Sure! Here's an extract from the log: <http://pastebin.com/L1ZnsQ0R>
>> >>>
>> >>> Before you click on the link (it's a big wall of text),
>> >>
>> >> I'm used to trawling the logs. Grep is a wonderful thing :-)
>> >>
>> >> At this stage it is apparent that I need to see
>> >> /var/lib/pengine/pe-input-4.bz2 from hypatia-corosync.
>> >> Do you have this file still?
>> >
>> > No, so I re-ran the test. Here's the log extract from the test I did today
>> > <http://pastebin.com/6QYH2jkf>.
>> >
>> > Based on what you asked for from the previous extract, I think what you want
>> > from this test is pe-input-5. Just to play it safe, I copied and bunzip2'ed all
>> > three pe-input files mentioned in the log messages:
>> >
>> > pe-input-4: <http://pastebin.com/Txx50BJp>
>> > pe-input-5: <http://pastebin.com/zzppL6DF>
>> > pe-input-6: <http://pastebin.com/1dRgURK5>
>> >
>> > I pray to the gods of Grep that you find a clue in all of that!
>> >
>> >>> here are what I think
>> >>> are the landmarks:
>> >>>
>> >>> - The extract starts just after the node boots, at the start of syslog at time
>> >>> 10:49:21.
>> >>> - I've highlighted when pacemakerd starts, at 10:49:46.
>> >>> - I've highlighted when drbd reports that the 'admin' resource is UpToDate, at
>> >>> 10:50:10.
>> >>> - One last highlight: When pacemaker finally promotes the drbd resource to
>> >>> Primary on both nodes, at 11:05:11.
>> >>>
>> >>>> Details:
>> >>>>>
>> >>>>> # rpm -q kernel cman pacemaker drbd
>> >>>>> kernel-2.6.32-220.4.1.el6.x86_64
>> >>>>> cman-3.0.12.1-23.el6.x86_64
>> >>>>> pacemaker-1.1.6-3.el6.x86_64
>> >>>>> drbd-8.4.1-1.el6.x86_64
>> >>>>>
>> >>>>> Output of crm_mon after two-node reboot or pacemaker restart:
>> >>>>> <http://pastebin.com/jzrpCk3i>
>> >>>>> cluster.conf: <http://pastebin.com/sJw4KBws>
>> >>>>> "crm configure show": <http://pastebin.com/MgYCQ2JH>
>> >>>>> "drbdadm dump all": <http://pastebin.com/NrY6bskk>
>> >
>> > --
>> > Bill Seligman | Phone: (914) 591-2823
>> > Nevis Labs, Columbia Univ | mailto://seligman [at] nevis
>> > PO Box 137 |
>> > Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/
>> >
>> >
>> > _______________________________________________
>> > Linux-HA mailing list
>> > Linux-HA [at] lists
>> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> > See also: http://linux-ha.org/ReportingProblems
>
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
|
