Mailing List Archive: Linux-HA: Pacemaker

Call cib_query failed (-41): Remote node did not respond

Index | Next | Previous | View Threaded

brian at interlinx

Jun 26, 2012, 1:45 PM

Post #1 of 20 (960 views)
Permalink

Call cib_query failed (-41): Remote node did not respond

So, I have an 18 node cluster here (so a small haystack, indeed, but
still a haystack in which to try to find a needle) where a certain
set of (yet unknown, figuring that out is part of this process)
operations are pooching pacemaker. The symptom is that on one or
more nodes I get the following kinds of errors:

# cibadmin -Q
Call cib_query failed (-41): Remote node did not respond

along with similar things in the log:

Jun 26 19:51:38 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 7 failed: (rc=-41) Remote node did not respond
Jun 26 19:51:38 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 8 failed: (rc=-41) Remote node did not respond
Jun 26 19:51:38 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 9 failed: (rc=-41) Remote node did not respond
Jun 26 19:51:38 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 10 failed: (rc=-41) Remote node did not respond
Jun 26 19:51:39 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 11 failed: (rc=-41) Remote node did not respond

Clearly some node in the cluster has a problem, but nothing in any
of these messages is helping me figure out which one it is.

Any hints on how I figure out which node this "iu-18" node is having
problems communicating with?

Cheers,
b.

Attachments:

signature.asc (0.26 KB)

andrew at beekhof

Jun 26, 2012, 6:54 PM

Post #2 of 20 (938 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On Wed, Jun 27, 2012 at 6:45 AM, Brian J. Murrell <brian [at] interlinx> wrote:
> So, I have an 18 node cluster here (so a small haystack, indeed, but
> still a haystack in which to try to find a needle) where a certain
> set of (yet unknown, figuring that out is part of this process)
> operations are pooching pacemaker. The symptom is that on one or
> more nodes I get the following kinds of errors:
>
> # cibadmin -Q
> Call cib_query failed (-41): Remote node did not respond
>
> along with similar things in the log:
>
> Jun 26 19:51:38 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 7 failed: (rc=-41) Remote node did not respond
> Jun 26 19:51:38 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 8 failed: (rc=-41) Remote node did not respond
> Jun 26 19:51:38 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 9 failed: (rc=-41) Remote node did not respond
> Jun 26 19:51:38 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 10 failed: (rc=-41) Remote node did not respond
> Jun 26 19:51:39 iu-18 crmd: [19119]: WARN: cib_rsc_callback: Resource update 11 failed: (rc=-41) Remote node did not respond
>
> Clearly some node in the cluster has a problem, but nothing in any
> of these messages is helping me figure out which one it is.
>
> Any hints on how I figure out which node this "iu-18" node is having
> problems communicating with?

The DC, possibly you didn't have one at that moment in time.
Were there (m)any membership events occurring at the time?

_______________________________________________
Pacemaker mailing list: Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

andrew at beekhof

Jun 26, 2012, 6:54 PM

Post #3 of 20 (937 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

brian at interlinx

Jun 27, 2012, 6:30 AM

Post #4 of 20 (935 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On 12-06-26 09:54 PM, Andrew Beekhof wrote:
>
> The DC, possibly you didn't have one at that moment in time.

It was the DC in fact. I restarted corosync on that node and the
timeouts went away. But note I "re"started, not started. It was
running at the time, just not properly, apparently.

> Were there (m)any membership events occurring at the time?

I'm not sure.

I do seem to be able to reproduce this situation though with some
software I have that's driving pacemaker configuration building.

I essentially have 34 resources across 17 nodes that I need to populate
pacemaker with, complete with location constraints. This populating is
done with a pair of cibadmin commands, one for the resource and one for
the constraint. These pairs of commands are being run for each resource
on the nodes on which they will run.

So, that's 17 pairs of cibadmin commands being run, one pair on each
node, concurrently -- so yes, lots of thrashing of the CIB. Is the CIB
and/or cibadmin not up to this kind of thrashing?

Typically while this is happening some number of cibadmin commands will
start failing with:

Call cib_create failed (-41): Remote node did not respond

and then calls to (say) "cibadmin -Q" on every node except the DC will
start failing with:

Call cib_query failed (-41): Remote node did not respond

After restarting corosync on the DC, (most if not all of) the non-DC
nodes are now able to return from "cibadmin -Q" but they have differing
CIB contents. That state doesn't seem to last long and all nodes except
the (typically new/different) DC node again suffer "Remote node did not
respond". A restart of that new DC again yields some/most of the nodes
able to complete queries again, bug again, with differing CIB content.

I am using corosync-1.4.1-4.el6_2.3 and pacemaker-1.1.6-3.el6 on these
nodes.

Any ideas? Am I really pushing the CIB too hard with all of the
concurrent modifications?

b.

Attachments:

signature.asc (0.26 KB)

andrew at beekhof

Jun 27, 2012, 8:29 PM

Post #5 of 20 (928 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On Wed, Jun 27, 2012 at 11:30 PM, Brian J. Murrell
<brian [at] interlinx> wrote:
> On 12-06-26 09:54 PM, Andrew Beekhof wrote:
>>
>> The DC, possibly you didn't have one at that moment in time.
>
> It was the DC in fact. I restarted corosync on that node and the
> timeouts went away. But note I "re"started, not started. It was
> running at the time, just not properly, apparently.
>
>> Were there (m)any membership events occurring at the time?
>
> I'm not sure.
>
> I do seem to be able to reproduce this situation though with some
> software I have that's driving pacemaker configuration building.
>
> I essentially have 34 resources across 17 nodes that I need to populate
> pacemaker with, complete with location constraints. This populating is
> done with a pair of cibadmin commands, one for the resource and one for
> the constraint. These pairs of commands are being run for each resource
> on the nodes on which they will run.
>
> So, that's 17 pairs of cibadmin commands being run, one pair on each
> node, concurrently -- so yes, lots of thrashing of the CIB. Is the CIB
> and/or cibadmin not up to this kind of thrashing?

As with any interesting question, "it depends".

Newer versions have become more efficient over time, so an upgrade may help.
Even something as trivial as switching our logging library made a
significant difference.
For 1.1.8 we've also switched the IPC library which should bring
another performance boost.

If the services you're adding are clones, that can also have a big
impact as the number of probe operations is clones * clone-max *
nodes.
Thats a lot of updates hitting the cib when the cluster first starts up.

To mitigate the thrashing, try setting the batch-limit parameter in
the cib (man pengine).

>
> Typically while this is happening some number of cibadmin commands will
> start failing with:
>
> Call cib_create failed (-41): Remote node did not respond
>
> and then calls to (say) "cibadmin -Q" on every node except the DC will
> start failing with:
>
> Call cib_query failed (-41): Remote node did not respond
>
> After restarting corosync on the DC, (most if not all of) the non-DC
> nodes are now able to return from "cibadmin -Q" but they have differing
> CIB contents. That state doesn't seem to last long and all nodes except
> the (typically new/different) DC node again suffer "Remote node did not
> respond". A restart of that new DC again yields some/most of the nodes
> able to complete queries again, bug again, with differing CIB content.

Really doesn't sound good.
Could you check CPU usage of the various cluster processes with top
while this is occurring?
Otherwise perhaps the traffic is making corosync twitchy and you're hitting:
http://bugzilla.redhat.com/show_bug.cgi?id=820821

> I am using corosync-1.4.1-4.el6_2.3 and pacemaker-1.1.6-3.el6 on these
> nodes.
>
> Any ideas? Am I really pushing the CIB too hard with all of the
> concurrent modifications?
>
> b.
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker [at] oss
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

_______________________________________________
Pacemaker mailing list: Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

andrew at beekhof

Jun 27, 2012, 8:29 PM

Post #6 of 20 (924 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

andrew at beekhof

Jun 27, 2012, 8:30 PM

Post #7 of 20 (931 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On Thu, Jun 28, 2012 at 1:29 PM, Andrew Beekhof <andrew [at] beekhof> wrote:
> On Wed, Jun 27, 2012 at 11:30 PM, Brian J. Murrell
> <brian [at] interlinx> wrote:
>> On 12-06-26 09:54 PM, Andrew Beekhof wrote:
>>>
>>> The DC, possibly you didn't have one at that moment in time.
>>
>> It was the DC in fact. I restarted corosync on that node and the
>> timeouts went away. But note I "re"started, not started. It was
>> running at the time, just not properly, apparently.
>>
>>> Were there (m)any membership events occurring at the time?
>>
>> I'm not sure.
>>
>> I do seem to be able to reproduce this situation though with some
>> software I have that's driving pacemaker configuration building.
>>
>> I essentially have 34 resources across 17 nodes that I need to populate
>> pacemaker with, complete with location constraints. This populating is
>> done with a pair of cibadmin commands, one for the resource and one for
>> the constraint. These pairs of commands are being run for each resource
>> on the nodes on which they will run.
>>
>> So, that's 17 pairs of cibadmin commands being run, one pair on each
>> node, concurrently -- so yes, lots of thrashing of the CIB. Is the CIB
>> and/or cibadmin not up to this kind of thrashing?
>
> As with any interesting question, "it depends".
>
> Newer versions have become more efficient over time, so an upgrade may help.
> Even something as trivial as switching our logging library made a
> significant difference.
> For 1.1.8 we've also switched the IPC library which should bring
> another performance boost.
>
> If the services you're adding are clones, that can also have a big
> impact as the number of probe operations is clones * clone-max *
> nodes.
> Thats a lot of updates hitting the cib when the cluster first starts up.
>
> To mitigate the thrashing, try setting the batch-limit parameter in
> the cib (man pengine).
>
>>
>> Typically while this is happening some number of cibadmin commands will
>> start failing with:
>>
>> Call cib_create failed (-41): Remote node did not respond
>>
>> and then calls to (say) "cibadmin -Q" on every node except the DC will
>> start failing with:
>>
>> Call cib_query failed (-41): Remote node did not respond
>>
>> After restarting corosync on the DC, (most if not all of) the non-DC
>> nodes are now able to return from "cibadmin -Q" but they have differing
>> CIB contents. That state doesn't seem to last long and all nodes except
>> the (typically new/different) DC node again suffer "Remote node did not
>> respond". A restart of that new DC again yields some/most of the nodes
>> able to complete queries again, bug again, with differing CIB content.
>
> Really doesn't sound good.
> Could you check CPU usage of the various cluster processes with top
> while this is occurring?
> Otherwise perhaps the traffic is making corosync twitchy and you're hitting:
> http://bugzilla.redhat.com/show_bug.cgi?id=820821
>
>> I am using corosync-1.4.1-4.el6_2.3 and pacemaker-1.1.6-3.el6 on these
>> nodes.
>>
>> Any ideas? Am I really pushing the CIB too hard with all of the
>> concurrent modifications?

The updates from you aren't the problem. Its the number of resource
operations (that need to be stored in the CIB) that result from your
changes that might be causing the problem.

_______________________________________________
Pacemaker mailing list: Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

andrew at beekhof

Jun 27, 2012, 8:30 PM

Post #8 of 20 (924 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

brian at interlinx

Jul 3, 2012, 12:15 PM

Post #9 of 20 (890 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On 12-06-27 11:30 PM, Andrew Beekhof wrote:
>
> The updates from you aren't the problem. Its the number of resource
> operations (that need to be stored in the CIB) that result from your
> changes that might be causing the problem.

Just to follow this up for anyone currently following or anyone finding
this thread in the future...

It turns out that the problem is simply the size of the HA cluster that
I want to create. The details are in the bug I filed at
http://bugs.clusterlabs.org/show_bug.cgi?id=5076 but the short story is
that I can add the number of resources and constrains I want to add
(i.e. 32-34 of each, as previously described in this thread),
concurrently even, so long as there is not more than 4 nodes per
corosync/pacemaker cluster.

Even adding 4 passive nodes (I only tried 8 total of 8 nodes, but not
values between 4 and 8 so the tipping point might be somewhere in
between 4 and 8) -- nodes that do no CIB operations of their own made
pacemaker crumble.

So the summary seems to be that pacemaker cannot scale to more than a
handful of nodes, even when the nodes are big: 12 core Xeon nodes with
gobs of memory.

I can only guess that everybody is using pacemaker in "pair" (or not
much bigger) type configurations currently. Is that accurate?

Perhaps there is some tuning that can be done to scale somewhat, but
realistically, I am looking for pacemaker clusters in the tens, if not
into the hundreds of nodes. However, I really wonder if any amount of
tuning could be done to achieve clusters that large given the small
number of nodes supported with the default tuning values.

Thoughts?

b.

Attachments:

signature.asc (0.26 KB)

dvossel at redhat

Jul 3, 2012, 1:26 PM

Post #10 of 20 (892 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

----- Original Message -----
> From: "Brian J. Murrell" <brian [at] interlinx>
> To: pacemaker [at] clusterlabs
> Sent: Tuesday, July 3, 2012 2:15:09 PM
> Subject: Re: [Pacemaker] Call cib_query failed (-41): Remote node did not respond
>
> On 12-06-27 11:30 PM, Andrew Beekhof wrote:
> >
> > The updates from you aren't the problem. Its the number of
> > resource
> > operations (that need to be stored in the CIB) that result from
> > your
> > changes that might be causing the problem.
>
> Just to follow this up for anyone currently following or anyone
> finding
> this thread in the future...
>
> It turns out that the problem is simply the size of the HA cluster
> that
> I want to create. The details are in the bug I filed at
> http://bugs.clusterlabs.org/show_bug.cgi?id=5076 but the short story
> is
> that I can add the number of resources and constrains I want to add
> (i.e. 32-34 of each, as previously described in this thread),
> concurrently even, so long as there is not more than 4 nodes per
> corosync/pacemaker cluster.
>
> Even adding 4 passive nodes (I only tried 8 total of 8 nodes, but not
> values between 4 and 8 so the tipping point might be somewhere in
> between 4 and 8) -- nodes that do no CIB operations of their own made
> pacemaker crumble.
>
>
> So the summary seems to be that pacemaker cannot scale to more than a
> handful of nodes, even when the nodes are big: 12 core Xeon nodes
> with
> gobs of memory.

This is not a definite. Perhaps you are experiencing this given the pacemaker version you are running and the torture test you are running with all those parallel commands, but I wouldn't go as far as to say pacemaker cannot scale to more than a handful of nodes. It completely depends on the situation. 16 nodes with 32 resources might work... 3 nodes with 100 resources might not. There is a limit to how far deployments can scale, but it is not easy to quantify values that hold any real truth across all deployments. I'm sure you know this, I just wanted to be explicit about this so there is no confusion caused by people who may use your example as a concrete metric.

>
> I can only guess that everybody is using pacemaker in "pair" (or not
> much bigger) type configurations currently. Is that accurate?
>

From the deployments I've seen on the mailing list and bug reports, the most common clusters appear to be around the 2-6 node mark.

> Perhaps there is some tuning that can be done to scale somewhat, but
> realistically, I am looking for pacemaker clusters in the tens, if
> not
> into the hundreds of nodes. However, I really wonder if any amount

The messaging involved with keeping the all the local resource operations in the CIB synced across that many nodes is pretty insane. If you are set on using pacemaker, the best approach for scaling for your situation would probably be to try and figure out how to break nodes into smaller clusters that are easier to manage. I have not heard of a single deployment as large as you are thinking of.

-- Vossel

> of
> tuning could be done to achieve clusters that large given the small
> number of nodes supported with the default tuning values.
>
>
> Thoughts?
>
> b.
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker [at] oss
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

_______________________________________________
Pacemaker mailing list: Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

andrew at beekhof

Jul 3, 2012, 3:17 PM

Post #11 of 20 (888 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On Wed, Jul 4, 2012 at 5:15 AM, Brian J. Murrell <brian [at] interlinx> wrote:
> Thoughts?

Even adding passive nodes multiplies the number of probe operations
that need to be performed and loaded into the cib.
Did you try any of the settings I suggested?

_______________________________________________
Pacemaker mailing list: Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

andrew at beekhof

Jul 3, 2012, 3:17 PM

Post #12 of 20 (887 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

brian at interlinx

Jul 3, 2012, 3:36 PM

Post #13 of 20 (887 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On 12-07-03 06:17 PM, Andrew Beekhof wrote:
>
> Even adding passive nodes multiplies the number of probe operations
> that need to be performed and loaded into the cib.

So it seems. I just would have not thought they be such a load since
from a simplistic perspective, since they are not trying to update the
CIB, it seems they just need an update of it when the rest of the nodes
doing the updating are done. But I do admit that could be a simplistic
view.

> Did you try any of the settings I suggested?

The only setting I saw you suggest was "batch-limit" and at first glance
it did not seem clear to me which way to adjust this (up or down) and I
was running out of time for experimentation and just needed to get to
something that works.

So now that pressure is off and I have a bit of time to experiment, what
value would you suggest for that parameter given the 32 resource and
constraints I want to add on a 16 node cluster?

Cheers,
b.

Attachments:

signature.asc (0.26 KB)

brian at interlinx

Jul 3, 2012, 5:06 PM

Post #14 of 20 (885 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On 12-07-03 04:26 PM, David Vossel wrote:
>
> This is not a definite. Perhaps you are experiencing this given the pacemaker version you are running

Yes, that is absolutely possible and it certainly has been under
consideration throughout this process. I did also recognize however,
that I am running the latest stable (1.1.6) release and while I might be
able to experiment with with a development branch in the lab, I could
not use it in production. So while it would be an interesting
experiment, my primary goal had to be getting 1.1.6 to run stably.

> and the torture test you are running with all those parallel commands,

It is worth keeping in mind that all of those parallel commands are just
as parallel with the 4 node cluster as they are with the 8 (4 nodes
actively modifying the CIB + 4 completely idle nodes) and 16 node
clusters -- both of which failed.

Just because I reduced the number of nodes doesn't mean that I reduced
the parallelism any. The commands being run on each node are not
serialized and are all launched in parallel on the 4 node cluster as
much as they were with the 16 node cluster.

So strictly speaking, it doesn't seem that parallelism in the CIB
modifications are as much of a factor as simply the number of nodes in
the cluster, even when some (i.e. in the 8 node test I did) of the nodes
are entirely passive and not modifying the CIB at all.

> but I wouldn't go as far as to say pacemaker cannot scale to more than a handful of nodes.

I'd totally welcome being shown the error of my ways.

> I'm sure you know this, I just wanted to be explicit about this so there is no confusion caused by people who may use your example as a concrete metric.

But of course. In my experiments, it was clear that the cib process
could peak a single core on my 12 core Xeons with just 4 nodes in the
cluster at times.

Therefore it is also clear that some time down the road, assuming CPU is
the limiting factor here, it's quite easy to see how a faster CPU core,
or multithreading the cib would allow for better scaling, but my point
was simply at the current time, and again, assuming (since I don't know
for sure what the limiting factor really is) CPU is the limiting factor
here, somewhere between 4-8 nodes is the limit with more or less default
tunings.

> From the deployments I've seen on the mailing list and bug reports, the most common clusters appear to be around the 2-6 node mark.

Which seems consistent.

> The messaging involved with keeping the all the local resource operations in the CIB synced across that many nodes is pretty insane.

Indeed, and I most certainly had considered that. What really threw a
curve in that train of thought for me though was that even idle,
non-CIB-modifying nodes (i.e. turning a working 4 node cluster into a
non-working 8 node cluster by adding 4 nodes that do nothing with the
CIB) can tip a working configuration over into non-working.

I could most certainly see how the contention of 8 nodes all trying to
jam stuff into the CIB might be taxing with all of the locking that
needs to go on, etc, but for those 4 added idle nodes to add enough
complexity to make an working 4 node cluster not work is puzzling.
Puzzling enough (granted, to somebody who knows zilch about the
messaging that goes on with CIB operations) to make is smell more like a
bug than simple contention.

> If you are set on using pacemaker,

Well, I am not necessarily married to it. It did just seem like the
tool with the critical mass behind it. As sketchy as it might seem to
ask, (and I only am since you seem to be hinting that there might be a
better tool for the job) is there a tool more suited to the job?

> the best approach for scaling for your situation would probably be to try and figure out how to break nodes into smaller clusters that are easier to manage.

Indeed, that is what I ended up doing. Now my 16 node cluster is 4 4
node clusters. The problem with that though, is that when a node in a
cluster fails, it has only 3 other nodes to spread it's resources around
onto, and if 2 should fail, 2 nodes are trying to service twice their
normal load. The benefit of larger clusters is clear. in giving
pacemaker more nodes to evenly distribute resources to, impacting the
load of other the other nodes minimally when one or more nodes of the
cluster do fail.

> I have not heard of a single deployment as large as you are thinking of.

Heh. Not atypical of me to push the envelope I'm afraid. :-/

Cheers, and many thanks for your input. It is valuable to this discussion.

b.

Attachments:

signature.asc (0.26 KB)

andrew at beekhof

Jul 3, 2012, 11:12 PM

Post #15 of 20 (883 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On Wed, Jul 4, 2012 at 10:06 AM, Brian J. Murrell <brian [at] interlinx> wrote:
> On 12-07-03 04:26 PM, David Vossel wrote:
>>
>> This is not a definite. Perhaps you are experiencing this given the pacemaker version you are running
>
> Yes, that is absolutely possible and it certainly has been under
> consideration throughout this process. I did also recognize however,
> that I am running the latest stable (1.1.6) release and while I might be
> able to experiment with with a development branch in the lab, I could
> not use it in production. So while it would be an interesting
> experiment, my primary goal had to be getting 1.1.6 to run stably.
>
>> and the torture test you are running with all those parallel commands,
>
> It is worth keeping in mind that all of those parallel commands are just
> as parallel with the 4 node cluster as they are with the 8 (4 nodes
> actively modifying the CIB + 4 completely idle nodes) and 16 node
> clusters -- both of which failed.
>
> Just because I reduced the number of nodes doesn't mean that I reduced
> the parallelism any.

Yes. You did. You reduced the number of "check what state the
resource is on every node" probes.

> The commands being run on each node are not
> serialized and are all launched in parallel on the 4 node cluster as
> much as they were with the 16 node cluster.
>
> So strictly speaking, it doesn't seem that parallelism in the CIB
> modifications are as much of a factor as simply the number of nodes in
> the cluster, even when some (i.e. in the 8 node test I did) of the nodes
> are entirely passive and not modifying the CIB at all.

Now I'm getting annoyed.
I keep explaining this is not true yet you keep repeating the above assertion.

Please go back an re-read my previous answers (both here and
off-list). Properly. I will be happy to clarify anything that is
still unclear.

>
>> but I wouldn't go as far as to say pacemaker cannot scale to more than a handful of nodes.
>
> I'd totally welcome being shown the error of my ways.
>
>> I'm sure you know this, I just wanted to be explicit about this so there is no confusion caused by people who may use your example as a concrete metric.
>
> But of course. In my experiments, it was clear that the cib process
> could peak a single core on my 12 core Xeons with just 4 nodes in the
> cluster at times.
>
> Therefore it is also clear that some time down the road, assuming CPU is
> the limiting factor here, it's quite easy to see how a faster CPU core,
> or multithreading the cib would allow for better scaling, but my point
> was simply at the current time, and again, assuming (since I don't know
> for sure what the limiting factor really is) CPU is the limiting factor
> here, somewhere between 4-8 nodes is the limit with more or less default
> tunings.
>
>> From the deployments I've seen on the mailing list and bug reports, the most common clusters appear to be around the 2-6 node mark.
>
> Which seems consistent.
>
>> The messaging involved with keeping the all the local resource operations in the CIB synced across that many nodes is pretty insane.
>
> Indeed, and I most certainly had considered that. What really threw a
> curve in that train of thought for me though was that even idle,
> non-CIB-modifying nodes (i.e. turning a working 4 node cluster into a
> non-working 8 node cluster by adding 4 nodes that do nothing with the
> CIB) can tip a working configuration over into non-working.
>
> I could most certainly see how the contention of 8 nodes all trying to
> jam stuff into the CIB might be taxing with all of the locking that
> needs to go on, etc, but for those 4 added idle nodes to add enough
> complexity to make an working 4 node cluster not work is puzzling.
> Puzzling enough (granted, to somebody who knows zilch about the
> messaging that goes on with CIB operations) to make is smell more like a
> bug than simple contention.
>
>> If you are set on using pacemaker,
>
> Well, I am not necessarily married to it. It did just seem like the
> tool with the critical mass behind it. As sketchy as it might seem to
> ask, (and I only am since you seem to be hinting that there might be a
> better tool for the job) is there a tool more suited to the job?
>
>> the best approach for scaling for your situation would probably be to try and figure out how to break nodes into smaller clusters that are easier to manage.
>
> Indeed, that is what I ended up doing. Now my 16 node cluster is 4 4
> node clusters. The problem with that though, is that when a node in a
> cluster fails, it has only 3 other nodes to spread it's resources around
> onto, and if 2 should fail, 2 nodes are trying to service twice their
> normal load. The benefit of larger clusters is clear. in giving
> pacemaker more nodes to evenly distribute resources to, impacting the
> load of other the other nodes minimally when one or more nodes of the
> cluster do fail.
>
>> I have not heard of a single deployment as large as you are thinking of.
>
> Heh. Not atypical of me to push the envelope I'm afraid. :-/
>
> Cheers, and many thanks for your input. It is valuable to this discussion.
>
> b.
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker [at] oss
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

_______________________________________________
Pacemaker mailing list: Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

andrew at beekhof

Jul 3, 2012, 11:12 PM

Post #16 of 20 (881 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

andreas at hastexo

Jul 4, 2012, 1:27 AM

Post #17 of 20 (887 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On 07/04/2012 12:36 AM, Brian J. Murrell wrote:
> On 12-07-03 06:17 PM, Andrew Beekhof wrote:
>>
>> Even adding passive nodes multiplies the number of probe operations
>> that need to be performed and loaded into the cib.
>
> So it seems. I just would have not thought they be such a load since
> from a simplistic perspective, since they are not trying to update the
> CIB, it seems they just need an update of it when the rest of the nodes
> doing the updating are done. But I do admit that could be a simplistic
> view.
>
>> Did you try any of the settings I suggested?

beside increasing the batch limit to a higher value ... did you also
tune corosync totem timings? There are some parameters like token,
send_join, join and token_retransmits_before_loss_const that typically
need to be increased when a lot of cluster communication traffic has to
be handled.

Best Regards,
Andreas

--
Need help with Pacemaker?
http://www.hastexo.com/now

>
> The only setting I saw you suggest was "batch-limit" and at first glance
> it did not seem clear to me which way to adjust this (up or down) and I
> was running out of time for experimentation and just needed to get to
> something that works.
>
> So now that pressure is off and I have a bit of time to experiment, what
> value would you suggest for that parameter given the 32 resource and
> constraints I want to add on a 16 node cluster?
>
> Cheers,
> b.
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker [at] oss
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

Attachments:

signature.asc (0.22 KB)

brian at interlinx

Jul 4, 2012, 4:58 AM

Post #18 of 20 (883 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On 12-07-04 02:12 AM, Andrew Beekhof wrote:
> On Wed, Jul 4, 2012 at 10:06 AM, Brian J. Murrell <brian-SquOHqY54CVWr29BmMi2cA [at] public> wrote:
>>
>> Just because I reduced the number of nodes doesn't mean that I reduced
>> the parallelism any.
>
> Yes. You did. You reduced the number of "check what state the
> resource is on every node" probes.

Let me apologize as I was not clear. I meant I did not reduce the
amount of parallelism in *my* CIB modify operations. I was simply
clarifying that my operations on a single node are not serialized and
thus reducing the total number of nodes and increasing the number of
operations per node was not reducing the contention of those operations
by putting more operations into a serial queue per node.

> Now I'm getting annoyed.
> I keep explaining this is not true yet you keep repeating the above assertion.

Yes, I understand what you are saying. The ordering of the messages on
the list is unfortunate and some seem to have been crossing each other.
The message you were replying to, which is annoying you was composed
before your subsequent messages and was in response to somebody else on
the list.

My apologies for the confusion.

b.

Attachments:

signature.asc (0.26 KB)

brian at interlinx

Jul 4, 2012, 9:51 AM

Post #19 of 20 (888 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On 12-07-04 04:27 AM, Andreas Kurz wrote:
>
> beside increasing the batch limit to a higher value ... did you also
> tune corosync totem timings?

Not yet.

But a closer look at the logs reveals a bunch of these:

Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25046 spawned to record non-fatal assertion failure line 1594: rc == 0
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-4.lab.example.com"
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-4.lab.example.com.cib failed: cluster delivery failed (rc=-1)
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25048 spawned to record non-fatal assertion failure line 1594: rc == 0
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-6.lab.example.com"
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-6.lab.example.com.cib failed: cluster delivery failed (rc=-1)
Jun 28 14:56:56 node-2 abrt[25049]: not dumping repeating crash in '/usr/sbin/corosync'
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25050 spawned to record non-fatal assertion failure line 1594: rc == 0
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-10.lab.example.com
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-10.lab.example.com.cib failed: cluster delivery failed (rc=-1)
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25051 spawned to record non-fatal assertion failure line 1594: rc == 0
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-7.lab.example.com"
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-7.lab.example.com.cib failed: cluster delivery failed (rc=-1)
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25052 spawned to record non-fatal assertion failure line 1594: rc == 0
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-4.lab.example.com"
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-4.lab.example.com.cib failed: cluster delivery failed (rc=-1)
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25053 spawned to record non-fatal assertion failure line 1594: rc == 0
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-6.lab.example.com"
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-6.lab.example.com.cib failed: cluster delivery failed (rc=-1)
Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25054 spawned to record non-fatal assertion failure line 1594: rc == 0

Google could not seem to turn up anything about the assertion message.

I also saw these after setting the batch-limit to 1 and repeating my 8
node (4 active, 4 idle) experiment today.

But surely, it is easy to understand why pacemaker would have problems
if corosync is aborting on a failed assertion.

Any clues what this one is about? This is corosync-1.4.1-4.el6_2.3.x86_64.

Cheers,
b.

Attachments:

signature.asc (0.26 KB)

andrew at beekhof

Aug 14, 2012, 9:16 PM

Post #20 of 20 (691 views)
Permalink

Re: Call cib_query failed (-41): Remote node did not respond [In reply to]

On 05/07/2012, at 2:51 AM, Brian J. Murrell <brian [at] interlinx> wrote:

> On 12-07-04 04:27 AM, Andreas Kurz wrote:
>>
>> beside increasing the batch limit to a higher value ... did you also
>> tune corosync totem timings?
>
> Not yet.
>
> But a closer look at the logs reveals a bunch of these:
>
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25046 spawned to record non-fatal assertion failure line 1594: rc == 0
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-4.lab.example.com"
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-4.lab.example.com.cib failed: cluster delivery failed (rc=-1)
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25048 spawned to record non-fatal assertion failure line 1594: rc == 0
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-6.lab.example.com"
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-6.lab.example.com.cib failed: cluster delivery failed (rc=-1)
> Jun 28 14:56:56 node-2 abrt[25049]: not dumping repeating crash in '/usr/sbin/corosync'
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25050 spawned to record non-fatal assertion failure line 1594: rc == 0
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-10.lab.example.com
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-10.lab.example.com.cib failed: cluster delivery failed (rc=-1)
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25051 spawned to record non-fatal assertion failure line 1594: rc == 0
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-7.lab.example.com"
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-7.lab.example.com.cib failed: cluster delivery failed (rc=-1)
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25052 spawned to record non-fatal assertion failure line 1594: rc == 0
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-4.lab.example.com"
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-4.lab.example.com.cib failed: cluster delivery failed (rc=-1)
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25053 spawned to record non-fatal assertion failure line 1594: rc == 0
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Message not sent (-1): <copy t="cib" cib_op="cib_replace" cib_delegated_from="node-6.lab.example.com"
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] WARN: route_ais_message: Sending message to node-6.lab.example.com.cib failed: cluster delivery failed (rc=-1)
> Jun 28 14:56:56 node-2 corosync[30497]: [pcmk ] ERROR: send_cluster_msg_raw: Child 25054 spawned to record non-fatal assertion failure line 1594: rc == 0
>
> Google could not seem to turn up anything about the assertion message.
>
> I also saw these after setting the batch-limit to 1 and repeating my 8
> node (4 active, 4 idle) experiment today.

Thats unusual.

>
> But surely, it is easy to understand why pacemaker would have problems
> if corosync is aborting on a failed assertion.

Technically thats still pacemaker - the plugin code that gets loaded into corosync.

>
> Any clues what this one is about? This is corosync-1.4.1-4.el6_2.3.x86_64.

Looking at the source code, it seems totem (a corosync thing) is refusing to send the message to the other nodes.
I don't know under what conditions it would do this though.

Possibly it is saying "try again" but we don't have logic in place to do so.
You might have more success with pacemaker in cman mode - in this situation i know we do have retry logic in place when sending a cluster message fails.

>
> Cheers,
> b.
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker [at] oss
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: Pacemaker [at] oss
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads