Mailing List Archive: Linux-HA: HA-WG-Technical

RA spec: explicit "probe" operation?

Index | Next | Previous | View Threaded

lmb at suse

Jun 28, 2011, 2:10 AM

Post #1 of 31 (2397 views)
Permalink

RA spec: explicit "probe" operation?

Hi all,

triggered by the linux-ha-dev discussion, I'd also like to open the
discussion on another item on the revised specification list; namely, a
dedicated "probe" operation.

To recap, right now, Pacemaker uses the "monitor" operation to check if
the resource is active at all (prior to starting anything, as part of
the discovery process on a node).

Now, at this stage, in an empty cluster, nothing else will be active
yet either; so something that would be an error later, at "start" for
example, may just be expected. (Such as a file missing or a command
returning a weird state because they try to access shared storage.)

This, apparently, isn't all that easy to get right. A specific "probe"
operation, that is not tasked with verifying if the resource is healthy,
just if it is at all active, might be clearer, or at least that has been
suggested in the past.

The alternative would be to clarify the "monitor" semantics; "monitor"
just almost never strikes me as the right place to return
"ERR_INSTALLED" or "ERR_CONFIGURED", unless the evidence is really
strong (such as syntax violations in the parameters, for example).
These requirements can only be checked in full when we're attempting the
operation that actually needs them; "monitor" isn't "validate-all", it's
meant to find out the state of the resource only. IMHO, most of the
"ocf_is_probe" checks indicate that the monitor op is trying too much.

Personally, I'm leaning towards the latter; I don't really like the
"probe" operation idea, but being a completely impartial moderator, I'm
alas forced to bring it up, how bad the idea might be. ;-) Any
comments?

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

florian.haas at linbit

Jun 28, 2011, 3:51 AM

Post #2 of 31 (2340 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-06-28 11:10, Lars Marowsky-Bree wrote:
> Personally, I'm leaning towards the latter; I don't really like the
> "probe" operation idea, but being a completely impartial moderator, I'm
> alas forced to bring it up, how bad the idea might be. ;-) Any
> comments?

How about instead defining specific instances when the cluster _must_
call validate-all (I think it never does, now, but feel free to correct
me), retain the definition of the probe operation as is (a monitor
action that does not recur), and then restrict monitor to only check for
resource status and failure, rather than correct configuration?

Cheers,
Florian

Attachments:

signature.asc (0.26 KB)

lmb at suse

Jun 28, 2011, 5:32 AM

Post #3 of 31 (2336 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-06-28T12:51:43, Florian Haas <florian.haas [at] linbit> wrote:

> How about instead defining specific instances when the cluster _must_
> call validate-all (I think it never does, now, but feel free to correct
> me),

I'm not sure I follow where you're going here - which such specific
instances come to mind? Basically, there are only two points where validate-all
make sense:

a) automatically directly prior to an intended "start" - in which case
it is redundant, since the "start" can report exactly the same.

b) As a help for the UIs, to check if the parameters make sense and are
possibly even correct. (Which is what validate-all was intended for, to
provide deeper checking than a simple syntax check that the UI can
provide based on the data type.)

It'd probably be good to recommend that UIs actually do this when a
resource is added (and all its pre-requisites are running).

> retain the definition of the probe operation as is (a monitor action
> that does not recur),

Well, from the point of view of the definition of "monitor", that
doesn't matter. In theory, the repeat schedule for "monitor" wasn't
meant to ever be required, since "monitor" was intended to always
provide a correct result. Which leads me to:

> and then restrict monitor to only check for resource status and
> failure, rather than correct configuration?

This bit actually makes the problem go away indeed. The spec needs to
clarify that the "monitor" operation's primary goal is to ascertain the
state (running/failed/stopped/unknown).

That some monitors went beyond this (with the best of intentions) is
what actually is causing most of our problems in scenarios where this
doesn't make sense.

The RA can sometimes ascertain that the resource will never be able to
be started on that node unless the admin intervenes or the environment
is changed by other resources being brought online first. (Which is what
ERR_INSTALLED being returned by monitor_0 basically implies.)

Or that the semantics are completely broken (ip=430.a.49.2),
which is a valid cause for "ERR_CONFIGURED".

monitor_0 can reasonably check for this, iff carefully implemented (the
problem arises from those RAs that aren't, or where the logic has bugs);
splitting it off into a separate mandatory call to "validate-all" is not
necessarily a good idea, since it would double the number of startup
probe actions.

That would be the third option; not change anything, make "ocf_is_probe"
(and how to detect them) official, and document how implementors have to
be really careful about going beyond the mere state check.

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

dejan at suse

Jun 28, 2011, 7:36 AM

Post #4 of 31 (2334 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

Hi,

On Tue, Jun 28, 2011 at 02:32:24PM +0200, Lars Marowsky-Bree wrote:
> On 2011-06-28T12:51:43, Florian Haas <florian.haas [at] linbit> wrote:
>
> > How about instead defining specific instances when the cluster _must_
> > call validate-all (I think it never does, now, but feel free to correct
> > me),
>
> I'm not sure I follow where you're going here - which such specific
> instances come to mind? Basically, there are only two points where validate-all
> make sense:
>
> a) automatically directly prior to an intended "start" - in which case
> it is redundant, since the "start" can report exactly the same.
>
> b) As a help for the UIs, to check if the parameters make sense and are
> possibly even correct. (Which is what validate-all was intended for, to
> provide deeper checking than a simple syntax check that the UI can
> provide based on the data type.)
>
> It'd probably be good to recommend that UIs actually do this when a
> resource is added (and all its pre-requisites are running).

Not a bad idea, but how would the UI know that all requisite
resources are running? This is what we already discussed several
months ago, when I suggested that ptest somehow delivers the
dependencies, but everybody frowned to that (IIRC).

> > retain the definition of the probe operation as is (a monitor action
> > that does not recur),
>
> Well, from the point of view of the definition of "monitor", that
> doesn't matter. In theory, the repeat schedule for "monitor" wasn't
> meant to ever be required, since "monitor" was intended to always
> provide a correct result.

How do you mean "be required"? As something to check whether
it's a probe?

> Which leads me to:
>
> > and then restrict monitor to only check for resource status and
> > failure, rather than correct configuration?
>
> This bit actually makes the problem go away indeed. The spec needs to
> clarify that the "monitor" operation's primary goal is to ascertain the
> state (running/failed/stopped/unknown).
>
> That some monitors went beyond this (with the best of intentions) is
> what actually is causing most of our problems in scenarios where this
> doesn't make sense.

Are you suggesting that an RA shouldn't be doing deeper checks?
Though this could really be up for discussion, but so far the
idea was that an RA instance should do a bit more than just
check whether a process was running. After all, that's what
makes OCF RA better than LSB.

> The RA can sometimes ascertain that the resource will never be able to
> be started on that node unless the admin intervenes or the environment
> is changed by other resources being brought online first. (Which is what
> ERR_INSTALLED being returned by monitor_0 basically implies.)

Which doesn't work for all resources, as we recently discussed.
Some, such as oracle or db2, may even have binaries on shared
storage.

> Or that the semantics are completely broken (ip=430.a.49.2),
> which is a valid cause for "ERR_CONFIGURED".
>
> monitor_0 can reasonably check for this, iff carefully implemented (the
> problem arises from those RAs that aren't, or where the logic has bugs);
> splitting it off into a separate mandatory call to "validate-all" is not
> necessarily a good idea, since it would double the number of startup
> probe actions.
>
>
> That would be the third option; not change anything, make "ocf_is_probe"
> (and how to detect them) official, and document how implementors have to
> be really careful about going beyond the mere state check.

This is probably the most reasonable thing to do. Otherwise,
we'll go into changing all RAs, and that wouldn't be justified
in this case.

BTW, I have an (almost) ready RA driver for shell based RAs.
This driver takes care of probes, so that an RA using the driver
can split probe and monitor code. Actually, wrong handling of
probes was the main motivation to implement it.

Cheers,

Dejan

>
> Regards,
> Lars
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
> _______________________________________________
> ha-wg-technical mailing list
> ha-wg-technical [at] lists
> https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jun 28, 2011, 2:02 PM

Post #5 of 31 (2339 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-06-28T16:36:45, Dejan Muhamedagic <dejan [at] suse> wrote:

> > It'd probably be good to recommend that UIs actually do this when a
> > resource is added (and all its pre-requisites are running).
> Not a bad idea, but how would the UI know that all requisite
> resources are running? This is what we already discussed several
> months ago, when I suggested that ptest somehow delivers the
> dependencies, but everybody frowned to that (IIRC).

The admin could tell the UI to check the parameters (most of the time,
people will add new resources when their pre-requisites are running, so
that is easy). Or indeed, it could parse ptest output.

But that is actually beyond this spec discussion; what I was trying to
understand is when "validate-all" would be mandatory to run like Florian
suggested; the above is only an example where it _could_ be run.

> > Well, from the point of view of the definition of "monitor", that
> > doesn't matter. In theory, the repeat schedule for "monitor" wasn't
> > meant to ever be required, since "monitor" was intended to always
> > provide a correct result.
> How do you mean "be required"? As something to check whether
> it's a probe?

I meant "required to be known to the RA", sorry, I thought the context
was clear.

> Are you suggesting that an RA shouldn't be doing deeper checks?
> Though this could really be up for discussion, but so far the
> idea was that an RA instance should do a bit more than just
> check whether a process was running. After all, that's what
> makes OCF RA better than LSB.

No, that wasn't why we invented OCF RAs instead of going with LSB. The
main distinction is that OCF RAs take instance parameters. (And that we
got to define new actions for some.)

And that "monitor" _can_ do more, but it doesn't _need_ to; the primary
goal is to determine running/stopped/failed/unknown state, and it
shouldn't report anything else if it can't be sure about it.

> > The RA can sometimes ascertain that the resource will never be able to
> > be started on that node unless the admin intervenes or the environment
> > is changed by other resources being brought online first. (Which is what
> > ERR_INSTALLED being returned by monitor_0 basically implies.)
> Which doesn't work for all resources, as we recently discussed.
> Some, such as oracle or db2, may even have binaries on shared
> storage.

Which is exactly the point. I'm not sure where you're contradicting me?

> BTW, I have an (almost) ready RA driver for shell based RAs.
> This driver takes care of probes, so that an RA using the driver
> can split probe and monitor code. Actually, wrong handling of
> probes was the main motivation to implement it.

I'm not sure at all what you're talking about here; what is this?

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

florian.haas at linbit

Jun 28, 2011, 10:56 PM

Post #6 of 31 (2329 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 06/28/2011 02:32 PM, Lars Marowsky-Bree wrote:
> On 2011-06-28T12:51:43, Florian Haas <florian.haas [at] linbit> wrote:
>
>> How about instead defining specific instances when the cluster _must_
>> call validate-all (I think it never does, now, but feel free to correct
>> me),
>
> I'm not sure I follow where you're going here - which such specific
> instances come to mind? Basically, there are only two points where validate-all
> make sense:
>
> a) automatically directly prior to an intended "start" - in which case
> it is redundant, since the "start" can report exactly the same.
>
> b) As a help for the UIs, to check if the parameters make sense and are
> possibly even correct. (Which is what validate-all was intended for, to
> provide deeper checking than a simple syntax check that the UI can
> provide based on the data type.)

c) Whenever a node comes online. Then, the cluster could run
validate-all for all defined resources on the newly joined node. Since
validate-all would be the operation that is "allowed" to return
$OCF_ERR_CONFIGURED or $OCF_ERR_INSTALLED or $OCF_ERR_PERM, the cluster
would immediately know which nodes are eligible for running the
resource. It could, possibly, set an implicit -INF location constraint
for resources on ineligible nodes. Only then would it proceed to probe,
and only on the eligible nodes.

>> retain the definition of the probe operation as is (a monitor action
>> that does not recur),
>
> Well, from the point of view of the definition of "monitor", that
> doesn't matter. In theory, the repeat schedule for "monitor" wasn't
> meant to ever be required, since "monitor" was intended to always
> provide a correct result. Which leads me to:
>
>> and then restrict monitor to only check for resource status and
>> failure, rather than correct configuration?
>
> This bit actually makes the problem go away indeed. The spec needs to
> clarify that the "monitor" operation's primary goal is to ascertain the
> state (running/failed/stopped/unknown).

More precisely, running/runtime failure (as opposed to configuration
error or unsatisfied prerequisites)/stopped/unknown. Although I'm not so
sure if "unknown" is actually sensible. We have no "unknown" state at
this point, unless you are referring to the unmanaged state we get into
if stop fails and fencing is either not configured or fails as well.

> That some monitors went beyond this (with the best of intentions) is
> what actually is causing most of our problems in scenarios where this
> doesn't make sense.
>
>
> The RA can sometimes ascertain that the resource will never be able to
> be started on that node unless the admin intervenes or the environment
> is changed by other resources being brought online first. (Which is what
> ERR_INSTALLED being returned by monitor_0 basically implies.)
>
> Or that the semantics are completely broken (ip=430.a.49.2),
> which is a valid cause for "ERR_CONFIGURED".
>
> monitor_0 can reasonably check for this, iff carefully implemented (the
> problem arises from those RAs that aren't, or where the logic has bugs);
> splitting it off into a separate mandatory call to "validate-all" is not
> necessarily a good idea, since it would double the number of startup
> probe actions.

I don't think so; see the cluster behavior I suggested above.

Cheers,
Florian

Attachments:

signature.asc (0.26 KB)

tserong at novell

Jun 28, 2011, 11:40 PM

Post #7 of 31 (2338 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 29/06/11 15:56, Florian Haas wrote:
> On 06/28/2011 02:32 PM, Lars Marowsky-Bree wrote:
>> On 2011-06-28T12:51:43, Florian Haas<florian.haas [at] linbit> wrote:
>>
>>> How about instead defining specific instances when the cluster _must_
>>> call validate-all (I think it never does, now, but feel free to correct
>>> me),
>>
>> I'm not sure I follow where you're going here - which such specific
>> instances come to mind? Basically, there are only two points where validate-all
>> make sense:
>>
>> a) automatically directly prior to an intended "start" - in which case
>> it is redundant, since the "start" can report exactly the same.
>>
>> b) As a help for the UIs, to check if the parameters make sense and are
>> possibly even correct. (Which is what validate-all was intended for, to
>> provide deeper checking than a simple syntax check that the UI can
>> provide based on the data type.)
>
> c) Whenever a node comes online. Then, the cluster could run
> validate-all for all defined resources on the newly joined node. Since
> validate-all would be the operation that is "allowed" to return
> $OCF_ERR_CONFIGURED or $OCF_ERR_INSTALLED or $OCF_ERR_PERM, the cluster
> would immediately know which nodes are eligible for running the
> resource. It could, possibly, set an implicit -INF location constraint
> for resources on ineligible nodes. Only then would it proceed to probe,
> and only on the eligible nodes.

That might fail or get weird if one resource depends on another
(resource B's config is on cluster filesystem A), in which case we'd
need to take colocation and ordering constraints into account and only
run validate-all on resources whose dependencies are already running...?

Regards,

Tim
--
Tim Serong <tserong [at] novell>
Senior Clustering Engineer, OPS Engineering, Novell Inc.
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jun 29, 2011, 2:06 AM

Post #8 of 31 (2322 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-06-29T07:56:48, Florian Haas <florian.haas [at] linbit> wrote:

> c) Whenever a node comes online. Then, the cluster could run
> validate-all for all defined resources on the newly joined node.

Like Tim said, this fails for exactly the same reason that "monitor_0"
has undesirable properties, I'm afraid.

Also, "monitor_0" would still need to run. (Theoretically, just because
it's misconfigured and shouldn't run there doesn't mean it isn't active
in a failed state.)

Unless we probe in dependency order, but even that would be wrong - we'd
possibly not probe for resources that are active out of turn. (Think
groups: a-b-c, c shouldn't be active if a-b aren't running, but an IP
could still have been started on boot.)

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

andrew at beekhof

Jun 29, 2011, 5:32 PM

Post #9 of 31 (2315 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

Random thought...

What about having monitor just be about started/stopped/failed (as
suggested above) and have the cluster automatically call validate-all
(which would check for tools and config options) before start ops?

On Wed, Jun 29, 2011 at 7:06 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> On 2011-06-29T07:56:48, Florian Haas <florian.haas [at] linbit> wrote:
>
>> c) Whenever a node comes online. Then, the cluster could run
>> validate-all for all defined resources on the newly joined node.
>
> Like Tim said, this fails for exactly the same reason that "monitor_0"
> has undesirable properties, I'm afraid.
>
> Also, "monitor_0" would still need to run. (Theoretically, just because
> it's misconfigured and shouldn't run there doesn't mean it isn't active
> in a failed state.)
>
> Unless we probe in dependency order, but even that would be wrong - we'd
> possibly not probe for resources that are active out of turn. (Think
> groups: a-b-c, c shouldn't be active if a-b aren't running, but an IP
> could still have been started on boot.)
>
>
> Regards,
> Lars
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
> _______________________________________________
> ha-wg-technical mailing list
> ha-wg-technical [at] lists
> https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical
>
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jun 30, 2011, 1:10 AM

Post #10 of 31 (2313 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-06-30T10:32:56, Andrew Beekhof <andrew [at] beekhof> wrote:

> Random thought...
>
> What about having monitor just be about started/stopped/failed (as
> suggested above) and have the cluster automatically call validate-all
> (which would check for tools and config options) before start ops?

What would the point be?

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

andrew at beekhof

Jun 30, 2011, 2:07 AM

Post #11 of 31 (2307 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Thu, Jun 30, 2011 at 6:10 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> On 2011-06-30T10:32:56, Andrew Beekhof <andrew [at] beekhof> wrote:
>
>> Random thought...
>>
>> What about having monitor just be about started/stopped/failed (as
>> suggested above) and have the cluster automatically call validate-all
>> (which would check for tools and config options) before start ops?
>
> What would the point be?

It would alleviate the part where RA writers need to know (and we need
to document) when to call validate-all.
Failures would also show up in the CIB (and therefor the tools) under
the validate-all op, not start - this might be slightly more helpful
for users' debugging.

>
>
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
>
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jun 30, 2011, 2:09 AM

Post #12 of 31 (2310 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-06-30T19:07:42, Andrew Beekhof <andrew [at] beekhof> wrote:

> It would alleviate the part where RA writers need to know (and we need
> to document) when to call validate-all.

Uhm, RA writers don't need to know when to call validate-all.

> Failures would also show up in the CIB (and therefor the tools) under
> the validate-all op, not start - this might be slightly more helpful
> for users' debugging.

At the expense of doubling the number of operations we need to call for
"start", and doubling the effort - since, clearly, "start" needs to
check all these requirements again.

(Just like any op needs to check its own requirements; or at least
implicitly does, since it otherwise would fail to complete.)

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

andrew at beekhof

Jun 30, 2011, 2:36 AM

Post #13 of 31 (2310 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Thu, Jun 30, 2011 at 7:09 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> On 2011-06-30T19:07:42, Andrew Beekhof <andrew [at] beekhof> wrote:
>
>> It would alleviate the part where RA writers need to know (and we need
>> to document) when to call validate-all.
>
> Uhm, RA writers don't need to know when to call validate-all.

I was referring to this (which apparently I only half read):

> a) automatically directly prior to an intended "start" - in which case
> it is redundant, since the "start" can report exactly the same.

Start can do this, but only if we educate RA writers to do so.
I'd favor doing it explicitly and automagically.

>
>> Failures would also show up in the CIB (and therefor the tools) under
>> the validate-all op, not start - this might be slightly more helpful
>> for users' debugging.
>
> At the expense of doubling the number of operations we need to call for
> "start", and doubling the effort - since, clearly, "start" needs to
> check all these requirements again.

Every RA already calls validate-all before really trying to start?

> (Just like any op needs to check its own requirements; or at least
> implicitly does, since it otherwise would fail to complete.)
>
>
> Regards,
> Lars
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
>
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

dejan at suse

Jun 30, 2011, 3:48 AM

Post #14 of 31 (2318 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Tue, Jun 28, 2011 at 11:02:32PM +0200, Lars Marowsky-Bree wrote:
> On 2011-06-28T16:36:45, Dejan Muhamedagic <dejan [at] suse> wrote:
>
> > > It'd probably be good to recommend that UIs actually do this when a
> > > resource is added (and all its pre-requisites are running).
> > Not a bad idea, but how would the UI know that all requisite
> > resources are running? This is what we already discussed several
> > months ago, when I suggested that ptest somehow delivers the
> > dependencies, but everybody frowned to that (IIRC).
>
> The admin could tell the UI to check the parameters (most of the time,
> people will add new resources when their pre-requisites are running, so
> that is easy). Or indeed, it could parse ptest output.
>
> But that is actually beyond this spec discussion; what I was trying to

Yes, it's a different matter.

> understand is when "validate-all" would be mandatory to run like Florian
> suggested; the above is only an example where it _could_ be run.
>
> > > Well, from the point of view of the definition of "monitor", that
> > > doesn't matter. In theory, the repeat schedule for "monitor" wasn't
> > > meant to ever be required, since "monitor" was intended to always
> > > provide a correct result.
> > How do you mean "be required"? As something to check whether
> > it's a probe?
>
> I meant "required to be known to the RA", sorry, I thought the context
> was clear.
>
> > Are you suggesting that an RA shouldn't be doing deeper checks?
> > Though this could really be up for discussion, but so far the
> > idea was that an RA instance should do a bit more than just
> > check whether a process was running. After all, that's what
> > makes OCF RA better than LSB.
>
> No, that wasn't why we invented OCF RAs instead of going with LSB. The
> main distinction is that OCF RAs take instance parameters. (And that we
> got to define new actions for some.)
>
> And that "monitor" _can_ do more, but it doesn't _need_ to; the primary
> goal is to determine running/stopped/failed/unknown state, and it
> shouldn't report anything else if it can't be sure about it.

Of course, the RA won't make up things :)

> > > The RA can sometimes ascertain that the resource will never be able to
> > > be started on that node unless the admin intervenes or the environment
> > > is changed by other resources being brought online first. (Which is what
> > > ERR_INSTALLED being returned by monitor_0 basically implies.)
> > Which doesn't work for all resources, as we recently discussed.
> > Some, such as oracle or db2, may even have binaries on shared
> > storage.
>
> Which is exactly the point. I'm not sure where you're contradicting me?

ERR_INSTALLED says that the resource can never run on this node.
In case of probes, that may not be true if for instance shared
storage is not mounted. So, in some RAs probe must return
NOT_RUNNING if some requirements are not fulfilled.

> > BTW, I have an (almost) ready RA driver for shell based RAs.
> > This driver takes care of probes, so that an RA using the driver
> > can split probe and monitor code. Actually, wrong handling of
> > probes was the main motivation to implement it.
>
> I'm not sure at all what you're talking about here; what is this?

The idea is to streamline RA development further. Basically, an
RA could look like this:

REQUIRED_PARAMS="config ..."
REQUIRED_BINARIES="b1 b2 ..."

xyz_metadata() {
...
}
xyz_start() {
...
}
xyz_stop() {
...
}
xyz_monitor_10() {
...
}
xyz_monitor() {
...
}
xyz_probe() {
...
}
xyz_validate_all() {
...
}

ocf_rarun $@

ocf_rarun would take care of all the boring details, checking
parameters, doing validation when needed, invoking the right
monitor (or probe), etc.

Thanks,

Dejan

> Regards,
> Lars
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jun 30, 2011, 1:48 PM

Post #15 of 31 (2300 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-06-30T12:48:03, Dejan Muhamedagic <dejan [at] suse> wrote:

> > > > The RA can sometimes ascertain that the resource will never be able to
> > > > be started on that node unless the admin intervenes or the environment
> > > > is changed by other resources being brought online first. (Which is what
> > > > ERR_INSTALLED being returned by monitor_0 basically implies.)
> > > Which doesn't work for all resources, as we recently discussed.
> > > Some, such as oracle or db2, may even have binaries on shared
> > > storage.
> > Which is exactly the point. I'm not sure where you're contradicting me?
> ERR_INSTALLED says that the resource can never run on this node.
> In case of probes, that may not be true if for instance shared
> storage is not mounted. So, in some RAs probe must return
> NOT_RUNNING if some requirements are not fulfilled.

Which is exactly why "monitor" shouldn't return ERR_INSTALLED, if this
possibility exists. That was the whole point of this discussion?

Note that "binary not present", if one wants to be pedantic, is not
identical to "not running"; it is identical to "not start or stoppable",
in all likelihood, but ps could still be used to see if the process is
around. On the other hand, "data share not mounted" probably is a pretty
good indicator of "not running".

> ocf_rarun $@

Ah, so a template function that takes care of the code that is shared
across resource agents. Yes, that's a good idea, but a completely
different discussion than this one.

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

florian.haas at linbit

Jul 1, 2011, 12:47 AM

Post #16 of 31 (2297 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-06-30 11:36, Andrew Beekhof wrote:
> On Thu, Jun 30, 2011 at 7:09 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
>> On 2011-06-30T19:07:42, Andrew Beekhof <andrew [at] beekhof> wrote:
>>
>>> It would alleviate the part where RA writers need to know (and we need
>>> to document) when to call validate-all.
>>
>> Uhm, RA writers don't need to know when to call validate-all.
>
> I was referring to this (which apparently I only half read):
>
>> a) automatically directly prior to an intended "start" - in which case
>> it is redundant, since the "start" can report exactly the same.
>
> Start can do this, but only if we educate RA writers to do so.
> I'd favor doing it explicitly and automagically.

I'm in full agreement with Andrew on this one.

Cheers,
Florian

Attachments:

signature.asc (0.26 KB)

andrew at beekhof

Jul 1, 2011, 1:09 AM

Post #17 of 31 (2290 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Fri, Jul 1, 2011 at 5:47 PM, Florian Haas <florian.haas [at] linbit> wrote:
> On 2011-06-30 11:36, Andrew Beekhof wrote:
>> On Thu, Jun 30, 2011 at 7:09 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
>>> On 2011-06-30T19:07:42, Andrew Beekhof <andrew [at] beekhof> wrote:
>>>
>>>> It would alleviate the part where RA writers need to know (and we need
>>>> to document) when to call validate-all.
>>>
>>> Uhm, RA writers don't need to know when to call validate-all.
>>
>> I was referring to this (which apparently I only half read):
>>
>>> a) automatically directly prior to an intended "start" - in which case
>>> it is redundant, since the "start" can report exactly the same.
>>
>> Start can do this, but only if we educate RA writers to do so.
>> I'd favor doing it explicitly and automagically.
>
> I'm in full agreement with Andrew on this one.

Oh crap. Sorry everyone, I must be wrong ;-)
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

dejan at suse

Jul 1, 2011, 1:51 AM

Post #18 of 31 (2294 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Fri, Jul 01, 2011 at 06:09:10PM +1000, Andrew Beekhof wrote:
> On Fri, Jul 1, 2011 at 5:47 PM, Florian Haas <florian.haas [at] linbit> wrote:
> > On 2011-06-30 11:36, Andrew Beekhof wrote:
> >> On Thu, Jun 30, 2011 at 7:09 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> >>> On 2011-06-30T19:07:42, Andrew Beekhof <andrew [at] beekhof> wrote:
> >>>
> >>>> It would alleviate the part where RA writers need to know (and we need
> >>>> to document) when to call validate-all.
> >>>
> >>> Uhm, RA writers don't need to know when to call validate-all.
> >>
> >> I was referring to this (which apparently I only half read):
> >>
> >>> a) automatically directly prior to an intended "start" - in which case
> >>> it is redundant, since the "start" can report exactly the same.
> >>
> >> Start can do this, but only if we educate RA writers to do so.
> >> I'd favor doing it explicitly and automagically.
> >
> > I'm in full agreement with Andrew on this one.
>
> Oh crap. Sorry everyone, I must be wrong ;-)

I concur with both :)

> _______________________________________________
> ha-wg-technical mailing list
> ha-wg-technical [at] lists
> https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jul 6, 2011, 8:31 AM

Post #19 of 31 (2270 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-07-01T18:09:10, Andrew Beekhof <andrew [at] beekhof> wrote:

> >>>> It would alleviate the part where RA writers need to know (and we need
> >>>> to document) when to call validate-all.
> >>> Uhm, RA writers don't need to know when to call validate-all.
> >> I was referring to this (which apparently I only half read):
> >>> a) automatically directly prior to an intended "start" - in which case
> >>> it is redundant, since the "start" can report exactly the same.
> >> Start can do this, but only if we educate RA writers to do so.
> >> I'd favor doing it explicitly and automagically.
> > I'm in full agreement with Andrew on this one.
> Oh crap. Sorry everyone, I must be wrong ;-)

You both actually are. ;-)

Either "start" succeeds at this point, or it doesn't. Calling
"validate-all" doesn't provide any additional information that calling
"start" directly wouldn't; it is superfluous.

That's obvious, isn't it?

"validate-all" makes sense in situations where you're not planning to do
anything else immediately; like, a UI trying to figure out if the user
gave it reasonably useful parameters.

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

florian.haas at linbit

Jul 6, 2011, 8:43 AM

Post #20 of 31 (2269 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-07-06 17:31, Lars Marowsky-Bree wrote:
> On 2011-07-01T18:09:10, Andrew Beekhof <andrew [at] beekhof> wrote:
>
>>>>>> It would alleviate the part where RA writers need to know (and we need
>>>>>> to document) when to call validate-all.
>>>>> Uhm, RA writers don't need to know when to call validate-all.
>>>> I was referring to this (which apparently I only half read):
>>>>> a) automatically directly prior to an intended "start" - in which case
>>>>> it is redundant, since the "start" can report exactly the same.
>>>> Start can do this, but only if we educate RA writers to do so.
>>>> I'd favor doing it explicitly and automagically.
>>> I'm in full agreement with Andrew on this one.
>> Oh crap. Sorry everyone, I must be wrong ;-)
>
> You both actually are. ;-)
>
> Either "start" succeeds at this point, or it doesn't. Calling
> "validate-all" doesn't provide any additional information that calling
> "start" directly wouldn't; it is superfluous.
>
> That's obvious, isn't it?

So obvious that it's typical to see a new contributor forget to do any
sort of validation, on start or elsewhere.

Just sayin'.

Florian

Attachments:

signature.asc (0.26 KB)

lmb at suse

Jul 7, 2011, 3:42 AM

Post #21 of 31 (2265 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-07-06T17:43:01, Florian Haas <florian.haas [at] linbit> wrote:

> > Either "start" succeeds at this point, or it doesn't. Calling
> > "validate-all" doesn't provide any additional information that calling
> > "start" directly wouldn't; it is superfluous.
> >
> > That's obvious, isn't it?
> So obvious that it's typical to see a new contributor forget to do any
> sort of validation, on start or elsewhere.
>
> Just sayin'.

Yes. but those RAs would still be broken.

I agree we ought to invent a place where validate-all _is_ called, and
have UIs that actually utilize it, but we shouldn't hide a broken action
implementation behind it.

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

andrew at beekhof

Jul 7, 2011, 5:54 PM

Post #22 of 31 (2263 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Thu, Jul 7, 2011 at 8:42 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> On 2011-07-06T17:43:01, Florian Haas <florian.haas [at] linbit> wrote:
>
>> > Either "start" succeeds at this point, or it doesn't. Calling
>> > "validate-all" doesn't provide any additional information that calling
>> > "start" directly wouldn't; it is superfluous.
>> >
>> > That's obvious, isn't it?
>> So obvious that it's typical to see a new contributor forget to do any
>> sort of validation, on start or elsewhere.
>>
>> Just sayin'.
>
> Yes. but those RAs would still be broken.
>
> I agree we ought to invent a place where validate-all _is_ called, and
> have UIs that actually utilize it, but we shouldn't hide a broken action
> implementation behind it.
>

Either:

1) start must always call validate-all, or
2) RA writers duplicate the validate-all checks in start

Otherwise your assertion that:

> > Calling
> > "validate-all" doesn't provide any additional information that calling
> > "start" directly wouldn't; it is superfluous.

Is incorrect.

I think most of us are agreeing that neither are desirable and that

3) we automatically call validate-all before start

Is the better path forward.
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jul 8, 2011, 2:40 PM

Post #23 of 31 (2242 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-07-08T10:54:17, Andrew Beekhof <andrew [at] beekhof> wrote:

> Either:
>
> 1) start must always call validate-all, or

RAs are supposed to be reasonably paranoid.

Validating the input on any request is just insanely good practice. (At
least as far as they are relevant to the request.)

As a somewhat silly example, consider validate-all checking that
"tmpdir" is non-empty and a valid directory. Consider that start removes
it. Consider what will happen if something goes wrong with the
parameter, or one of our users manages to call "start" manually. (You
know they will.)

The operation and the validation for the parameters that operation takes
belong in _one_ execution context, not split into two.

> Otherwise your assertion that:
>
> > > Calling
> > > "validate-all" doesn't provide any additional information that calling
> > > "start" directly wouldn't; it is superfluous.
>
> Is incorrect.

It is not. But if parameters are incorrect, start will fail, even if it
didn't explicitly validate them; otherwise, it can't reasonably expect
what will happen. Hence, it should validate them.

If "start" is called with wrong parameters, it _needs to cope with
that_. Consider:

Calling just "start" results in the service being up or not (with a
failure).

Calling validate-all + start reuslts in the service being up or not
(with a, perhaps, somewhat more detailed failure). But since start is
still allowed to fail (obviously), something can still go wrong there;
and then, start needs to report that properly.

But the service will still be either up or down, and that is what we
actually want to know at that stage; from the point of the cluster
manager, it doesn't contain more information. And we still have a start
operation that needs to be prepared to handle a failure.

> I think most of us are agreeing that neither are desirable and that
>
> 3) we automatically call validate-all before start
>
> Is the better path forward.

No! Please, don't. It incurs a completely pointless fork and the whole
setup of the RA. We need to make clusters -faster-, not double the
amount of actions they need to run before starting a service!

Not to mention that it will lead people to write even sloppier start
operations than they already do.

I really really REALLY think you're on the wrong path here. As in,
really quite wrong. Please, reconsider and think that through.

validate-all was never meant to be an automatic operation. It was meant
as a UI aid, not as a mandatory step. We _can_ redefine that, but then
most likely all RAs out there will need fixing. (More than they already
do.) For little to negative gain.

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

andrew at beekhof

Jul 9, 2011, 12:23 AM

Post #24 of 31 (2245 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Sat, Jul 9, 2011 at 7:40 AM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> On 2011-07-08T10:54:17, Andrew Beekhof <andrew [at] beekhof> wrote:
>
>> Either:
>>
>> 1) start must always call validate-all, or
>
> RAs are supposed to be reasonably paranoid.
>
> Validating the input on any request is just insanely good practice. (At
> least as far as they are relevant to the request.)
>
> As a somewhat silly example, consider validate-all checking that
> "tmpdir" is non-empty and a valid directory. Consider that start removes
> it. Consider what will happen if something goes wrong with the
> parameter, or one of our users manages to call "start" manually. (You
> know they will.)
>
> The operation and the validation for the parameters that operation takes
> belong in _one_ execution context, not split into two.

So you're arguing that validate-all should go away?

>
>> Otherwise your assertion that:
>>
>> > > Calling
>> > > "validate-all" doesn't provide any additional information that calling
>> > > "start" directly wouldn't; it is superfluous.
>>
>> Is incorrect.
>
> It is not. But if parameters are incorrect, start will fail,

Without doubt. However not all failures (and error codes) are created equal.
As long as something is designed to check this stuff _and_ its
actually being called - I don't actually care.

I just thought that place was validate-all.

> even if it
> didn't explicitly validate them; otherwise, it can't reasonably expect
> what will happen. Hence, it should validate them.
>
> If "start" is called with wrong parameters, it _needs to cope with
> that_. Consider:
>
> Calling just "start" results in the service being up or not (with a
> failure).
>
> Calling validate-all + start reuslts in the service being up or not
> (with a, perhaps, somewhat more detailed failure). But since start is
> still allowed to fail (obviously), something can still go wrong there;
> and then, start needs to report that properly.

A "somewhat more detailed failure" is the part I care about.
How many "it doesn't start" emails/reports do we get? Far too many to count.

I think having the cluster abort before that point would make all our
lives easier as it would be more obvious that the error is related to
the config or install.

>
> But the service will still be either up or down, and that is what we
> actually want to know at that stage; from the point of the cluster
> manager, it doesn't contain more information. And we still have a start
> operation that needs to be prepared to handle a failure.
>
>> I think most of us are agreeing that neither are desirable and that
>>
>> 3) we automatically call validate-all before start
>>
>> Is the better path forward.
>
> No! Please, don't. It incurs a completely pointless fork and the whole
> setup of the RA. We need to make clusters -faster-, not double the
> amount of actions they need to run before starting a service!

I think you're exaggerating the amount of overhead.

time(validate-all) != time(start) and time(fork) << time(start) in most cases.

> Not to mention that it will lead people to write even sloppier start
> operations than they already do.
>
> I really really REALLY think you're on the wrong path here. As in,
> really quite wrong. Please, reconsider and think that through.

Like I said above, I don't much care where the checks are located.
But having defined a method called validate-all, it seems the logical
place to put them.

If the logic should be executed as part of a start op and you don't
want 1), then we should remove the function completely.

> validate-all was never meant to be an automatic operation. It was meant
> as a UI aid, not as a mandatory step. We _can_ redefine that, but then
> most likely all RAs out there will need fixing. (More than they already
> do.) For little to negative gain.
>
>
> Regards,
> Lars
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
> _______________________________________________
> ha-wg-technical mailing list
> ha-wg-technical [at] lists
> https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical
>
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

dejan at suse

Jul 9, 2011, 6:09 AM

Post #25 of 31 (2239 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Fri, Jul 08, 2011 at 11:40:03PM +0200, Lars Marowsky-Bree wrote:
> On 2011-07-08T10:54:17, Andrew Beekhof <andrew [at] beekhof> wrote:
>
> > Either:
> >
> > 1) start must always call validate-all, or
>
> RAs are supposed to be reasonably paranoid.
>
> Validating the input on any request is just insanely good practice. (At
> least as far as they are relevant to the request.)
>
> As a somewhat silly example, consider validate-all checking that
> "tmpdir" is non-empty and a valid directory. Consider that start removes
> it. Consider what will happen if something goes wrong with the
> parameter, or one of our users manages to call "start" manually. (You
> know they will.)
>
> The operation and the validation for the parameters that operation takes
> belong in _one_ execution context, not split into two.
>
> > Otherwise your assertion that:
> >
> > > > Calling
> > > > "validate-all" doesn't provide any additional information that calling
> > > > "start" directly wouldn't; it is superfluous.
> >
> > Is incorrect.
>
> It is not. But if parameters are incorrect, start will fail, even if it
> didn't explicitly validate them; otherwise, it can't reasonably expect
> what will happen. Hence, it should validate them.
>
> If "start" is called with wrong parameters, it _needs to cope with
> that_. Consider:
>
> Calling just "start" results in the service being up or not (with a
> failure).
>
> Calling validate-all + start reuslts in the service being up or not
> (with a, perhaps, somewhat more detailed failure). But since start is
> still allowed to fail (obviously), something can still go wrong there;
> and then, start needs to report that properly.
>
> But the service will still be either up or down, and that is what we
> actually want to know at that stage; from the point of the cluster
> manager, it doesn't contain more information. And we still have a start
> operation that needs to be prepared to handle a failure.
>
> > I think most of us are agreeing that neither are desirable and that
> >
> > 3) we automatically call validate-all before start
> >
> > Is the better path forward.
>
> No! Please, don't. It incurs a completely pointless fork and the whole
> setup of the RA. We need to make clusters -faster-, not double the
> amount of actions they need to run before starting a service!
>
> Not to mention that it will lead people to write even sloppier start
> operations than they already do.
>
> I really really REALLY think you're on the wrong path here. As in,
> really quite wrong. Please, reconsider and think that through.
>
> validate-all was never meant to be an automatic operation. It was meant
> as a UI aid, not as a mandatory step. We _can_ redefine that, but then
> most likely all RAs out there will need fixing. (More than they already
> do.) For little to negative gain.

I'd agree.

Dejan

> Regards,
> Lars
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
> _______________________________________________
> ha-wg-technical mailing list
> ha-wg-technical [at] lists
> https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jul 11, 2011, 3:21 AM

Post #26 of 31 (655 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-07-09T17:23:04, Andrew Beekhof <andrew [at] beekhof> wrote:

> > As a somewhat silly example, consider validate-all checking that
> > "tmpdir" is non-empty and a valid directory. Consider that start removes
> > it. Consider what will happen if something goes wrong with the
> > parameter, or one of our users manages to call "start" manually. (You
> > know they will.)
> >
> > The operation and the validation for the parameters that operation takes
> > belong in _one_ execution context, not split into two.
>
> So you're arguing that validate-all should go away?

Uh? No. validate-all makes perfect sense for an UI or so that wants to
validate the resource agents parameters without actually starting or
stopping it.

> > It is not. But if parameters are incorrect, start will fail,
> Without doubt. However not all failures (and error codes) are created equal.

Incorrect. They are at least _created_ equal. ;-) But indeed, they might
not always check all the same things.

> A "somewhat more detailed failure" is the part I care about.
> How many "it doesn't start" emails/reports do we get? Far too many to count.

That's a systematic problem though that won't be solved by calling two
actions. They will still not get it - so the cluster calls two
operations instead of one, what are the chances that they'll read the
messages from "validate-all" when they currently don't read "start"? ;-)

> I think having the cluster abort before that point would make all our
> lives easier as it would be more obvious that the error is related to
> the config or install.

Right, the UI should actually make use of validate-all - because then
the errors will be reported _when the user actually configures it_, that
makes sense. And hopefully they'll get it then.

(Sure, that doesn't always make sense when they're using a huge shadow
CIB commit, but there's always something.)

How about a phrasing like this in the validate-all description:

"User interfaces should make appropriate use of this operation to
validate the resource's settings prior to adding it to the cluster
configuration."?

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

andrew at beekhof

Jul 11, 2011, 4:10 AM

Post #27 of 31 (652 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Mon, Jul 11, 2011 at 8:21 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> On 2011-07-09T17:23:04, Andrew Beekhof <andrew [at] beekhof> wrote:
>
>> > As a somewhat silly example, consider validate-all checking that
>> > "tmpdir" is non-empty and a valid directory. Consider that start removes
>> > it. Consider what will happen if something goes wrong with the
>> > parameter, or one of our users manages to call "start" manually. (You
>> > know they will.)
>> >
>> > The operation and the validation for the parameters that operation takes
>> > belong in _one_ execution context, not split into two.
>>
>> So you're arguing that validate-all should go away?
>
> Uh? No. validate-all makes perfect sense for an UI or so that wants to
> validate the resource agents parameters without actually starting or
> stopping it.

Okay. But you don't want "start calling validate-all" to be "best
practice", and code duplication is a bad thing... so what do you
propose?

>
>> > It is not. But if parameters are incorrect, start will fail,
>> Without doubt. However not all failures (and error codes) are created equal.
>
> Incorrect. They are at least _created_ equal. ;-) But indeed, they might
> not always check all the same things.

Which is important.
Its important that validate-all and start return appropriate error
messages instead of throwing up their hands and chucking out
OCF_ERR_GENERIC

>
>> A "somewhat more detailed failure" is the part I care about.
>> How many "it doesn't start" emails/reports do we get? Far too many to count.
>
> That's a systematic problem though that won't be solved by calling two
> actions. They will still not get it - so the cluster calls two
> operations instead of one, what are the chances that they'll read the
> messages from "validate-all" when they currently don't read "start"? ;-)
>
>> I think having the cluster abort before that point would make all our
>> lives easier as it would be more obvious that the error is related to
>> the config or install.
>
> Right, the UI should actually make use of validate-all - because then
> the errors will be reported _when the user actually configures it_, that
> makes sense. And hopefully they'll get it then.
>
> (Sure, that doesn't always make sense when they're using a huge shadow
> CIB commit, but there's always something.)
>
> How about a phrasing like this in the validate-all description:
>
> "User interfaces should make appropriate use of this operation to
> validate the resource's settings prior to adding it to the cluster
> configuration."?
>
>
> Regards,
> Lars
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
>
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jul 11, 2011, 4:19 AM

Post #28 of 31 (651 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-07-11T21:10:00, Andrew Beekhof <andrew [at] beekhof> wrote:

> > Uh? No. validate-all makes perfect sense for an UI or so that wants to
> > validate the resource agents parameters without actually starting or
> > stopping it.
> Okay. But you don't want "start calling validate-all" to be "best
> practice", and code duplication is a bad thing... so what do you
> propose?

Well, as far as "validate-all" is concerned, I'dn't actually change that
call at all, and not make it mandatory either. But it would be awesome
if the UIs actually called it - yet, that is beyond the specification, I
see no need for it to be mandatory.

But like I proposed below in the mail here - we can increase the
pressure a bit on validate-all actually being made use of by the UIs.

> > Incorrect. They are at least _created_ equal. ;-) But indeed, they might
> > not always check all the same things.
> Which is important.
> Its important that validate-all and start return appropriate error
> messages instead of throwing up their hands and chucking out
> OCF_ERR_GENERIC

Right. But that's already required, and I doubt we're going to get it by
making another call mandatory. People should already be doing that, and
users should be reading the reports - I don't think the spec can help
that here.

For the most part, it's a reporting issue - they see the PE warning or
the crm_mon summary line and don't grok what they need to do to fix it.
That the error is coming from validate-all instead of start won't make
it easier for them; what they need is UI assistance. (i.e., better log
file querying to get a message as to what is actually wrong.)

Again, that's an implementation detail, not a specification one.

If we were talking about the spec, I'd rather add some way to return a
descriptive _string_ from the RA to the UI in some standard format so
that the UIs can display a meaningful one-line summary, instead of
rendering our machine-parseable rc.

(i.e., have the RA call a "ocf_describe" binary to set that before the
shell script returns, never mind how that is internally implemented for
now.)

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

andrew at beekhof

Jul 11, 2011, 4:38 AM

Post #29 of 31 (650 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Mon, Jul 11, 2011 at 9:19 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> On 2011-07-11T21:10:00, Andrew Beekhof <andrew [at] beekhof> wrote:
>
>> > Uh? No. validate-all makes perfect sense for an UI or so that wants to
>> > validate the resource agents parameters without actually starting or
>> > stopping it.
>> Okay. But you don't want "start calling validate-all" to be "best
>> practice", and code duplication is a bad thing... so what do you
>> propose?
>
> Well, as far as "validate-all" is concerned, I'dn't actually change that
> call at all, and not make it mandatory either.

best practice != mandatory.

I've no problem with it being in the dev guide not the spec.
Same way we recommend that many/most people put a monitor loop at the
end of the start action.

> But it would be awesome
> if the UIs actually called it - yet, that is beyond the specification, I
> see no need for it to be mandatory.

So the shell will start calling validate-all too?

>
> But like I proposed below in the mail here - we can increase the
> pressure a bit on validate-all actually being made use of by the UIs.
>
>> > Incorrect. They are at least _created_ equal. ;-) But indeed, they might
>> > not always check all the same things.
>> Which is important.
>> Its important that validate-all and start return appropriate error
>> messages instead of throwing up their hands and chucking out
>> OCF_ERR_GENERIC
>
> Right. But that's already required,

10 head
20 desk
30 add
40 goto 10

Yes, but you only get it if start calls validate-all or the checks are
duplicated.
You don't want the first and the second is error prone.

> and I doubt we're going to get it by
> making another call mandatory. People should already be doing that, and
> users should be reading the reports - I don't think the spec can help
> that here.
>
> For the most part, it's a reporting issue - they see the PE warning or
> the crm_mon summary line and don't grok what they need to do to fix it.
> That the error is coming from validate-all instead of start won't make
> it easier for them;

In my mind it will help split the reports into two piles - user error
vs. app error.
Knowing that is a good first step.

> what they need is UI assistance. (i.e., better log
> file querying to get a message as to what is actually wrong.)
>
> Again, that's an implementation detail, not a specification one.
>
>
> If we were talking about the spec, I'd rather add some way to return a
> descriptive _string_ from the RA to the UI in some standard format so
> that the UIs can display a meaningful one-line summary, instead of
> rendering our machine-parseable rc.
>
> (i.e., have the RA call a "ocf_describe" binary to set that before the
> shell script returns, never mind how that is internally implemented for
> now.)

Interesting idea.

>
>
> Regards,
> Lars
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
>
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

lmb at suse

Jul 12, 2011, 4:27 AM

Post #30 of 31 (647 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On 2011-07-11T21:38:11, Andrew Beekhof <andrew [at] beekhof> wrote:

> > Well, as far as "validate-all" is concerned, I'dn't actually change that
> > call at all, and not make it mandatory either.
> best practice != mandatory.

Right, if you want to call it "BCP", sure, go ahead.

> I've no problem with it being in the dev guide not the spec.
> Same way we recommend that many/most people put a monitor loop at the
> end of the start action.

Right; we insist that "start" must only return once the service is fully
up and operational, and for some service,s the loop is the easiest way
to achieve that.

> > But it would be awesome if the UIs actually called it - yet, that is
> > beyond the specification, I see no need for it to be mandatory.
> So the shell will start calling validate-all too?

Uhm, how should I know? It probably should, yes, but that's something to
discuss with Dejan, and not exactly relevant to the specification of
what "validate-all" does and what implementors must provide and can rely
on.

> >> Its important that validate-all and start return appropriate error
> >> messages instead of throwing up their hands and chucking out
> >> OCF_ERR_GENERIC
> > Right. But that's already required,
>
> 10 head
> 20 desk
> 30 add
> 40 goto 10

Ah, such constructive communication styles abound here! ;-)

> Yes, but you only get it if start calls validate-all or the checks are
> duplicated.
> You don't want the first and the second is error prone.

I didn't say I don't want start *internally* to the RA call
validate-all; clearly, start needs to verify its requirements. I just
don't want to call it before a start as a separate action, because that
is just stupid.

> In my mind it will help split the reports into two piles - user error
> vs. app error.

But that is already possible; nothing in the spec prevents that. If you
see "start" fail with OCF_ERR_CONFIGURED, OCF_ERR_ARGS,
OCF_ERR_INSTALLED those are quite likely "user errors".

A RA that doesn't return them but sends back "ERR_GENERIC" for
everything is an *implementation*, not a *specification* problem.

> > If we were talking about the spec, I'd rather add some way to return a
> > descriptive _string_ from the RA to the UI in some standard format so
> > that the UIs can display a meaningful one-line summary, instead of
> > rendering our machine-parseable rc.
> >
> > (i.e., have the RA call a "ocf_describe" binary to set that before the
> > shell script returns, never mind how that is internally implemented for
> > now.)
>
> Interesting idea.

Actually it might be very valuable, if the CIB status section can take
~80 chars per failure. Perhaps that's the way out of this discussion?
;-)

Regards,
Lars

--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

andrew at beekhof

Jul 12, 2011, 10:44 PM

Post #31 of 31 (644 views)
Permalink

Re: RA spec: explicit "probe" operation? [In reply to]

On Tue, Jul 12, 2011 at 9:27 PM, Lars Marowsky-Bree <lmb [at] suse> wrote:
> On 2011-07-11T21:38:11, Andrew Beekhof <andrew [at] beekhof> wrote:
>
>> > Well, as far as "validate-all" is concerned, I'dn't actually change that
>> > call at all, and not make it mandatory either.
>> best practice != mandatory.
>
> Right, if you want to call it "BCP", sure, go ahead.
>
>> I've no problem with it being in the dev guide not the spec.
>> Same way we recommend that many/most people put a monitor loop at the
>> end of the start action.
>
> Right; we insist that "start" must only return once the service is fully
> up and operational, and for some service,s the loop is the easiest way
> to achieve that.
>
>> > But it would be awesome if the UIs actually called it - yet, that is
>> > beyond the specification, I see no need for it to be mandatory.
>> So the shell will start calling validate-all too?
>
> Uhm, how should I know? It probably should, yes, but that's something to
> discuss with Dejan, and not exactly relevant to the specification of
> what "validate-all" does and what implementors must provide and can rely
> on.

Well it is.
Since your position was "let the management tools handle it", its
reasonable to ask how the most commonly used tool would do so ;-)

Its also not clear to me how pushing it into the GUIs solves the
problem that validate-all will only work when all resource
dependencies are available.
This is why I was pushing for the cluster (either directly or
indirectly as part of the RA's start action) to be the one to make the
call.

>
>> >> Its important that validate-all and start return appropriate error
>> >> messages instead of throwing up their hands and chucking out
>> >> OCF_ERR_GENERIC
>> > Right. But that's already required,
>>
>> 10 head
>> 20 desk
>> 30 add
>> 40 goto 10
>
> Ah, such constructive communication styles abound here! ;-)
>
>> Yes, but you only get it if start calls validate-all or the checks are
>> duplicated.
>> You don't want the first and the second is error prone.
>
> I didn't say I don't want start *internally* to the RA call
> validate-all;

Well, you did actually. That was the only reason I mentioned doing it
automatically.
But if you're happy with it now, then I think we have a path forward :-)

Florian: could you amend the dev guide to recommend start internally
calls validate-all first up?

> clearly, start needs to verify its requirements. I just
> don't want to call it before a start as a separate action, because that
> is just stupid.

ok.
I don't think its that big of a deal, but whatever :-)

>> In my mind it will help split the reports into two piles - user error
>> vs. app error.
>
> But that is already possible; nothing in the spec prevents that. If you
> see "start" fail with OCF_ERR_CONFIGURED, OCF_ERR_ARGS,
> OCF_ERR_INSTALLED those are quite likely "user errors".
>
> A RA that doesn't return them but sends back "ERR_GENERIC" for
> everything is an *implementation*, not a *specification* problem.
>
>> > If we were talking about the spec, I'd rather add some way to return a
>> > descriptive _string_ from the RA to the UI in some standard format so
>> > that the UIs can display a meaningful one-line summary, instead of
>> > rendering our machine-parseable rc.
>> >
>> > (i.e., have the RA call a "ocf_describe" binary to set that before the
>> > shell script returns, never mind how that is internally implemented for
>> > now.)
>>
>> Interesting idea.
>
> Actually it might be very valuable, if the CIB status section can take
> ~80 chars per failure. Perhaps that's the way out of this discussion?
> ;-)
>
>
> Regards,
> Lars
>
> --
> Architect Storage/HA, OPS Engineering, Novell, Inc.
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
> _______________________________________________
> ha-wg-technical mailing list
> ha-wg-technical [at] lists
> https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical
>
_______________________________________________
ha-wg-technical mailing list
ha-wg-technical [at] lists
https://lists.linux-foundation.org/mailman/listinfo/ha-wg-technical

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads