Mailing List Archive: Linux-HA: Dev

Order Constraints and Clone Groups

Index | Next | Previous | View Threaded

simont at nse

Sep 5, 2008, 6:08 AM

Post #1 of 8 (1484 views)
Permalink

Order Constraints and Clone Groups

Hi All,

Before I dig any deeper looking for the root cause, I thought I would
run a scenario past the list to make sure I was not missing something
obvious that has already been addresses/fixed.

Take the following scenario:

Clariion iSCSI SAN
Two Hosts Connected
SLES10 SP2 Heartbeat + OCFS2 + Xen, Identical configuration

We have OCFS2 configured as standard with user mode cluster control from
heartbeat and all working fine.

Next we have Xen VM Instances reading their configurations from the
Clustered OCFS2 Filesystem and running against physical volumes
presented (shared) from the SAN, live migration etc.

Again all fine

So effectively we have:
an anonymous clone group running stonith (external/SSH)
an anonymous clone group running the OCFS2 configuration store
and a normal VM resource for each XEN Virtual machine

The problem comes with the specific interactions of the order
constraints.

We have a simple order constraint saying the XEN VM instance depends
upon the OCFS2 configuration store

With both nodes running, we can live migrate the Xen VM between the
hosts as we like and all works fine, but the problem starts if for
example we perform the following:

Start up VM, Let us say it starts on
Live Migrate a VM from Host A to Host B
Instruct Host A to go into standby

At this point, the Xen VM received a shutdown command, even though the
instruction to put Host A into standby has absolutely nothing to do with
Host B or the OCFS2 Clone instance on host B.

The Xen VM on B shuts down and then immediately restarts again on the
same host (B)

If I then take Host A back out of standby and make it active, again the
Xen host is shut down and re-started (again on the same host)

If I take the order constraint away from the configuration, the
behaviour does not happen and host A can be taken into standby correctly
without affecting host B or the running VM.

My feeling is that something in the order clause logic is getting
confused and thinking that the OCFS2 clone on Host B is shutting down,
which causes the VM to be shut down, then the CRM realises that the
OCFS2 clone is alive and well and starts the VM back up again.

Does this ring any bells, or should I start digging.

Config as follows:

<cib generated="true" admin_epoch="0" have_quorum="true"
ignore_dtd="false" num_peers="2" cib_feature_revision="2.0"
crm_feature_set="2.0" epoch="244" num_updates="17" cib-last-written="Fri
Sep 5 08:22:13 2008" ccm_transition="2"
dc_uuid="c6fbe051-effc-47a9-b644-706eca9aa94a">
<configuration>
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
<attributes>
<nvpair id="cib-bootstrap-options-dc-version"
name="dc-version" value="2.1.3-node:
a3184d5240c6e7032aef9cce6e5b7752ded544b3"/>
<nvpair name="last-lrm-refresh"
id="cib-bootstrap-options-last-lrm-refresh" value="1220617910"/>
</attributes>
</cluster_property_set>
<cluster_property_set id="cibbootstrap">
<attributes>
<nvpair id="cibbootstrap-01" name="transition-idle-timeout"
value="60"/>
<nvpair id="cibbootstrap-02"
name="default-resource-stickiness" value="INFINITY"/>
<nvpair id="cibbootstrap-03"
name="default-resource-failure-stickiness" value="-500"/>
<nvpair id="cibbootstrap-04" name="stonith-enabled"
value="true"/>
<nvpair id="cibbootstrap-05" name="stonith-action"
value="reboot"/>
<nvpair id="cibbootstrap-06" name="symmetric-cluster"
value="true"/>
<nvpair id="cibbootstrap-07" name="no-quorum-policy"
value="stop"/>
<nvpair id="cibbootstrap-08" name="stop-orphan-resources"
value="true"/>
<nvpair id="cibbootstrap-09" name="stop-orphan-actions"
value="true"/>
<nvpair id="cibbootstrap-10" name="is-managed-default"
value="true"/>
</attributes>
</cluster_property_set>
</crm_config>
<nodes>
<node uname="hex-xen3" type="normal"
id="c6fbe051-effc-47a9-b644-706eca9aa94a">
<instance_attributes
id="nodes-c6fbe051-effc-47a9-b644-706eca9aa94a">
<attributes>
<nvpair name="standby"
id="standby-c6fbe051-effc-47a9-b644-706eca9aa94a" value="off"/>
</attributes>
</instance_attributes>
</node>
<node id="a1db61ed-5363-465f-baa7-8d8dff6e1eb3" uname="hex-xen4"
type="normal">
<instance_attributes
id="nodes-a1db61ed-5363-465f-baa7-8d8dff6e1eb3">
<attributes>
<nvpair id="standby-a1db61ed-5363-465f-baa7-8d8dff6e1eb3"
name="standby" value="off"/>
</attributes>
</instance_attributes>
</node>
</nodes>
<resources>
<clone id="stonithcloneset" globally_unique="false">
<instance_attributes id="stonithcloneset">
<attributes>
<nvpair id="stonithcloneset-01" name="clone_node_max"
value="1"/>
</attributes>
</instance_attributes>
<primitive id="stonithclone" class="stonith"
type="external/ssh" provider="heartbeat">
<operations>
<op name="monitor" interval="5s" timeout="20s"
prereq="nothing" id="stonithclone-op-01"/>
<op name="start" timeout="20s" prereq="nothing"
id="stonithclone-op-02"/>
</operations>
<instance_attributes id="stonithclone">
<attributes>
<nvpair id="stonithclone-01" name="hostlist"
value="hex-xen3,hex-xen4"/>
</attributes>
</instance_attributes>
<meta_attributes id="stonithclone:1_meta_attrs">
<attributes>
<nvpair id="stonithclone:1_metaattr_target_role"
name="target_role" value="started"/>
</attributes>
</meta_attributes>
</primitive>
</clone>
<clone id="configstorecloneset" notify="true"
globally_unique="false">
<instance_attributes id="configstorecloneset">
<attributes>
<nvpair id="configstorecloneset-01" name="clone_node_max"
value="1"/>
<nvpair id="configstorecloneset-02" name="target_role"
value="started"/>
</attributes>
</instance_attributes>
<primitive id="configstoreclone" class="ocf" type="Filesystem"
provider="heartbeat">
<operations>
<op name="monitor" interval="20s" timeout="60s"
prereq="nothing" id="configstoreclone-op-01"/>
</operations>
<instance_attributes id="configstoreclone">
<attributes>
<nvpair id="configstoreclone-01" name="device"
value="/dev/emcpowerd"/>
<nvpair id="configstoreclone-02" name="directory"
value="/mnt/configstore"/>
<nvpair id="configstoreclone-03" name="fstype"
value="ocfs2"/>
</attributes>
</instance_attributes>
</primitive>
</clone>
<clone id="imagestorecloneset" notify="true"
globally_unique="false">
<instance_attributes id="imagestorecloneset">
<attributes>
<nvpair id="imagestorecloneset-01" name="clone_node_max"
value="1"/>
<nvpair id="imagestorecloneset-02" name="target_role"
value="started"/>
</attributes>
</instance_attributes>
<primitive id="imagestoreclone" class="ocf" type="Filesystem"
provider="heartbeat">
<operations>
<op name="monitor" interval="20s" timeout="60s"
prereq="nothing" id="imagestoreclone-op-01" on_fail="restart"/>
</operations>
<instance_attributes id="imagestoreclone">
<attributes>
<nvpair id="imagestoreclone-01" name="device"
value="/dev/emcpowerb"/>
<nvpair id="imagestoreclone-02" name="directory"
value="/mnt/imagestore"/>
<nvpair id="imagestoreclone-03" name="fstype"
value="ocfs2"/>
</attributes>
</instance_attributes>
</primitive>
</clone>
<primitive id="jsmx8" class="ocf" type="Xen"
provider="heartbeat">
<operations>
<op id="jsmx8-op-01" name="monitor" interval="10s"
timeout="60s" prereq="nothing"/>
<op id="jsmx8-op-02" name="start" timeout="60s"
start_delay="0"/>
<op id="jsmx8-op-03" name="stop" timeout="300s"/>
</operations>
<instance_attributes id="jsmx8">
<attributes>
<nvpair id="jsmx8-attr-01" name="xmfile"
value="/etc/xen/vm/jsmx8"/>
<nvpair id="jsmx8-attr-02" name="target_role"
value="started"/>
</attributes>
</instance_attributes>
<meta_attributes id="jsmx8-meta-01">
<attributes>
<nvpair id="jsmx8-meta-attr-01" name="allow_migrate"
value="true"/>
</attributes>
</meta_attributes>
</primitive>
</resources>
<constraints>
<rsc_location id="location_imagestorecloneset"
rsc="imagestorecloneset">
<rule id="prefered_location_imagestorecloneset" score="1500"
boolean_op="or">
<expression attribute="#uname"
id="5c68c5e7-40db-4fa1-b030-ca0adeee3210" operation="eq"
value="hex-xen3"/>
<expression attribute="#uname"
id="8fc08537-d47b-4186-9489-5eaf74accd6d" operation="eq"
value="hex-xen4"/>
</rule>
</rsc_location>
<rsc_location id="location_stonithcloneset"
rsc="stonithcloneset">
<rule id="prefered_location_stonithcloneset" score="1500"
boolean_op="or">
<expression attribute="#uname"
id="7c4839b5-52e2-4dba-a637-ab3dc5d238be" operation="eq"
value="hex-xen3"/>
<expression attribute="#uname"
id="d5221c98-a8f4-4c7b-bc26-f0700df3383a" operation="eq"
value="hex-xen4"/>
</rule>
</rsc_location>
<rsc_location id="location_configstorecloneset"
rsc="configstorecloneset">
<rule id="prefered_location_configstorecloneset" score="1500"
boolean_op="or">
<expression attribute="#uname"
id="d720af75-fa6b-4538-b9ce-4a2feb855449" operation="eq"
value="hex-xen3"/>
<expression attribute="#uname"
id="145fa052-1ef1-4504-ab9c-af14212ae4bb" operation="eq"
value="hex-xen4"/>
</rule>
</rsc_location>
<rsc_location id="location_jsmx8" rsc="jsmx8">
<rule id="prefered_location_jsmx8" score="1500"
boolean_op="or">
<expression attribute="#uname"
id="5427def3-eec8-41b8-8a87-ea1a42a62e0c" operation="eq"
value="hex-xen4"/>
<expression attribute="#uname"
id="48998826-fcc9-4cd2-9226-a61d5cb45e29" operation="eq"
value="hex-xen3"/>
</rule>
</rsc_location>
<rsc_order id="jsmx8-orderconstraints-01" from="jsmx8"
to="configstorecloneset"/>
<rsc_location id="cli-prefer-jsmx8" rsc="jsmx8">
<rule id="prefered_cli-prefer-jsmx8" score="INFINITY">
<expression attribute="#uname"
id="4a0dd6c5-4c96-4bfb-acee-4a65d0aeace0" operation="eq"
value="hex-xen4"/>
</rule>
</rsc_location>
<rsc_location id="cli-standby-jsmx8" rsc="jsmx8">
<rule id="prefered_cli-standby-jsmx8" score="-INFINITY">
<expression attribute="#uname"
id="0e17d595-858a-4fae-b18d-35a0efeb5a38" operation="eq"
value="hex-xen3"/>
</rule>
</rsc_location>
</constraints>
</configuration>
<status>
<node_state id="c6fbe051-effc-47a9-b644-706eca9aa94a"
uname="hex-xen3" crmd="online" crm-debug-origin="do_update_resource"
shutdown="0" in_ccm="true" ha="active" join="member" expected="member">
<transient_attributes id="c6fbe051-effc-47a9-b644-706eca9aa94a">
<instance_attributes
id="status-c6fbe051-effc-47a9-b644-706eca9aa94a">
<attributes>
<nvpair
id="status-c6fbe051-effc-47a9-b644-706eca9aa94a-probe_complete"
name="probe_complete" value="true"/>
</attributes>
</instance_attributes>
</transient_attributes>
<lrm id="c6fbe051-effc-47a9-b644-706eca9aa94a">
<lrm_resources>
<lrm_resource id="configstoreclone:1" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="configstoreclone:1_start_0"
operation="start" crm-debug-origin="do_update_resource"
transition_key="19:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;19:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="39" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:1_post_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="59:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;59:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="42" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:1_pre_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="59:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;59:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="33" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:1_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="21:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;21:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="36" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:1_monitor_20000"
operation="monitor" crm-debug-origin="do_update_resource"
transition_key="20:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;20:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="44" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="20000" op_digest="a15361c685ca5ba0171501813da8f5ee"/>
</lrm_resource>
<lrm_resource id="imagestoreclone:1" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="imagestoreclone:1_start_0"
operation="start" crm-debug-origin="do_update_resource"
transition_key="35:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;35:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="40" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:1_post_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="64:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;64:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="43" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:1_pre_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="64:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;64:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="34" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:1_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="36:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;36:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="35" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:1_monitor_20000"
operation="monitor" crm-debug-origin="do_update_resource"
transition_key="36:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;36:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="45" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="20000" op_digest="3655514ab43f26ed611a7c2f7d92e07a"/>
</lrm_resource>
<lrm_resource id="stonithclone:0" type="external/ssh"
class="stonith" provider="heartbeat">
<lrm_rsc_op id="stonithclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="4:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;4:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="2" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:0_start_0" operation="start"
crm-debug-origin="do_update_resource"
transition_key="9:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;9:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="38" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:0_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="12:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;12:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="37" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:0_monitor_5000"
operation="monitor" crm-debug-origin="do_update_resource"
transition_key="10:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;10:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="41" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="5000" op_digest="98a40cf66e5b74139d0b0203d76833f5"/>
</lrm_resource>
<lrm_resource id="jsmx8" type="Xen" class="ocf"
provider="heartbeat">
<lrm_rsc_op id="jsmx8_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs"
transition_key="10:23:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;10:23:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="26" crm_feature_set="2.0" rc_code="7" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_start_0" operation="start"
crm-debug-origin="do_update_resource"
transition_key="53:34:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;53:34:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="30" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="52:40:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;52:40:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="32" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
</lrm_resource>
<lrm_resource id="configstoreclone:0" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="configstoreclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="5:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;5:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="3" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_start_0"
operation="start" crm-debug-origin="build_active_RAs"
transition_key="11:1:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:1;11:1:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="8" crm_feature_set="2.0" rc_code="1" op_status="0" interval="0"
op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_pre_notify_stop_0"
operation="notify" crm-debug-origin="build_active_RAs"
transition_key="47:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;47:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="12" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_stop_0" operation="stop"
crm-debug-origin="build_active_RAs"
transition_key="3:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;3:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="15" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
</lrm_resource>
<lrm_resource id="imagestoreclone:0" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="imagestoreclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="6:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;6:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="4" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_start_0"
operation="start" crm-debug-origin="build_active_RAs"
transition_key="25:1:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:1;25:1:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="9" crm_feature_set="2.0" rc_code="1" op_status="0" interval="0"
op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_pre_notify_stop_0"
operation="notify" crm-debug-origin="build_active_RAs"
transition_key="48:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;48:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="13" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_stop_0" operation="stop"
crm-debug-origin="build_active_RAs"
transition_key="2:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;2:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="14" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
</lrm_resource>
</lrm_resources>
</lrm>
</node_state>
<node_state id="a1db61ed-5363-465f-baa7-8d8dff6e1eb3"
uname="hex-xen4" ha="active" in_ccm="true" crmd="online" join="member"
expected="member" crm-debug-origin="do_update_resource" shutdown="0">
<transient_attributes id="a1db61ed-5363-465f-baa7-8d8dff6e1eb3">
<instance_attributes
id="status-a1db61ed-5363-465f-baa7-8d8dff6e1eb3">
<attributes>
<nvpair
id="status-a1db61ed-5363-465f-baa7-8d8dff6e1eb3-probe_complete"
name="probe_complete" value="true"/>
</attributes>
</instance_attributes>
</transient_attributes>
<lrm id="a1db61ed-5363-465f-baa7-8d8dff6e1eb3">
<lrm_resources>
<lrm_resource id="stonithclone:1" type="external/ssh"
class="stonith" provider="heartbeat">
<lrm_rsc_op id="stonithclone:1_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="8:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;8:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="2" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:1_start_0" operation="start"
crm-debug-origin="build_active_RAs"
transition_key="14:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;14:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="7" crm_feature_set="2.0" rc_code="0" op_status="0" interval="0"
op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:1_monitor_5000"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="13:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;13:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="12" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="5000" op_digest="98a40cf66e5b74139d0b0203d76833f5"/>
</lrm_resource>
<lrm_resource id="configstoreclone:0" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="configstoreclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="9:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;9:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="3" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_start_0"
operation="start" crm-debug-origin="build_active_RAs"
transition_key="20:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;20:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="8" crm_feature_set="2.0" rc_code="0" op_status="0" interval="0"
op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_post_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="56:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;56:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="32" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_monitor_20000"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="20:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;20:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="13" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="20000" op_digest="a15361c685ca5ba0171501813da8f5ee"/>
<lrm_rsc_op id="configstoreclone:0_pre_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="57:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;57:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="23" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_post_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="58:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;58:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="28" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_pre_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="55:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;55:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="30" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
</lrm_resource>
<lrm_resource id="imagestoreclone:0" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="imagestoreclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="10:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;10:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="4" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_start_0"
operation="start" crm-debug-origin="build_active_RAs"
transition_key="36:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;36:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="9" crm_feature_set="2.0" rc_code="0" op_status="0" interval="0"
op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_post_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="61:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;61:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="33" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_monitor_20000"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="37:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;37:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="14" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="20000" op_digest="3655514ab43f26ed611a7c2f7d92e07a"/>
<lrm_rsc_op id="imagestoreclone:0_pre_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="62:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;62:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="24" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_post_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="63:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;63:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="25" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_pre_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="60:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;60:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="31" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
</lrm_resource>
<lrm_resource id="jsmx8" type="Xen" class="ocf"
provider="heartbeat">
<lrm_rsc_op id="jsmx8_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource"
transition_key="11:24:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;11:24:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="16" crm_feature_set="2.0" rc_code="7" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_start_0" operation="start"
crm-debug-origin="do_update_resource"
transition_key="50:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;50:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="34" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="49:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;49:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="29" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_monitor_10000" operation="monitor"
crm-debug-origin="do_update_resource"
transition_key="4:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;4:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="35" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="10000" op_digest="0a29de1107eccb4728e9fd1fdd3d8471"/>
</lrm_resource>
</lrm_resources>
</lrm>
</node_state>
</status>
</cib>

Simon Talbot MEng, ACGI
(Chief Engineer)
Tel: 020 3161 6001
Fax: 020 3161 6011

*** Voxige Wholesale Voice http://www.voxige.com ***

The information contained in this e-mail and any attachments are private

and confidential and may be legally privileged.

It is intended for the named addressee(s) only. If you are not the
intended recipient(s), you must not read, copy or use the information
contained in any way. If you receive this email or any attachments in
error, please notify us immediately by e-mail and destroy any copy you
have of it.

We accept no responsibility for any loss or damages whatsoever arising
in any way from receipt or use of this e-mail or any attachments. This
e-mail is not intended to create legally binding commitments on our
behalf, nor do its comments reflect our corporate views or policies.

_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

simont at nse

Sep 9, 2008, 4:38 PM

Post #2 of 8 (1384 views)
Permalink

Order Constraints and Clone Groups [In reply to]

Did anyone have any observations/pointers on where to start looking for
this problem?

Hi All,

Before I dig any deeper looking for the root cause, I thought I would
run a scenario past the list to make sure I was not missing something
obvious that has already been addresses/fixed.

Take the following scenario:

Clariion iSCSI SAN
Two Hosts Connected
SLES10 SP2 Heartbeat + OCFS2 + Xen, Identical configuration

We have OCFS2 configured as standard with user mode cluster control from
heartbeat and all working fine.

Next we have Xen VM Instances reading their configurations from the
Clustered OCFS2 Filesystem and running against physical volumes
presented (shared) from the SAN, live migration etc.

Again all fine

So effectively we have:
an anonymous clone group running stonith (external/SSH)
an anonymous clone group running the OCFS2 configuration store
and a normal VM resource for each XEN Virtual machine

The problem comes with the specific interactions of the order
constraints.

We have a simple order constraint saying the XEN VM instance depends
upon the OCFS2 configuration store

With both nodes running, we can live migrate the Xen VM between the
hosts as we like and all works fine, but the problem starts if for
example we perform the following:

Start up VM, Let us say it starts on
Live Migrate a VM from Host A to Host B
Instruct Host A to go into standby

At this point, the Xen VM received a shutdown command, even though the
instruction to put Host A into standby has absolutely nothing to do with
Host B or the OCFS2 Clone instance on host B.

The Xen VM on B shuts down and then immediately restarts again on the
same host (B)

If I then take Host A back out of standby and make it active, again the
Xen host is shut down and re-started (again on the same host)

If I take the order constraint away from the configuration, the
behaviour does not happen and host A can be taken into standby correctly
without affecting host B or the running VM.

My feeling is that something in the order clause logic is getting
confused and thinking that the OCFS2 clone on Host B is shutting down,
which causes the VM to be shut down, then the CRM realises that the
OCFS2 clone is alive and well and starts the VM back up again.

Does this ring any bells, or should I start digging.

Config as follows:

<cib generated="true" admin_epoch="0" have_quorum="true"
ignore_dtd="false" num_peers="2" cib_feature_revision="2.0"
crm_feature_set="2.0" epoch="244" num_updates="17" cib-last-written="Fri
Sep 5 08:22:13 2008" ccm_transition="2"
dc_uuid="c6fbe051-effc-47a9-b644-706eca9aa94a">
<configuration>
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
<attributes>
<nvpair id="cib-bootstrap-options-dc-version"
name="dc-version" value="2.1.3-node:
a3184d5240c6e7032aef9cce6e5b7752ded544b3"/>
<nvpair name="last-lrm-refresh"
id="cib-bootstrap-options-last-lrm-refresh" value="1220617910"/>
</attributes>
</cluster_property_set>
<cluster_property_set id="cibbootstrap">
<attributes>
<nvpair id="cibbootstrap-01" name="transition-idle-timeout"
value="60"/>
<nvpair id="cibbootstrap-02"
name="default-resource-stickiness" value="INFINITY"/>
<nvpair id="cibbootstrap-03"
name="default-resource-failure-stickiness" value="-500"/>
<nvpair id="cibbootstrap-04" name="stonith-enabled"
value="true"/>
<nvpair id="cibbootstrap-05" name="stonith-action"
value="reboot"/>
<nvpair id="cibbootstrap-06" name="symmetric-cluster"
value="true"/>
<nvpair id="cibbootstrap-07" name="no-quorum-policy"
value="stop"/>
<nvpair id="cibbootstrap-08" name="stop-orphan-resources"
value="true"/>
<nvpair id="cibbootstrap-09" name="stop-orphan-actions"
value="true"/>
<nvpair id="cibbootstrap-10" name="is-managed-default"
value="true"/>
</attributes>
</cluster_property_set>
</crm_config>
<nodes>
<node uname="hex-xen3" type="normal"
id="c6fbe051-effc-47a9-b644-706eca9aa94a">
<instance_attributes
id="nodes-c6fbe051-effc-47a9-b644-706eca9aa94a">
<attributes>
<nvpair name="standby"
id="standby-c6fbe051-effc-47a9-b644-706eca9aa94a" value="off"/>
</attributes>
</instance_attributes>
</node>
<node id="a1db61ed-5363-465f-baa7-8d8dff6e1eb3" uname="hex-xen4"
type="normal">
<instance_attributes
id="nodes-a1db61ed-5363-465f-baa7-8d8dff6e1eb3">
<attributes>
<nvpair id="standby-a1db61ed-5363-465f-baa7-8d8dff6e1eb3"
name="standby" value="off"/>
</attributes>
</instance_attributes>
</node>
</nodes>
<resources>
<clone id="stonithcloneset" globally_unique="false">
<instance_attributes id="stonithcloneset">
<attributes>
<nvpair id="stonithcloneset-01" name="clone_node_max"
value="1"/>
</attributes>
</instance_attributes>
<primitive id="stonithclone" class="stonith"
type="external/ssh" provider="heartbeat">
<operations>
<op name="monitor" interval="5s" timeout="20s"
prereq="nothing" id="stonithclone-op-01"/>
<op name="start" timeout="20s" prereq="nothing"
id="stonithclone-op-02"/>
</operations>
<instance_attributes id="stonithclone">
<attributes>
<nvpair id="stonithclone-01" name="hostlist"
value="hex-xen3,hex-xen4"/>
</attributes>
</instance_attributes>
<meta_attributes id="stonithclone:1_meta_attrs">
<attributes>
<nvpair id="stonithclone:1_metaattr_target_role"
name="target_role" value="started"/>
</attributes>
</meta_attributes>
</primitive>
</clone>
<clone id="configstorecloneset" notify="true"
globally_unique="false">
<instance_attributes id="configstorecloneset">
<attributes>
<nvpair id="configstorecloneset-01" name="clone_node_max"
value="1"/>
<nvpair id="configstorecloneset-02" name="target_role"
value="started"/>
</attributes>
</instance_attributes>
<primitive id="configstoreclone" class="ocf" type="Filesystem"
provider="heartbeat">
<operations>
<op name="monitor" interval="20s" timeout="60s"
prereq="nothing" id="configstoreclone-op-01"/>
</operations>
<instance_attributes id="configstoreclone">
<attributes>
<nvpair id="configstoreclone-01" name="device"
value="/dev/emcpowerd"/>
<nvpair id="configstoreclone-02" name="directory"
value="/mnt/configstore"/>
<nvpair id="configstoreclone-03" name="fstype"
value="ocfs2"/>
</attributes>
</instance_attributes>
</primitive>
</clone>
<clone id="imagestorecloneset" notify="true"
globally_unique="false">
<instance_attributes id="imagestorecloneset">
<attributes>
<nvpair id="imagestorecloneset-01" name="clone_node_max"
value="1"/>
<nvpair id="imagestorecloneset-02" name="target_role"
value="started"/>
</attributes>
</instance_attributes>
<primitive id="imagestoreclone" class="ocf" type="Filesystem"
provider="heartbeat">
<operations>
<op name="monitor" interval="20s" timeout="60s"
prereq="nothing" id="imagestoreclone-op-01" on_fail="restart"/>
</operations>
<instance_attributes id="imagestoreclone">
<attributes>
<nvpair id="imagestoreclone-01" name="device"
value="/dev/emcpowerb"/>
<nvpair id="imagestoreclone-02" name="directory"
value="/mnt/imagestore"/>
<nvpair id="imagestoreclone-03" name="fstype"
value="ocfs2"/>
</attributes>
</instance_attributes>
</primitive>
</clone>
<primitive id="jsmx8" class="ocf" type="Xen"
provider="heartbeat">
<operations>
<op id="jsmx8-op-01" name="monitor" interval="10s"
timeout="60s" prereq="nothing"/>
<op id="jsmx8-op-02" name="start" timeout="60s"
start_delay="0"/>
<op id="jsmx8-op-03" name="stop" timeout="300s"/>
</operations>
<instance_attributes id="jsmx8">
<attributes>
<nvpair id="jsmx8-attr-01" name="xmfile"
value="/etc/xen/vm/jsmx8"/>
<nvpair id="jsmx8-attr-02" name="target_role"
value="started"/>
</attributes>
</instance_attributes>
<meta_attributes id="jsmx8-meta-01">
<attributes>
<nvpair id="jsmx8-meta-attr-01" name="allow_migrate"
value="true"/>
</attributes>
</meta_attributes>
</primitive>
</resources>
<constraints>
<rsc_location id="location_imagestorecloneset"
rsc="imagestorecloneset">
<rule id="prefered_location_imagestorecloneset" score="1500"
boolean_op="or">
<expression attribute="#uname"
id="5c68c5e7-40db-4fa1-b030-ca0adeee3210" operation="eq"
value="hex-xen3"/>
<expression attribute="#uname"
id="8fc08537-d47b-4186-9489-5eaf74accd6d" operation="eq"
value="hex-xen4"/>
</rule>
</rsc_location>
<rsc_location id="location_stonithcloneset"
rsc="stonithcloneset">
<rule id="prefered_location_stonithcloneset" score="1500"
boolean_op="or">
<expression attribute="#uname"
id="7c4839b5-52e2-4dba-a637-ab3dc5d238be" operation="eq"
value="hex-xen3"/>
<expression attribute="#uname"
id="d5221c98-a8f4-4c7b-bc26-f0700df3383a" operation="eq"
value="hex-xen4"/>
</rule>
</rsc_location>
<rsc_location id="location_configstorecloneset"
rsc="configstorecloneset">
<rule id="prefered_location_configstorecloneset" score="1500"
boolean_op="or">
<expression attribute="#uname"
id="d720af75-fa6b-4538-b9ce-4a2feb855449" operation="eq"
value="hex-xen3"/>
<expression attribute="#uname"
id="145fa052-1ef1-4504-ab9c-af14212ae4bb" operation="eq"
value="hex-xen4"/>
</rule>
</rsc_location>
<rsc_location id="location_jsmx8" rsc="jsmx8">
<rule id="prefered_location_jsmx8" score="1500"
boolean_op="or">
<expression attribute="#uname"
id="5427def3-eec8-41b8-8a87-ea1a42a62e0c" operation="eq"
value="hex-xen4"/>
<expression attribute="#uname"
id="48998826-fcc9-4cd2-9226-a61d5cb45e29" operation="eq"
value="hex-xen3"/>
</rule>
</rsc_location>
<rsc_order id="jsmx8-orderconstraints-01" from="jsmx8"
to="configstorecloneset"/>
<rsc_location id="cli-prefer-jsmx8" rsc="jsmx8">
<rule id="prefered_cli-prefer-jsmx8" score="INFINITY">
<expression attribute="#uname"
id="4a0dd6c5-4c96-4bfb-acee-4a65d0aeace0" operation="eq"
value="hex-xen4"/>
</rule>
</rsc_location>
<rsc_location id="cli-standby-jsmx8" rsc="jsmx8">
<rule id="prefered_cli-standby-jsmx8" score="-INFINITY">
<expression attribute="#uname"
id="0e17d595-858a-4fae-b18d-35a0efeb5a38" operation="eq"
value="hex-xen3"/>
</rule>
</rsc_location>
</constraints>
</configuration>
<status>
<node_state id="c6fbe051-effc-47a9-b644-706eca9aa94a"
uname="hex-xen3" crmd="online" crm-debug-origin="do_update_resource"
shutdown="0" in_ccm="true" ha="active" join="member" expected="member">
<transient_attributes id="c6fbe051-effc-47a9-b644-706eca9aa94a">
<instance_attributes
id="status-c6fbe051-effc-47a9-b644-706eca9aa94a">
<attributes>
<nvpair
id="status-c6fbe051-effc-47a9-b644-706eca9aa94a-probe_complete"
name="probe_complete" value="true"/>
</attributes>
</instance_attributes>
</transient_attributes>
<lrm id="c6fbe051-effc-47a9-b644-706eca9aa94a">
<lrm_resources>
<lrm_resource id="configstoreclone:1" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="configstoreclone:1_start_0"
operation="start" crm-debug-origin="do_update_resource"
transition_key="19:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;19:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="39" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:1_post_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="59:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;59:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="42" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:1_pre_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="59:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;59:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="33" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:1_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="21:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;21:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="36" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:1_monitor_20000"
operation="monitor" crm-debug-origin="do_update_resource"
transition_key="20:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;20:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="44" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="20000" op_digest="a15361c685ca5ba0171501813da8f5ee"/>
</lrm_resource>
<lrm_resource id="imagestoreclone:1" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="imagestoreclone:1_start_0"
operation="start" crm-debug-origin="do_update_resource"
transition_key="35:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;35:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="40" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:1_post_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="64:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;64:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="43" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:1_pre_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="64:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;64:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="34" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:1_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="36:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;36:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="35" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:1_monitor_20000"
operation="monitor" crm-debug-origin="do_update_resource"
transition_key="36:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;36:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="45" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="20000" op_digest="3655514ab43f26ed611a7c2f7d92e07a"/>
</lrm_resource>
<lrm_resource id="stonithclone:0" type="external/ssh"
class="stonith" provider="heartbeat">
<lrm_rsc_op id="stonithclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="4:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;4:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="2" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:0_start_0" operation="start"
crm-debug-origin="do_update_resource"
transition_key="9:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;9:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="38" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:0_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="12:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;12:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="37" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:0_monitor_5000"
operation="monitor" crm-debug-origin="do_update_resource"
transition_key="10:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;10:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="41" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="5000" op_digest="98a40cf66e5b74139d0b0203d76833f5"/>
</lrm_resource>
<lrm_resource id="jsmx8" type="Xen" class="ocf"
provider="heartbeat">
<lrm_rsc_op id="jsmx8_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs"
transition_key="10:23:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;10:23:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="26" crm_feature_set="2.0" rc_code="7" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_start_0" operation="start"
crm-debug-origin="do_update_resource"
transition_key="53:34:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;53:34:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="30" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="52:40:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;52:40:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="32" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
</lrm_resource>
<lrm_resource id="configstoreclone:0" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="configstoreclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="5:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;5:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="3" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_start_0"
operation="start" crm-debug-origin="build_active_RAs"
transition_key="11:1:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:1;11:1:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="8" crm_feature_set="2.0" rc_code="1" op_status="0" interval="0"
op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_pre_notify_stop_0"
operation="notify" crm-debug-origin="build_active_RAs"
transition_key="47:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;47:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="12" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_stop_0" operation="stop"
crm-debug-origin="build_active_RAs"
transition_key="3:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;3:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="15" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
</lrm_resource>
<lrm_resource id="imagestoreclone:0" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="imagestoreclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="6:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;6:0:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="4" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_start_0"
operation="start" crm-debug-origin="build_active_RAs"
transition_key="25:1:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:1;25:1:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="9" crm_feature_set="2.0" rc_code="1" op_status="0" interval="0"
op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_pre_notify_stop_0"
operation="notify" crm-debug-origin="build_active_RAs"
transition_key="48:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;48:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="13" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_stop_0" operation="stop"
crm-debug-origin="build_active_RAs"
transition_key="2:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;2:2:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="14" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
</lrm_resource>
</lrm_resources>
</lrm>
</node_state>
<node_state id="a1db61ed-5363-465f-baa7-8d8dff6e1eb3"
uname="hex-xen4" ha="active" in_ccm="true" crmd="online" join="member"
expected="member" crm-debug-origin="do_update_resource" shutdown="0">
<transient_attributes id="a1db61ed-5363-465f-baa7-8d8dff6e1eb3">
<instance_attributes
id="status-a1db61ed-5363-465f-baa7-8d8dff6e1eb3">
<attributes>
<nvpair
id="status-a1db61ed-5363-465f-baa7-8d8dff6e1eb3-probe_complete"
name="probe_complete" value="true"/>
</attributes>
</instance_attributes>
</transient_attributes>
<lrm id="a1db61ed-5363-465f-baa7-8d8dff6e1eb3">
<lrm_resources>
<lrm_resource id="stonithclone:1" type="external/ssh"
class="stonith" provider="heartbeat">
<lrm_rsc_op id="stonithclone:1_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="8:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;8:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="2" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:1_start_0" operation="start"
crm-debug-origin="build_active_RAs"
transition_key="14:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;14:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="7" crm_feature_set="2.0" rc_code="0" op_status="0" interval="0"
op_digest="8c0ca4101f0e75e8b1e817edcb6a2584"/>
<lrm_rsc_op id="stonithclone:1_monitor_5000"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="13:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;13:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="12" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="5000" op_digest="98a40cf66e5b74139d0b0203d76833f5"/>
</lrm_resource>
<lrm_resource id="configstoreclone:0" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="configstoreclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="9:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;9:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="3" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_start_0"
operation="start" crm-debug-origin="build_active_RAs"
transition_key="20:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;20:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="8" crm_feature_set="2.0" rc_code="0" op_status="0" interval="0"
op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_post_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="56:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;56:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="32" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_monitor_20000"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="20:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;20:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="13" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="20000" op_digest="a15361c685ca5ba0171501813da8f5ee"/>
<lrm_rsc_op id="configstoreclone:0_pre_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="57:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;57:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="23" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_post_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="58:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;58:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="28" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
<lrm_rsc_op id="configstoreclone:0_pre_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="55:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;55:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="30" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="c87214085bda47fb697367dd3d17956a"/>
</lrm_resource>
<lrm_resource id="imagestoreclone:0" type="Filesystem"
class="ocf" provider="heartbeat">
<lrm_rsc_op id="imagestoreclone:0_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="10:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;10:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="4" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0"
op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_start_0"
operation="start" crm-debug-origin="build_active_RAs"
transition_key="36:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;36:11:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="9" crm_feature_set="2.0" rc_code="0" op_status="0" interval="0"
op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_post_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="61:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;61:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="33" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_monitor_20000"
operation="monitor" crm-debug-origin="build_active_RAs"
transition_key="37:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;37:12:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="14" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="20000" op_digest="3655514ab43f26ed611a7c2f7d92e07a"/>
<lrm_rsc_op id="imagestoreclone:0_pre_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="62:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;62:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="24" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_post_notify_stop_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="63:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;63:41:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="25" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
<lrm_rsc_op id="imagestoreclone:0_pre_notify_start_0"
operation="notify" crm-debug-origin="do_update_resource"
transition_key="60:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;60:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="31" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="d6f3e59976d73891724a1a506bd58c8d"/>
</lrm_resource>
<lrm_resource id="jsmx8" type="Xen" class="ocf"
provider="heartbeat">
<lrm_rsc_op id="jsmx8_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource"
transition_key="11:24:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:7;11:24:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="16" crm_feature_set="2.0" rc_code="7" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_start_0" operation="start"
crm-debug-origin="do_update_resource"
transition_key="50:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;50:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="34" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_stop_0" operation="stop"
crm-debug-origin="do_update_resource"
transition_key="49:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;49:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="29" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="0" op_digest="68449352fe969f1d67b81bf6cb6faae8"/>
<lrm_rsc_op id="jsmx8_monitor_10000" operation="monitor"
crm-debug-origin="do_update_resource"
transition_key="4:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
transition_magic="0:0;4:42:d54f29d5-3ad0-4948-be14-8c5f549ab96b"
call_id="35" crm_feature_set="2.0" rc_code="0" op_status="0"
interval="10000" op_digest="0a29de1107eccb4728e9fd1fdd3d8471"/>
</lrm_resource>
</lrm_resources>
</lrm>
</node_state>
</status>
</cib>

Simon Talbot MEng, ACGI
(Chief Engineer)
Tel: 020 3161 6001
Fax: 020 3161 6011

*** Voxige Wholesale Voice http://www.voxige.com ***

The information contained in this e-mail and any attachments are private

and confidential and may be legally privileged.

It is intended for the named addressee(s) only. If you are not the
intended recipient(s), you must not read, copy or use the information
contained in any way. If you receive this email or any attachments in
error, please notify us immediately by e-mail and destroy any copy you
have of it.

We accept no responsibility for any loss or damages whatsoever arising
in any way from receipt or use of this e-mail or any attachments. This
e-mail is not intended to create legally binding commitments on our
behalf, nor do its comments reflect our corporate views or policies.

_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

lmb at suse

Sep 10, 2008, 3:03 AM

Post #3 of 8 (1385 views)
Permalink

Re: Order Constraints and Clone Groups [In reply to]

On 2008-09-10T00:38:34, Simon Talbot <simont [at] nse> wrote:

> Did anyone have any observations/pointers on where to start looking for
> this problem?
>
> Hi All,
>
> Before I dig any deeper looking for the root cause, I thought I would
> run a scenario past the list to make sure I was not missing something
> obvious that has already been addresses/fixed.
>
> Take the following scenario:
>
> Clariion iSCSI SAN
> Two Hosts Connected
> SLES10 SP2 Heartbeat + OCFS2 + Xen, Identical configuration

For SLES10 SP2, please file a support request with NTS.

And please also install the latest SP2 updates (that should provide you
with 2.1.4).

> Config as follows:

Please instead use hb_report and provide it as part of the support
request.

Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG N�rnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

beekhof at gmail

Sep 15, 2008, 6:41 AM

Post #4 of 8 (1347 views)
Permalink

Re: Order Constraints and Clone Groups [In reply to]

On Fri, Sep 5, 2008 at 15:08, Simon Talbot <simont [at] nse> wrote:
> Hi All,
>
> Before I dig any deeper looking for the root cause, I thought I would
> run a scenario past the list to make sure I was not missing something
> obvious that has already been addresses/fixed.
>
> Take the following scenario:
>
> Clariion iSCSI SAN
> Two Hosts Connected
> SLES10 SP2 Heartbeat + OCFS2 + Xen, Identical configuration
>
> We have OCFS2 configured as standard with user mode cluster control from
> heartbeat and all working fine.
>
> Next we have Xen VM Instances reading their configurations from the
> Clustered OCFS2 Filesystem and running against physical volumes
> presented (shared) from the SAN, live migration etc.
>
> Again all fine
>
> So effectively we have:
> an anonymous clone group running stonith (external/SSH)
> an anonymous clone group running the OCFS2 configuration store
> and a normal VM resource for each XEN Virtual machine
>
> The problem comes with the specific interactions of the order
> constraints.
>
> We have a simple order constraint saying the XEN VM instance depends
> upon the OCFS2 configuration store
>
> With both nodes running, we can live migrate the Xen VM between the
> hosts as we like and all works fine, but the problem starts if for
> example we perform the following:
>
> Start up VM, Let us say it starts on

I assume you meant to follow up with "Host A" here :)

> Live Migrate a VM from Host A to Host B
> Instruct Host A to go into standby
>
> At this point, the Xen VM received a shutdown command, even though the
> instruction to put Host A into standby has absolutely nothing to do with
> Host B or the OCFS2 Clone instance on host B.

You'd think that. But the cluster isn't quite so smart (yet).

It just knows that you told it to restart Xen if OCFS2 changes state
(which it did, but not on the node you care about).
The solution is to use score=0 for the constraint (preventing the
restart) and add a colocation constraint between them (so that it wont
try to run anywhere that OCFS2 isn't running).

>
> The Xen VM on B shuts down and then immediately restarts again on the
> same host (B)
>
> If I then take Host A back out of standby and make it active, again the
> Xen host is shut down and re-started (again on the same host)
>
> If I take the order constraint away from the configuration, the
> behaviour does not happen and host A can be taken into standby correctly
> without affecting host B or the running VM.
>
> My feeling is that something in the order clause logic is getting
> confused and thinking that the OCFS2 clone on Host B is shutting down,
> which causes the VM to be shut down, then the CRM realises that the
> OCFS2 clone is alive and well and starts the VM back up again.
>
> Does this ring any bells, or should I start digging.
>
> Config as follows:

Please supply as an attachment next time. Mail clients tend to mangle
long lines.
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

simont at nse

Sep 15, 2008, 8:33 AM

Post #5 of 8 (1350 views)
Permalink

RE: Order Constraints and Clone Groups [In reply to]

Thanks very much for that Andrew, we had come to the same conclusion
with score=0 and it does prevent the symptom, but causes other problems
which your suggestion of a collocation constraint may help with.

When you say the cluster isn't that smart yet, what does 'yet' mean in
terms of timeframe? -- I assume this is a somewhat non-trivial
modification?

Is collocation smart enough to work out that the rule you are talking
about refers to a clone instance, rather than the clone set itself (I
assume you configure the collocation with reference to the whole
cloneset, rather than any particular clone member)?

And finally, for the benefit of myself and the list as a whole, what
exactly does an advisory order rule (Score=0) mean -- I get a feeling
for what it means, but I have not found it described in any
documentation?

Many thanks,

Simon

-----Original Message-----
From: linux-ha-dev-bounces [at] lists
[mailto:linux-ha-dev-bounces [at] lists] On Behalf Of Andrew
Beekhof
Sent: 15 September 2008 14:42
To: High-Availability Linux Development List
Subject: Re: [Linux-ha-dev] Order Constraints and Clone Groups

On Fri, Sep 5, 2008 at 15:08, Simon Talbot <simont [at] nse> wrote:
> Hi All,
>
> Before I dig any deeper looking for the root cause, I thought I would
> run a scenario past the list to make sure I was not missing something
> obvious that has already been addresses/fixed.
>
> Take the following scenario:
>
> Clariion iSCSI SAN
> Two Hosts Connected
> SLES10 SP2 Heartbeat + OCFS2 + Xen, Identical configuration
>
> We have OCFS2 configured as standard with user mode cluster control
from
> heartbeat and all working fine.
>
> Next we have Xen VM Instances reading their configurations from the
> Clustered OCFS2 Filesystem and running against physical volumes
> presented (shared) from the SAN, live migration etc.
>
> Again all fine
>
> So effectively we have:
> an anonymous clone group running stonith (external/SSH)
> an anonymous clone group running the OCFS2 configuration store
> and a normal VM resource for each XEN Virtual machine
>
> The problem comes with the specific interactions of the order
> constraints.
>
> We have a simple order constraint saying the XEN VM instance depends
> upon the OCFS2 configuration store
>
> With both nodes running, we can live migrate the Xen VM between the
> hosts as we like and all works fine, but the problem starts if for
> example we perform the following:
>
> Start up VM, Let us say it starts on

I assume you meant to follow up with "Host A" here :)

> Live Migrate a VM from Host A to Host B
> Instruct Host A to go into standby
>
> At this point, the Xen VM received a shutdown command, even though the
> instruction to put Host A into standby has absolutely nothing to do
with
> Host B or the OCFS2 Clone instance on host B.

You'd think that. But the cluster isn't quite so smart (yet).

It just knows that you told it to restart Xen if OCFS2 changes state
(which it did, but not on the node you care about).
The solution is to use score=0 for the constraint (preventing the
restart) and add a colocation constraint between them (so that it wont
try to run anywhere that OCFS2 isn't running).

>
> The Xen VM on B shuts down and then immediately restarts again on the
> same host (B)
>
> If I then take Host A back out of standby and make it active, again
the
> Xen host is shut down and re-started (again on the same host)
>
> If I take the order constraint away from the configuration, the
> behaviour does not happen and host A can be taken into standby
correctly
> without affecting host B or the running VM.
>
> My feeling is that something in the order clause logic is getting
> confused and thinking that the OCFS2 clone on Host B is shutting down,
> which causes the VM to be shut down, then the CRM realises that the
> OCFS2 clone is alive and well and starts the VM back up again.
>
> Does this ring any bells, or should I start digging.
>
> Config as follows:

Please supply as an attachment next time. Mail clients tend to mangle
long lines.
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

lmb at suse

Sep 15, 2008, 11:06 AM

Post #6 of 8 (1346 views)
Permalink

Re: Order Constraints and Clone Groups [In reply to]

On 2008-09-15T16:33:48, Simon Talbot <simont [at] nse> wrote:

> Thanks very much for that Andrew, we had come to the same conclusion
> with score=0 and it does prevent the symptom, but causes other problems
> which your suggestion of a collocation constraint may help with.

What other problems does it cause?

> Is collocation smart enough to work out that the rule you are talking
> about refers to a clone instance, rather than the clone set itself (I
> assume you configure the collocation with reference to the whole
> cloneset, rather than any particular clone member)?

Yes. You would configure collocation between the <clone> id and the Xen
guest.

> And finally, for the benefit of myself and the list as a whole, what
> exactly does an advisory order rule (Score=0) mean -- I get a feeling
> for what it means, but I have not found it described in any
> documentation?

rsc_order score="INFINITY" means that the resource depends on the clone
as a whole, think of it as a full M:N mesh, and will be ordered
afterwards - however, that also means that whenever a single clone
instance changes state, the dependencies are "broken" and thus a restart
is triggered.

rsc_order score="0" means that the resource still depends on the clone,
but only optionally. If the resource already runs, it won't be restarted
if the clone restarts somewhere.

Now, you might think that this could also cause the resource to be
started if the clone doesn't run anywhere. That's where the
rsc_collocation constraint comes in; the resource then couldn't run
anywhere either. But if the clone runs, the ordering constraint enforces
the right sequence.

By the same logic, a restart of the clone instance on nodeX also causes
a restart of the resource (should it happen to be on nodeX). Or, a
migration of the resource to another node. I think.

Does that help?

I actually keep thinking that rsc_collocation automatically implies a
rsc_order score="0" - if it's not started before, the dependency
wouldn't be there during start, which would violate the collocation. By
extension, it must be implied.

Or, at least make order default to score="0" if the target is a clone. I
really think that's what users will want by default - as long as there's
one surviving instance, don't restart anything depending on it.

Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG N�rnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

beekhof at gmail

Sep 15, 2008, 11:07 AM

Post #7 of 8 (1346 views)
Permalink

Re: Order Constraints and Clone Groups [In reply to]

On Mon, Sep 15, 2008 at 17:33, Simon Talbot <simont [at] nse> wrote:
> Thanks very much for that Andrew, we had come to the same conclusion
> with score=0 and it does prevent the symptom, but causes other problems
> which your suggestion of a collocation constraint may help with.

It will.

>
> When you say the cluster isn't that smart yet, what does 'yet' mean in
> terms of timeframe?

I want to solve this for 1.2 (1.0 being practically done already)
So depending on how many bugs people find in 1.0, probably later this
year or early next.

> -- I assume this is a somewhat non-trivial modification?

indeed

>
> Is collocation smart enough to work out that the rule you are talking
> about refers to a clone instance, rather than the clone set itself

yes

> (I
> assume you configure the collocation with reference to the whole
> cloneset, rather than any particular clone member)?

right

>
> And finally, for the benefit of myself and the list as a whole, what
> exactly does an advisory order rule (Score=0) mean -- I get a feeling
> for what it means, but I have not found it described in any
> documentation?

http://clusterlabs.org/mw/Image:Ordering_Explained.pdf

>
> -----Original Message-----
> From: linux-ha-dev-bounces [at] lists
> [mailto:linux-ha-dev-bounces [at] lists] On Behalf Of Andrew
> Beekhof
> Sent: 15 September 2008 14:42
> To: High-Availability Linux Development List
> Subject: Re: [Linux-ha-dev] Order Constraints and Clone Groups
>
> On Fri, Sep 5, 2008 at 15:08, Simon Talbot <simont [at] nse> wrote:
>> Hi All,
>>
>> Before I dig any deeper looking for the root cause, I thought I would
>> run a scenario past the list to make sure I was not missing something
>> obvious that has already been addresses/fixed.
>>
>> Take the following scenario:
>>
>> Clariion iSCSI SAN
>> Two Hosts Connected
>> SLES10 SP2 Heartbeat + OCFS2 + Xen, Identical configuration
>>
>> We have OCFS2 configured as standard with user mode cluster control
> from
>> heartbeat and all working fine.
>>
>> Next we have Xen VM Instances reading their configurations from the
>> Clustered OCFS2 Filesystem and running against physical volumes
>> presented (shared) from the SAN, live migration etc.
>>
>> Again all fine
>>
>> So effectively we have:
>> an anonymous clone group running stonith (external/SSH)
>> an anonymous clone group running the OCFS2 configuration store
>> and a normal VM resource for each XEN Virtual machine
>>
>> The problem comes with the specific interactions of the order
>> constraints.
>>
>> We have a simple order constraint saying the XEN VM instance depends
>> upon the OCFS2 configuration store
>>
>> With both nodes running, we can live migrate the Xen VM between the
>> hosts as we like and all works fine, but the problem starts if for
>> example we perform the following:
>>
>> Start up VM, Let us say it starts on
>
> I assume you meant to follow up with "Host A" here :)
>
>> Live Migrate a VM from Host A to Host B
>> Instruct Host A to go into standby
>>
>> At this point, the Xen VM received a shutdown command, even though the
>> instruction to put Host A into standby has absolutely nothing to do
> with
>> Host B or the OCFS2 Clone instance on host B.
>
> You'd think that. But the cluster isn't quite so smart (yet).
>
> It just knows that you told it to restart Xen if OCFS2 changes state
> (which it did, but not on the node you care about).
> The solution is to use score=0 for the constraint (preventing the
> restart) and add a colocation constraint between them (so that it wont
> try to run anywhere that OCFS2 isn't running).
>
>>
>> The Xen VM on B shuts down and then immediately restarts again on the
>> same host (B)
>>
>> If I then take Host A back out of standby and make it active, again
> the
>> Xen host is shut down and re-started (again on the same host)
>>
>> If I take the order constraint away from the configuration, the
>> behaviour does not happen and host A can be taken into standby
> correctly
>> without affecting host B or the running VM.
>>
>> My feeling is that something in the order clause logic is getting
>> confused and thinking that the OCFS2 clone on Host B is shutting down,
>> which causes the VM to be shut down, then the CRM realises that the
>> OCFS2 clone is alive and well and starts the VM back up again.
>>
>> Does this ring any bells, or should I start digging.
>>
>> Config as follows:
>
> Please supply as an attachment next time. Mail clients tend to mangle
> long lines.
> _______________________________________________________
> Linux-HA-Dev: Linux-HA-Dev [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
> _______________________________________________________
> Linux-HA-Dev: Linux-HA-Dev [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
>
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

simont at nse

Sep 16, 2008, 3:52 PM

Post #8 of 8 (1338 views)
Permalink

RE: Order Constraints and Clone Groups [In reply to]

Thanks very much, the Advisory Ordering in conjunction with the
collocation constraints seems to achieve the desired goal if albeit in a
somewhat roundabout way. We have now scaled the test environment up to 4
nodes and will start to address the next problems,

Best regards,

Simon

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe
Tel: 020 3161 6001
Fax: 020 3161 6011

The information contained in this e-mail and any attachments are private

and confidential and may be legally privileged.

It is intended for the named addressee(s) only. If you are not the
intended recipient(s), you must not read, copy or use the information
contained in any way. If you receive this email or any attachments in
error, please notify us immediately by e-mail and destroy any copy you
have of it.

We accept no responsibility for any loss or damages whatsoever arising
in any way from receipt or use of this e-mail or any attachments. This
e-mail is not intended to create legally binding commitments on our
behalf, nor do its comments reflect our corporate views or policies.

-----Original Message-----
From: linux-ha-dev-bounces [at] lists
[mailto:linux-ha-dev-bounces [at] lists] On Behalf Of Andrew
Beekhof
Sent: 15 September 2008 19:08
To: High-Availability Linux Development List
Subject: Re: [Linux-ha-dev] Order Constraints and Clone Groups

On Mon, Sep 15, 2008 at 17:33, Simon Talbot <simont [at] nse> wrote:
> Thanks very much for that Andrew, we had come to the same conclusion
> with score=0 and it does prevent the symptom, but causes other
problems
> which your suggestion of a collocation constraint may help with.

It will.

>
> When you say the cluster isn't that smart yet, what does 'yet' mean in
> terms of timeframe?

I want to solve this for 1.2 (1.0 being practically done already)
So depending on how many bugs people find in 1.0, probably later this
year or early next.

> -- I assume this is a somewhat non-trivial modification?

indeed

>
> Is collocation smart enough to work out that the rule you are talking
> about refers to a clone instance, rather than the clone set itself

yes

> (I
> assume you configure the collocation with reference to the whole
> cloneset, rather than any particular clone member)?

right

>
> And finally, for the benefit of myself and the list as a whole, what
> exactly does an advisory order rule (Score=0) mean -- I get a feeling
> for what it means, but I have not found it described in any
> documentation?

http://clusterlabs.org/mw/Image:Ordering_Explained.pdf

>
> -----Original Message-----
> From: linux-ha-dev-bounces [at] lists
> [mailto:linux-ha-dev-bounces [at] lists] On Behalf Of Andrew
> Beekhof
> Sent: 15 September 2008 14:42
> To: High-Availability Linux Development List
> Subject: Re: [Linux-ha-dev] Order Constraints and Clone Groups
>
> On Fri, Sep 5, 2008 at 15:08, Simon Talbot <simont [at] nse> wrote:
>> Hi All,
>>
>> Before I dig any deeper looking for the root cause, I thought I would
>> run a scenario past the list to make sure I was not missing something
>> obvious that has already been addresses/fixed.
>>
>> Take the following scenario:
>>
>> Clariion iSCSI SAN
>> Two Hosts Connected
>> SLES10 SP2 Heartbeat + OCFS2 + Xen, Identical configuration
>>
>> We have OCFS2 configured as standard with user mode cluster control
> from
>> heartbeat and all working fine.
>>
>> Next we have Xen VM Instances reading their configurations from the
>> Clustered OCFS2 Filesystem and running against physical volumes
>> presented (shared) from the SAN, live migration etc.
>>
>> Again all fine
>>
>> So effectively we have:
>> an anonymous clone group running stonith (external/SSH)
>> an anonymous clone group running the OCFS2 configuration store
>> and a normal VM resource for each XEN Virtual machine
>>
>> The problem comes with the specific interactions of the order
>> constraints.
>>
>> We have a simple order constraint saying the XEN VM instance depends
>> upon the OCFS2 configuration store
>>
>> With both nodes running, we can live migrate the Xen VM between the
>> hosts as we like and all works fine, but the problem starts if for
>> example we perform the following:
>>
>> Start up VM, Let us say it starts on
>
> I assume you meant to follow up with "Host A" here :)
>
>> Live Migrate a VM from Host A to Host B
>> Instruct Host A to go into standby
>>
>> At this point, the Xen VM received a shutdown command, even though
the
>> instruction to put Host A into standby has absolutely nothing to do
> with
>> Host B or the OCFS2 Clone instance on host B.
>
> You'd think that. But the cluster isn't quite so smart (yet).
>
> It just knows that you told it to restart Xen if OCFS2 changes state
> (which it did, but not on the node you care about).
> The solution is to use score=0 for the constraint (preventing the
> restart) and add a colocation constraint between them (so that it wont
> try to run anywhere that OCFS2 isn't running).
>
>>
>> The Xen VM on B shuts down and then immediately restarts again on the
>> same host (B)
>>
>> If I then take Host A back out of standby and make it active, again
> the
>> Xen host is shut down and re-started (again on the same host)
>>
>> If I take the order constraint away from the configuration, the
>> behaviour does not happen and host A can be taken into standby
> correctly
>> without affecting host B or the running VM.
>>
>> My feeling is that something in the order clause logic is getting
>> confused and thinking that the OCFS2 clone on Host B is shutting
down,
>> which causes the VM to be shut down, then the CRM realises that the
>> OCFS2 clone is alive and well and starts the VM back up again.
>>
>> Does this ring any bells, or should I start digging.
>>
>> Config as follows:
>
> Please supply as an attachment next time. Mail clients tend to mangle
> long lines.
> _______________________________________________________
> Linux-HA-Dev: Linux-HA-Dev [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
> _______________________________________________________
> Linux-HA-Dev: Linux-HA-Dev [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
>
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads