abe3425 at simplex-cn
Apr 6, 2011, 8:16 PM
Post #1 of 6
(356 views)
Permalink
|
|
node$B5Z$S(Bnic$B%@%&%s$N860x$K$D$$$F(B
|
|
$B0$It$H?=$7$^$9!#(B
$B1?MQCf$K%O!<%H%S!<%H$H(BNIC$B$N%@%&%s$r8!CN$7$^$7$?!#(B
$B$?$@!"%O!<%H%S!<%H$G(Bnode$B%@%&%s$r8!CN$7$F$+$i(B4$BJ,8e$K(BNIC$B$,%@%&%s$7$F$$$k$?(B
$B$a!"$I$A$i$,%H%j%,!<$K$J$C$FH/@8$7$?$N$+$,$o$+$j$^$;$s!#(B
$B%m%0$rFI$_<h$kNO$,$J$$$N$G!"N>%5!<%P$,$I$N$h$&$J=hM}$r$7$?$N$+$465<xD:$1(B
$B$J$$$G$7$g$&$+!)Kt!"860xEyJ,$+$kJ}$$$i$C$7$c$$$^$7$?$i$465<xD:$-$?$$$HB8(B
$B$8$^$9!#(B
------------------------------------------------------------------------
$B4D6-(B:
RHEL 4.4
heartbeat 2.0.4
$B%N!<%I!'(B server01(bond1:eth1/eth3$B!K(Bserver02(bond1:eth1/eth3$B!K$N(B2$BBf9=@.(B
$B!!"((Bbond1$B$O%O!<%H%S!<%H%Q%1%C%H [at] lM(B
$B!!"(%5!<%P@\B3@h$N(BNW$B5!4o$G$O%(%i!<$J$7(B
------------------------------------------------------------------------
server01
------------------------------------------------------------------------
Mar 26 23:22:33 server01 heartbeat: [31257]: WARN: node server02: is dead
Mar 26 23:22:33 server01 heartbeat: [31257]: info: Link server02:bond1 dead.
Mar 26 23:22:33 server01 crmd: [31276]: notice: crmd_ha_status_callback: Status update: Node server02 now has status [dead]
Mar 26 23:22:33 server01 crmd: [31276]: info: mem_handle_event: Got an event OC_EV_MS_NOT_PRIMARY from ccm
Mar 26 23:22:33 server01 cib: [31272]: info: mem_handle_event: Got an event OC_EV_MS_NOT_PRIMARY from ccm
Mar 26 23:22:33 server01 crmd: [31276]: info: mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0, new_idx=0, old_idx=4
Mar 26 23:22:33 server01 crmd: [31276]: info: crmd_ccm_msg_callback: Quorum lost after event=NOT PRIMARY (id=2)
Mar 26 23:22:33 server01 cib: [31272]: info: mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0, new_idx=0, old_idx=4
Mar 26 23:22:33 server01 cib: [31272]: info: cib_diff_notify: Local-only Change (client:31276, call: 23): 0.99.955 (ok)
Mar 26 23:22:33 server01 cib: [28801]: info: write_cib_contents: Wrote version 0.99.955 of the CIB to disk (digest: 9ac99cfb7f4fb83d7c9487d6052379d7)
Mar 26 23:22:39 server01 ccm: [31271]: info: Break tie for 2 nodes cluster
Mar 26 23:22:39 server01 cib: [31272]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:22:39 server01 crmd: [31276]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:22:39 server01 cib: [31272]: info: mem_handle_event: no mbr_track info
Mar 26 23:22:39 server01 crmd: [31276]: info: mem_handle_event: no mbr_track info
Mar 26 23:22:39 server01 cib: [31272]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:22:39 server01 crmd: [31276]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:22:39 server01 cib: [31272]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
Mar 26 23:22:39 server01 crmd: [31276]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
Mar 26 23:22:39 server01 cib: [31272]: info: cib_ccm_msg_callback: LOST: server02
Mar 26 23:22:39 server01 crmd: [31276]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=3)
Mar 26 23:22:39 server01 cib: [31272]: info: cib_ccm_msg_callback: PEER: server01
Mar 26 23:22:39 server01 crmd: [31276]: WARN: check_dead_member: Our DC node (server02) left the cluster
Mar 26 23:22:39 server01 crmd: [31276]: info: ccm_event_detail: NEW MEMBERSHIP: trans=3, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=3
Mar 26 23:22:39 server01 cib: [31272]: info: cib_diff_notify: Local-only Change (client:31276, call: 24): 0.99.955 (ok)
Mar 26 23:22:39 server01 crmd: [31276]: info: ccm_event_detail: CURRENT: server01 [nodeid=0, born=3]
Mar 26 23:22:39 server01 cib: [28863]: info: write_cib_contents: Wrote version 0.99.955 of the CIB to disk (digest: 26d70b35a031b907118436a201aadcea)
Mar 26 23:22:39 server01 crmd: [31276]: info: ccm_event_detail: LOST: server02 [nodeid=1, born=1]
Mar 26 23:22:39 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=check_dead_member ]
Mar 26 23:22:39 server01 crmd: [31276]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:22:39 server01 crmd: [31276]: info: do_election_count_vote: Updated voted hash for server01 to vote
Mar 26 23:22:39 server01 crmd: [31276]: info: do_election_count_vote: Election ignore: our vote (server01)
Mar 26 23:22:39 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Mar 26 23:22:39 server01 crmd: [31276]: info: start_subsystem: Starting sub-system "tengine"
Mar 26 23:22:39 server01 crmd: [31276]: info: start_subsystem: Starting sub-system "pengine"
Mar 26 23:22:39 server01 crmd: [31276]: info: do_dc_takeover: Taking over DC status for this partition
Mar 26 23:22:39 server01 cib: [31272]: info: cib_process_readwrite: We are now in R/W mode
Mar 26 23:22:39 server01 crmd: [31276]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:22:39 server01 crmd: [31276]: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
Mar 26 23:22:39 server01 cib: [31272]: info: cib_diff_notify: Update (client: 31276, call:27): 0.99.955 -> 0.99.956 (ok)
Mar 26 23:22:39 server01 cib: [28866]: info: write_cib_contents: Wrote version 0.99.956 of the CIB to disk (digest: 3e2730aebcc63e91c43e701b3d581a94)
Mar 26 23:22:39 server01 crmd: [31276]: info: update_dc: Set DC to server01 (1.0.7)
Mar 26 23:22:39 server01 pengine: [28865]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 26 23:22:39 server01 pengine: [28865]: info: init_start: Starting pengine
Mar 26 23:22:39 server01 tengine: [28864]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 26 23:22:39 server01 tengine: [28864]: info: G_main_add_TriggerHandler: Added signal manual handler
Mar 26 23:22:39 server01 cib: [31272]: info: cib_null_callback: Setting cib_diff_notify callbacks for tengine: on
Mar 26 23:22:39 server01 tengine: [28864]: info: init_start: Registering TE UUID: 3f179a7b-b333-4c12-8acb-c3f33d4b6d1b
Mar 26 23:22:39 server01 tengine: [28864]: info: set_graph_functions: Setting custom graph functions
Mar 26 23:22:40 server01 tengine: [28864]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses
Mar 26 23:22:40 server01 tengine: [28864]: info: init_start: Starting tengine
Mar 26 23:22:40 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 26 23:22:40 server01 crmd: [31276]: info: do_state_transition: All 1 cluster nodes responded to the join offer.
Mar 26 23:22:40 server01 crmd: [31276]: info: update_attrd: Connecting to attrd...
Mar 26 23:22:40 server01 cib: [31272]: info: sync_our_cib: Syncing CIB to all peers
Mar 26 23:22:40 server01 attrd: [31275]: info: attrd_local_callback: Sending full refresh
Mar 26 23:22:40 server01 cib: [31272]: info: cib_diff_notify: Update (client: 31276, call:30): 0.99.956 -> 0.99.957 (ok)
Mar 26 23:22:40 server01 crmd: [31276]: info: update_dc: Set DC to server01 (1.0.7)
Mar 26 23:22:40 server01 tengine: [28864]: info: te_update_diff: Processing diff (cib_update): 0.99.956 -> 0.99.957
Mar 26 23:22:40 server01 cib: [31272]: info: cib_diff_notify: Update (client: 31276, call:31): 0.99.957 -> 0.100.958 (ok)
Mar 26 23:22:40 server01 crmd: [31276]: info: append_restart_list: Resource ip_sample01 does not support reloads
Mar 26 23:22:40 server01 tengine: [28864]: info: te_update_diff: Processing diff (cib_bump): 0.99.957 -> 0.100.958
Mar 26 23:22:40 server01 cib: [31272]: info: cib_diff_notify: Update (client: 31276, call:32): 0.100.958 -> 0.100.959 (ok)
Mar 26 23:22:40 server01 tengine: [28864]: info: te_update_diff: Processing diff (cib_update): 0.100.958 -> 0.100.959
Mar 26 23:22:40 server01 cib: [28895]: info: write_cib_contents: Wrote version 0.100.959 of the CIB to disk (digest: 0678707c70ba9579c2cdca52254a028f)
Mar 26 23:22:40 server01 crmd: [31276]: info: do_dc_join_ack: join-1: Updating node state to member for server01)
Mar 26 23:22:40 server01 cib: [31272]: info: cib_diff_notify: Update (client: 31276, call:33): 0.100.959 -> 0.100.960 (ok)
Mar 26 23:22:40 server01 tengine: [28864]: info: te_update_diff: Processing diff (cib_update): 0.100.959 -> 0.100.960
Mar 26 23:22:40 server01 tengine: [28864]: info: process_graph_event: Action ip_sample01_monitor_0 initiated by a different transitioner
Mar 26 23:22:40 server01 tengine: [28864]: info: update_abort_priority: Abort priority upgraded to 1000000
Mar 26 23:22:40 server01 tengine: [28864]: info: update_abort_priority: 'DC Takeover'-class abort superceeded
Mar 26 23:22:40 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 26 23:22:40 server01 crmd: [31276]: info: do_state_transition: All 1 cluster nodes are eligable to run resources.
Mar 26 23:22:40 server01 cib: [28896]: info: write_cib_contents: Wrote version 0.100.960 of the CIB to disk (digest: ed7bfe60632f0e49720bfdfc14c26a2b)
Mar 26 23:22:40 server01 pengine: [28865]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" cib_feature_revision="1.3" ignore_dtd="false" num_peers="2" ccm_transition="3" generated="true" dc_uuid="1c847fdd-4f55-4d04-ae67-09ea48ffaff5" epoch="100" num_updates="960"/>
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value 'false' for cluster option 'stonith-enabled'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
Mar 26 23:22:40 server01 pengine: [28865]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
Mar 26 23:22:40 server01 pengine: [28865]: info: determine_online_status: Node server01 is online
Mar 26 23:22:41 server01 pengine: [28865]: info: native_print: ip_sample01 (heartbeat::ocf:IPaddr): Started server01
Mar 26 23:22:41 server01 pengine: [28865]: notice: NoRoleChange: Leave resource ip_sample01 (server01)
Mar 26 23:22:41 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Mar 26 23:22:41 server01 tengine: [28864]: info: unpack_graph: Unpacked transition 0: 0 actions in 0 synapses
Mar 26 23:22:41 server01 pengine: [28865]: info: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-306.bz2
Mar 26 23:22:41 server01 tengine: [28864]: info: run_graph: Transition 0: (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0)
Mar 26 23:22:41 server01 tengine: [28864]: info: notify_crmd: Transition 0 status: te_complete - <null>
Mar 26 23:22:41 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Mar 26 23:24:44 server01 cib: [31272]: info: cib_stats: Processed 13 operations (26923.00us average, 0% utilization) in the last 10min
Mar 26 23:26:20 server01 kernel: NETDEV WATCHDOG: eth1: transmit timed out
Mar 26 23:26:21 server01 kernel: bnx2: eth1 NIC Link is Down
Mar 26 23:26:21 server01 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Mar 26 23:26:21 server01 kernel: bonding: bond1: making interface eth3 the new active one.
Mar 26 23:26:21 server01 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Mar 26 23:26:21 server01 heartbeat: [31257]: CRIT: Cluster node server02 returning after partition.
Mar 26 23:26:21 server01 heartbeat: [31257]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
Mar 26 23:26:21 server01 heartbeat: [31257]: WARN: Deadtime value may be too small.
Mar 26 23:26:21 server01 heartbeat: [31257]: info: See FAQ for information on tuning deadtime.
Mar 26 23:26:21 server01 heartbeat: [31257]: info: URL: http://linux-ha.org/FAQ#heavy_load
Mar 26 23:26:21 server01 heartbeat: [31257]: info: Link server02:bond1 up.
Mar 26 23:26:21 server01 heartbeat: [31257]: WARN: Late heartbeat: Node server02: interval 257960 ms
Mar 26 23:26:21 server01 heartbeat: [31257]: info: Status update for node server02: status active
Mar 26 23:26:21 server01 crmd: [31276]: notice: crmd_ha_status_callback: Status update: Node server02 now has status [active]
Mar 26 23:26:21 server01 cib: [31272]: info: cib_diff_notify: Local-only Change (client:31276, call: 36): 0.100.960 (ok)
Mar 26 23:26:21 server01 tengine: [28864]: info: te_update_diff: Processing diff (cib_update): 0.100.960 -> 0.100.960
Mar 26 23:26:21 server01 cib: [31410]: info: write_cib_contents: Wrote version 0.100.960 of the CIB to disk (digest: fba625f2faf735f82b53e0428ce7cc14)
Mar 26 23:26:21 server01 kernel: bonding: bond1: link status definitely up for interface eth1.
Mar 26 23:26:21 server01 kernel: bonding: bond1: making interface eth1 the new active one.
Mar 26 23:26:21 server01 kernel: bnx2: eth1 NIC Link is Down
Mar 26 23:26:21 server01 kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Mar 26 23:26:22 server01 heartbeat: [31257]: info: all clients are now paused
Mar 26 23:26:24 server01 heartbeat: [31257]: WARN: 4 lost packet(s) for [server02] [4866987:4866992]
Mar 26 23:26:24 server01 heartbeat: [31257]: info: all clients are now resumed
Mar 26 23:26:24 server01 heartbeat: [31257]: info: No pkts missing from server02!
Mar 26 23:26:27 server01 crmd: [31276]: WARN: crmd_ha_msg_callback: Ignoring HA message (op=noop) from server02: not in our membership list (size=1)
Mar 26 23:26:28 server01 cib: [31272]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:26:28 server01 cib: [31272]: info: mem_handle_event: no mbr_track info
Mar 26 23:26:28 server01 crmd: [31276]: WARN: crmd_ha_msg_callback: Ignoring HA message (op=noop) from server02: not in our membership list (size=1)
Mar 26 23:26:28 server01 cib: [31272]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:26:28 server01 cib: [31272]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
Mar 26 23:26:28 server01 crmd: [31276]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:26:28 server01 cib: [31272]: info: cib_ccm_msg_callback: PEER: server02
Mar 26 23:26:28 server01 crmd: [31276]: info: mem_handle_event: no mbr_track info
Mar 26 23:26:28 server01 cib: [31272]: info: cib_ccm_msg_callback: PEER: server01
Mar 26 23:26:28 server01 crmd: [31276]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:26:28 server01 crmd: [31276]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
Mar 26 23:26:28 server01 crmd: [31276]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=2)
Mar 26 23:26:28 server01 crmd: [31276]: info: ccm_event_detail: NEW MEMBERSHIP: trans=2, nodes=2, new=1, lost=0 n_idx=0, new_idx=2, old_idx=4
Mar 26 23:26:28 server01 cib: [31272]: info: cib_diff_notify: Local-only Change (client:31276, call: 37): 0.100.960 (ok)
Mar 26 23:26:28 server01 crmd: [31276]: info: ccm_event_detail: CURRENT: server02 [nodeid=1, born=1]
Mar 26 23:26:28 server01 tengine: [28864]: info: te_update_diff: Processing diff (cib_update): 0.100.960 -> 0.100.960
Mar 26 23:26:28 server01 cib: [2246]: info: write_cib_contents: Wrote version 0.100.960 of the CIB to disk (digest: 457e15e0002417fe470cb4e1c207ce0e)
Mar 26 23:26:28 server01 crmd: [31276]: info: ccm_event_detail: CURRENT: server01 [nodeid=0, born=2]
Mar 26 23:26:28 server01 crmd: [31276]: info: ccm_event_detail: NEW: server02 [nodeid=1, born=1]
Mar 26 23:26:29 server01 crmd: [31276]: info: do_election_count_vote: Election check: vote from server02
Mar 26 23:26:29 server01 crmd: [31276]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:26:29 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_IDLE -> S_RELEASE_DC [ input=I_RELEASE_DC cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Mar 26 23:26:29 server01 crmd: [31276]: info: do_dc_release: DC role released
Mar 26 23:26:29 server01 crmd: [31276]: info: stop_subsystem: Sent -TERM to pengine: [28865]
Mar 26 23:26:29 server01 cib: [31272]: info: cib_process_readwrite: We are now in R/O mode
Mar 26 23:26:29 server01 crmd: [31276]: info: stop_subsystem: Sent -TERM to tengine: [28864]
Mar 26 23:26:29 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_RELEASE_DC -> S_PENDING [ input=I_RELEASE_SUCCESS cause=C_FSA_INTERNAL origin=do_dc_release ]
Mar 26 23:26:29 server01 crmd: [31276]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:26:29 server01 pengine: [28865]: info: pengine_shutdown: Exiting PEngine (SIGTERM)
Mar 26 23:26:29 server01 tengine: [28864]: info: update_abort_priority: Abort priority upgraded to 1000000
Mar 26 23:26:29 server01 tengine: [28864]: info: update_abort_priority: Abort action 2 superceeded by 3
Mar 26 23:26:29 server01 tengine: [28864]: info: notify_crmd: Exiting after transition
Mar 26 23:26:29 server01 crmd: [31276]: info: crmdManagedChildDied: Process pengine:[28865] exited (signal=0, exitcode=0)
Mar 26 23:26:29 server01 crmd: [31276]: info: crmdManagedChildDied: Process tengine:[28864] exited (signal=0, exitcode=0)
Mar 26 23:26:29 server01 crmd: [31276]: WARN: G_SIG_dispatch: Dispatch function for SIGCHLD took too long to execute: 100 ms (> 10 ms) (GSource: 0x807daf0)
Mar 26 23:26:29 server01 crmd: [31276]: info: process_client_disconnect: Received HUP from pengine:[-1]
Mar 26 23:26:29 server01 crmd: [31276]: info: process_client_disconnect: Received HUP from tengine:[-1]
Mar 26 23:26:30 server01 crmd: [31276]: info: update_dc: Set DC to server02 (1.0.7)
Mar 26 23:26:30 server01 cib: [31272]: WARN: cib_process_diff: Diff 0.99.957 -> 0.99.958 not applied to 0.100.960: current "epoch" is greater than required
Mar 26 23:26:30 server01 cib: [31272]: WARN: do_cib_notify: cib_apply_diff of <diff > FAILED: Application of an update diff failed
Mar 26 23:26:30 server01 cib: [31272]: WARN: cib_process_request: cib_apply_diff operation failed: Application of an update diff failed
Mar 26 23:26:31 server01 cib: [31272]: info: sync_our_cib: Syncing CIB to all peers
Mar 26 23:26:33 server01 crmd: [31276]: info: update_dc: Set DC to server02 (1.0.7)
Mar 26 23:26:33 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:44): 0.100.960 -> 0.100.961 (ok)
Mar 26 23:26:33 server01 crmd: [31276]: info: append_restart_list: Resource ip_sample01 does not support reloads
Mar 26 23:26:33 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 26 23:26:33 server01 crmd: [31276]: info: do_election_count_vote: Election check: vote from server02
Mar 26 23:26:33 server01 crmd: [31276]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:26:33 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_NOT_DC -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Mar 26 23:26:33 server01 crmd: [31276]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:26:33 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:45): 0.100.961 -> 0.100.962 (ok)
Mar 26 23:26:33 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:46): 0.100.962 -> 0.101.963 (ok)
Mar 26 23:26:33 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:47): 0.101.963 -> 0.101.964 (ok)
Mar 26 23:26:33 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:48): 0.101.964 -> 0.101.965 (ok)
Mar 26 23:26:33 server01 cib: [2361]: info: write_cib_contents: Wrote version 0.101.965 of the CIB to disk (digest: 4b29679f39cfa464fc1b6cfef3945da1)
Mar 26 23:26:34 server01 crmd: [31276]: info: update_dc: Set DC to server02 (1.0.7)
Mar 26 23:26:34 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:51): 0.101.965 -> 0.101.966 (ok)
Mar 26 23:26:34 server01 cib: [2378]: info: write_cib_contents: Wrote version 0.101.966 of the CIB to disk (digest: f6345a73547f8c0b7e7c2b5dcd86866a)
Mar 26 23:26:35 server01 crmd: [31276]: info: update_dc: Set DC to server02 (1.0.7)
Mar 26 23:26:35 server01 crmd: [31276]: info: append_restart_list: Resource ip_sample01 does not support reloads
Mar 26 23:26:35 server01 crmd: [31276]: info: do_state_transition: server01: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 26 23:26:35 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:54): 0.101.966 -> 0.101.967 (ok)
Mar 26 23:26:35 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:55): 0.101.967 -> 0.102.968 (ok)
Mar 26 23:26:35 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:56): 0.102.968 -> 0.102.969 (ok)
Mar 26 23:26:35 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:57): 0.102.969 -> 0.102.970 (ok)
Mar 26 23:26:35 server01 cib: [2396]: info: write_cib_contents: Wrote version 0.102.970 of the CIB to disk (digest: 954b7768896a9fe88bbc66b061b8c8ed)
Mar 26 23:26:36 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:58): 0.102.970 -> 0.102.971 (ok)
Mar 26 23:26:36 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:59): 0.102.971 -> 0.102.972 (ok)
Mar 26 23:26:36 server01 cib: [2397]: info: write_cib_contents: Wrote version 0.102.972 of the CIB to disk (digest: 99557e3341ebd53b8a513bc2766c05f3)
Mar 26 23:26:37 server01 crmd: [31276]: info: do_lrm_rsc_op: Performing op=ip_sample01_stop_0 key=7:4:38d96444-0d9f-4b9b-bf00-d79d4dfbd3ae)
Mar 26 23:26:37 server01 crmd: [31276]: WARN: process_lrm_event: LRM operation ip_sample01_monitor_10000 (call=4, rc=-2) Cancelled
Mar 26 23:26:37 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:65): 0.102.972 -> 0.102.973 (ok)
Mar 26 23:26:37 server01 cib: [31272]: info: cib_diff_notify: Update (client: 16606, call:66): 0.102.973 -> 0.102.974 (ok)
Mar 26 23:26:37 server01 cib: [2407]: info: write_cib_contents: Wrote version 0.102.974 of the CIB to disk (digest: a9d539a53d6c02d5dfb7888448021854)
Mar 26 23:26:37 server01 lrmd: [31273]: info: RA output: (ip_sample01:stop:stderr) SIOCDELRT: No such process
Mar 26 23:26:37 server01 IPaddr[2404]: INFO: /sbin/ifconfig bond0:0 192.168.1.1 down
Mar 26 23:26:37 server01 crmd: [31276]: info: process_lrm_event: LRM operation ip_sample01_stop_0 (call=6, rc=0) complete
Mar 26 23:26:38 server01 crmd: [31276]: info: do_lrm_rsc_op: Performing op=ip_sample01_start_0 key=4:5:38d96444-0d9f-4b9b-bf00-d79d4dfbd3ae)
Mar 26 23:26:38 server01 cib: [31272]: info: cib_diff_notify: Update (client: 31276, call:44): 0.102.974 -> 0.102.975 (ok)
Mar 26 23:26:38 server01 cib: [2427]: info: write_cib_contents: Wrote version 0.102.975 of the CIB to disk (digest: 008efadbd4bbcfce2df6d0a0abf85e7a)
Mar 26 23:26:38 server01 IPaddr[2422]: INFO: Using calculated netmask for 192.168.1.1: 255.255.255.0
Mar 26 23:26:38 server01 IPaddr[2422]: DEBUG: Using calculated broadcast for 192.168.1.1: 192.168.1.255
Mar 26 23:26:38 server01 IPaddr[2422]: INFO: eval /sbin/ifconfig bond0:0 192.168.1.1 netmask 255.255.255.0 broadcast 192.168.1.255
Mar 26 23:26:38 server01 IPaddr[2422]: DEBUG: Sending Gratuitous Arp for 192.168.1.1 on bond0:0 [bond0]
Mar 26 23:26:38 server01 crmd: [31276]: info: process_lrm_event: LRM operation ip_sample01_start_0 (call=7, rc=0) complete
Mar 26 23:26:38 server01 crmd: [31276]: info: append_restart_list: Resource ip_sample01 does not support reloads
Mar 26 23:26:39 server01 cib: [31272]: info: cib_diff_notify: Update (client: 31276, call:45): 0.102.975 -> 0.102.976 (ok)
Mar 26 23:26:39 server01 crmd: [31276]: info: do_lrm_rsc_op: Performing op=ip_sample01_monitor_10000 key=5:5:38d96444-0d9f-4b9b-bf00-d79d4dfbd3ae)
Mar 26 23:26:39 server01 cib: [2500]: info: write_cib_contents: Wrote version 0.102.976 of the CIB to disk (digest: 161c849592bfc6262e0db96f3b4e8c36)
Mar 26 23:26:39 server01 crmd: [31276]: info: process_lrm_event: LRM operation ip_sample01_monitor_10000 (call=8, rc=0) complete
Mar 26 23:26:40 server01 cib: [31272]: info: cib_diff_notify: Update (client: 31276, call:46): 0.102.976 -> 0.102.977 (ok)
Mar 26 23:26:40 server01 cib: [2517]: info: write_cib_contents: Wrote version 0.102.977 of the CIB to disk (digest: 38f7498a262c2c842d7de0f6af81236b)
------------------------------------------------------------------------
server02
------------------------------------------------------------------------
Mar 26 23:22:32 server02 heartbeat: [16581]: WARN: node server01: is dead
Mar 26 23:22:32 server02 heartbeat: [16581]: info: Link server01:bond1 dead.
Mar 26 23:22:32 server02 crmd: [16606]: notice: crmd_ha_status_callback: Status update: Node server01 now has status [dead]
Mar 26 23:22:32 server02 ccm: [16601]: info: Break tie for 2 nodes cluster
Mar 26 23:22:32 server02 crmd: [16606]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:22:32 server02 crmd: [16606]: info: mem_handle_event: no mbr_track info
Mar 26 23:22:32 server02 crmd: [16606]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:22:32 server02 cib: [16602]: info: cib_diff_notify: Local-only Change (client:16606, call: 29): 0.99.955 (ok)
Mar 26 23:22:33 server02 crmd: [16606]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
Mar 26 23:22:33 server02 cib: [16602]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:22:33 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.99.955 -> 0.99.955
Mar 26 23:22:33 server02 crmd: [16606]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=3)
Mar 26 23:22:33 server02 cib: [16602]: info: mem_handle_event: no mbr_track info
Mar 26 23:22:33 server02 tengine: [16988]: WARN: match_down_event: No match for shutdown action on 1c847fdd-4f55-4d04-ae67-09ea48ffaff5
Mar 26 23:22:33 server02 cib: [16602]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:22:33 server02 tengine: [16988]: info: extract_event: Stonith/shutdown of 1c847fdd-4f55-4d04-ae67-09ea48ffaff5 not matched
Mar 26 23:22:33 server02 cib: [16602]: info: mem_handle_event: instance=3, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
Mar 26 23:22:33 server02 crmd: [16606]: info: ccm_event_detail: NEW MEMBERSHIP: trans=3, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=3
Mar 26 23:22:33 server02 tengine: [16988]: info: update_abort_priority: Abort priority upgraded to 1000000
Mar 26 23:22:33 server02 cib: [16602]: info: cib_ccm_msg_callback: LOST: server01
Mar 26 23:22:33 server02 crmd: [16606]: info: ccm_event_detail: CURRENT: server02 [nodeid=1, born=3]
Mar 26 23:22:33 server02 tengine: [16988]: info: te_update_diff: Aborting on transient_attributes deletions
Mar 26 23:22:33 server02 cib: [16602]: info: cib_ccm_msg_callback: PEER: server02
Mar 26 23:22:33 server02 crmd: [16606]: info: ccm_event_detail: LOST: server01 [nodeid=0, born=2]
Mar 26 23:22:33 server02 cib: [16602]: info: cib_diff_notify: Local-only Change (client:16606, call: 30): 0.99.955 (ok)
Mar 26 23:22:33 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ]
Mar 26 23:22:33 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.99.955 -> 0.99.955
Mar 26 23:22:33 server02 crmd: [16606]: info: do_state_transition: All 1 cluster nodes are eligable to run resources.
Mar 26 23:22:33 server02 cib: [2022]: info: write_cib_contents: Wrote version 0.99.955 of the CIB to disk (digest: 014f2ac3cc64cbe0bbefa712d310036b)
Mar 26 23:22:33 server02 pengine: [16989]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" cib_feature_revision="1.3" ignore_dtd="false" num_peers="2" generated="true" epoch="99" num_updates="955" cib-last-written="Sat Jan 29 13:22:48 2011" ccm_transition="3" dc_uuid="41b08ac0-6f59-4771-ad28-45b18ad47ce4"/>
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value 'false' for cluster option 'stonith-enabled'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
Mar 26 23:22:33 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
Mar 26 23:22:33 server02 pengine: [16989]: info: determine_online_status: Node server02 is online
Mar 26 23:22:33 server02 pengine: [16989]: info: native_print: ip_sample01 (heartbeat::ocf:IPaddr): Stopped
Mar 26 23:22:33 server02 pengine: [16989]: notice: StartRsc: server02 Start ip_sample01
Mar 26 23:22:33 server02 pengine: [16989]: notice: Recurring: server02 ip_sample01_monitor_10000
Mar 26 23:22:33 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Mar 26 23:22:33 server02 tengine: [16988]: info: unpack_graph: Unpacked transition 3: 2 actions in 2 synapses
Mar 26 23:22:33 server02 pengine: [16989]: info: process_pe_message: Transition 3: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-328.bz2
Mar 26 23:22:33 server02 tengine: [16988]: info: send_rsc_command: Initiating action 3: ip_sample01_start_0 on server02
Mar 26 23:22:33 server02 crmd: [16606]: info: do_lrm_rsc_op: Performing op=ip_sample01_start_0 key=3:3:38d96444-0d9f-4b9b-bf00-d79d4dfbd3ae)
Mar 26 23:22:34 server02 IPaddr[2051]: INFO: Using calculated netmask for 192.168.1.1: 255.255.255.0
Mar 26 23:22:34 server02 IPaddr[2051]: DEBUG: Using calculated broadcast for 192.168.1.1: 192.168.1.255
Mar 26 23:22:34 server02 IPaddr[2051]: INFO: eval /sbin/ifconfig bond0:0 192.168.1.1 netmask 255.255.255.0 broadcast 192.168.1.255
Mar 26 23:22:34 server02 IPaddr[2051]: DEBUG: Sending Gratuitous Arp for 192.168.1.1 on bond0:0 [bond0]
Mar 26 23:22:34 server02 crmd: [16606]: info: process_lrm_event: LRM operation ip_sample01_start_0 (call=3, rc=0) complete
Mar 26 23:22:34 server02 crmd: [16606]: info: append_restart_list: Resource ip_sample01 does not support reloads
Mar 26 23:22:34 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:33): 0.99.955 -> 0.99.956 (ok)
Mar 26 23:22:34 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.99.955 -> 0.99.956
Mar 26 23:22:34 server02 tengine: [16988]: info: match_graph_event: Action ip_sample01_start_0 (3) confirmed on 41b08ac0-6f59-4771-ad28-45b18ad47ce4
Mar 26 23:22:34 server02 tengine: [16988]: info: send_rsc_command: Initiating action 4: ip_sample01_monitor_10000 on server02
Mar 26 23:22:34 server02 cib: [2126]: info: write_cib_contents: Wrote version 0.99.956 of the CIB to disk (digest: 0232c552f023d587ed461befa08d618c)
Mar 26 23:22:34 server02 crmd: [16606]: info: do_lrm_rsc_op: Performing op=ip_sample01_monitor_10000 key=4:3:38d96444-0d9f-4b9b-bf00-d79d4dfbd3ae)
Mar 26 23:22:34 server02 crmd: [16606]: info: process_lrm_event: LRM operation ip_sample01_monitor_10000 (call=4, rc=0) complete
Mar 26 23:22:34 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:34): 0.99.956 -> 0.99.957 (ok)
Mar 26 23:22:34 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.99.956 -> 0.99.957
Mar 26 23:22:34 server02 cib: [2141]: info: write_cib_contents: Wrote version 0.99.957 of the CIB to disk (digest: 3ebc188cf72cbbcbb79e79d891da5510)
Mar 26 23:22:34 server02 tengine: [16988]: info: match_graph_event: Action ip_sample01_monitor_10000 (4) confirmed on 41b08ac0-6f59-4771-ad28-45b18ad47ce4
Mar 26 23:22:34 server02 tengine: [16988]: info: run_graph: Transition 3: (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0)
Mar 26 23:22:34 server02 tengine: [16988]: info: notify_crmd: Transition 3 status: te_complete - <null>
Mar 26 23:22:34 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Mar 26 23:22:45 server02 ntpd[4614]: frequency error 500 PPM exceeds tolerance 500 PPM
Mar 26 23:23:51 server02 ntpd[4614]: frequency error 500 PPM exceeds tolerance 500 PPM
Mar 26 23:24:03 server02 cib: [16602]: info: cib_stats: Processed 6 operations (23333.00us average, 0% utilization) in the last 10min
Mar 26 23:26:21 server02 heartbeat: [16581]: CRIT: Cluster node server01 returning after partition.
Mar 26 23:26:21 server02 heartbeat: [16581]: info: For information on cluster partitions, See URL: http://linux-ha.org/SplitBrain
Mar 26 23:26:21 server02 heartbeat: [16581]: WARN: Deadtime value may be too small.
Mar 26 23:26:21 server02 heartbeat: [16581]: info: See FAQ for information on tuning deadtime.
Mar 26 23:26:21 server02 heartbeat: [16581]: info: URL: http://linux-ha.org/FAQ#heavy_load
Mar 26 23:26:21 server02 heartbeat: [16581]: info: Link server01:bond1 up.
Mar 26 23:26:21 server02 heartbeat: [16581]: WARN: Late heartbeat: Node server01: interval 258390 ms
Mar 26 23:26:21 server02 heartbeat: [16581]: info: Status update for node server01: status active
Mar 26 23:26:21 server02 crmd: [16606]: notice: crmd_ha_status_callback: Status update: Node server01 now has status [active]
Mar 26 23:26:21 server02 cib: [16602]: info: cib_diff_notify: Local-only Change (client:16606, call: 35): 0.99.957 (ok)
Mar 26 23:26:21 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.99.957 -> 0.99.957
Mar 26 23:26:21 server02 cib: [4694]: info: write_cib_contents: Wrote version 0.99.957 of the CIB to disk (digest: 2110c2fa7f41646c1da4d09059ea4921)
Mar 26 23:26:21 server02 heartbeat: [16581]: info: all clients are now paused
Mar 26 23:26:23 server02 heartbeat: [16581]: WARN: 1 lost packet(s) for [server01] [4866870:4866872]
Mar 26 23:26:23 server02 heartbeat: [16581]: info: No pkts missing from server01!
Mar 26 23:26:25 server02 ccm: [16601]: info: Break tie for 2 nodes cluster
Mar 26 23:26:25 server02 cib: [16602]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:26:25 server02 cib: [16602]: info: mem_handle_event: no mbr_track info
Mar 26 23:26:25 server02 cib: [16602]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:26:25 server02 cib: [16602]: info: mem_handle_event: instance=1, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3
Mar 26 23:26:25 server02 cib: [16602]: info: cib_ccm_msg_callback: PEER: server02
Mar 26 23:26:25 server02 crmd: [16606]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:26:25 server02 crmd: [16606]: info: mem_handle_event: no mbr_track info
Mar 26 23:26:25 server02 crmd: [16606]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:26:25 server02 crmd: [16606]: info: mem_handle_event: instance=1, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3
Mar 26 23:26:25 server02 crmd: [16606]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=1)
Mar 26 23:26:25 server02 crmd: [16606]: info: ccm_event_detail: NEW MEMBERSHIP: trans=1, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=3
Mar 26 23:26:25 server02 crmd: [16606]: info: ccm_event_detail: CURRENT: server02 [nodeid=1, born=1]
Mar 26 23:26:25 server02 cib: [4817]: info: write_cib_contents: Wrote version 0.99.957 of the CIB to disk (digest: 3ed552fb98aec34e0e16e9dbc98d7e24)
Mar 26 23:26:27 server02 heartbeat: [16581]: info: all clients are now resumed
Mar 26 23:26:28 server02 cib: [16602]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:26:28 server02 cib: [16602]: info: mem_handle_event: no mbr_track info
Mar 26 23:26:28 server02 cib: [16602]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:26:28 server02 cib: [16602]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
Mar 26 23:26:28 server02 cib: [16602]: info: cib_ccm_msg_callback: PEER: server02
Mar 26 23:26:28 server02 cib: [16602]: info: cib_ccm_msg_callback: PEER: server01
Mar 26 23:26:28 server02 crmd: [16606]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm
Mar 26 23:26:28 server02 crmd: [16606]: info: mem_handle_event: no mbr_track info
Mar 26 23:26:28 server02 crmd: [16606]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm
Mar 26 23:26:28 server02 crmd: [16606]: info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
Mar 26 23:26:28 server02 crmd: [16606]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=2)
Mar 26 23:26:28 server02 crmd: [16606]: info: ccm_event_detail: NEW MEMBERSHIP: trans=2, nodes=2, new=1, lost=0 n_idx=0, new_idx=2, old_idx=4
Mar 26 23:26:28 server02 crmd: [16606]: info: ccm_event_detail: CURRENT: server02 [nodeid=1, born=1]
Mar 26 23:26:28 server02 cib: [16602]: info: cib_diff_notify: Local-only Change (client:16606, call: 37): 0.99.957 (ok)
Mar 26 23:26:28 server02 crmd: [16606]: info: ccm_event_detail: CURRENT: server01 [nodeid=0, born=2]
Mar 26 23:26:28 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.99.957 -> 0.99.957
Mar 26 23:26:28 server02 cib: [4838]: info: write_cib_contents: Wrote version 0.99.957 of the CIB to disk (digest: 6de17ec49c44adfb227eecf9edf29bd4)
Mar 26 23:26:28 server02 crmd: [16606]: info: ccm_event_detail: NEW: server01 [nodeid=0, born=2]
Mar 26 23:26:29 server02 crmd: [16606]: ERROR: crmd_ha_msg_callback: Another DC detected: server01 (op=noop)
Mar 26 23:26:29 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_callback ]
Mar 26 23:26:29 server02 crmd: [16606]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:26:29 server02 crmd: [16606]: info: do_election_count_vote: Updated voted hash for server02 to vote
Mar 26 23:26:29 server02 crmd: [16606]: info: do_election_count_vote: Election ignore: our vote (server02)
Mar 26 23:26:29 server02 crmd: [16606]: info: do_election_check: Still waiting on 1 non-votes (2 total)
Mar 26 23:26:30 server02 crmd: [16606]: info: do_election_count_vote: Updated voted hash for server01 to no-vote
Mar 26 23:26:30 server02 crmd: [16606]: info: do_election_count_vote: Election ignore: no-vote from server01
Mar 26 23:26:30 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Mar 26 23:26:30 server02 crmd: [16606]: info: start_subsystem: Starting sub-system "tengine"
Mar 26 23:26:30 server02 crmd: [16606]: WARN: start_subsystem: Client tengine already running as pid 16988
Mar 26 23:26:30 server02 crmd: [16606]: info: start_subsystem: Starting sub-system "pengine"
Mar 26 23:26:30 server02 crmd: [16606]: WARN: start_subsystem: Client pengine already running as pid 16989
Mar 26 23:26:30 server02 crmd: [16606]: info: do_dc_takeover: Taking over DC status for this partition
Mar 26 23:26:30 server02 crmd: [16606]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:26:30 server02 crmd: [16606]: info: do_dc_join_offer_all: join-2: Waiting on 2 outstanding join acks
Mar 26 23:26:30 server02 cib: [16602]: info: cib_process_readwrite: We are now in R/O mode
Mar 26 23:26:30 server02 cib: [16602]: info: cib_process_readwrite: We are now in R/W mode
Mar 26 23:26:30 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:40): 0.99.957 -> 0.99.958 (ok)
Mar 26 23:26:30 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.99.957 -> 0.99.958
Mar 26 23:26:30 server02 cib: [4857]: info: write_cib_contents: Wrote version 0.99.958 of the CIB to disk (digest: 61ec9152a9a19e21d9f6c30ae8663a2b)
Mar 26 23:26:30 server02 crmd: [16606]: info: update_dc: Set DC to server02 (1.0.7)
Mar 26 23:26:31 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 26 23:26:31 server02 crmd: [16606]: info: do_state_transition: All 2 cluster nodes responded to the join offer.
Mar 26 23:26:31 server02 crmd: [16606]: info: do_dc_join_finalize: join-2: Asking server01 for its copy of the CIB
Mar 26 23:26:32 server02 cib: [16602]: info: cib_replace_notify: Replaced: 0.99.958 -> 0.100.960 from (null)
Mar 26 23:26:32 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:42): 0.99.958 -> 0.100.960 (ok)
Mar 26 23:26:32 server02 crmd: [16606]: info: populate_cib_nodes: Requesting the list of configured nodes
Mar 26 23:26:32 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_replace): 0.99.958 -> 0.100.960
Mar 26 23:26:32 server02 tengine: [16988]: info: extract_event: Aborting on transient_attributes changes for 1c847fdd-4f55-4d04-ae67-09ea48ffaff5
Mar 26 23:26:32 server02 tengine: [16988]: info: update_abort_priority: Abort priority upgraded to 1000000
Mar 26 23:26:32 server02 tengine: [16988]: info: process_graph_event: Detected action ip_sample01_monitor_0 from a different transition: 0 vs. 3
Mar 26 23:26:32 server02 tengine: [16988]: info: te_update_diff: Aborting on transient_attributes deletions
Mar 26 23:26:32 server02 cib: [4904]: info: write_cib_contents: Wrote version 0.100.960 of the CIB to disk (digest: f778888e1598aac1558e50eb961bba22)
Mar 26 23:26:32 server02 crmd: [16606]: notice: populate_cib_nodes: Node: server02 (uuid: 41b08ac0-6f59-4771-ad28-45b18ad47ce4)
Mar 26 23:26:33 server02 crmd: [16606]: notice: populate_cib_nodes: Node: server01 (uuid: 1c847fdd-4f55-4d04-ae67-09ea48ffaff5)
Mar 26 23:26:33 server02 attrd: [16605]: info: attrd_local_callback: Sending full refresh
Mar 26 23:26:33 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_FINALIZE_JOIN -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ]
Mar 26 23:26:33 server02 crmd: [16606]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:26:33 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:44): 0.100.960 -> 0.100.961 (ok)
Mar 26 23:26:33 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.100.960 -> 0.100.961
Mar 26 23:26:33 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:45): 0.100.961 -> 0.100.962 (ok)
Mar 26 23:26:33 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.100.961 -> 0.100.962
Mar 26 23:26:33 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:46): 0.100.962 -> 0.101.963 (ok)
Mar 26 23:26:33 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_bump): 0.100.962 -> 0.101.963
Mar 26 23:26:33 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:47): 0.101.963 -> 0.101.964 (ok)
Mar 26 23:26:33 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.101.963 -> 0.101.964
Mar 26 23:26:33 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:48): 0.101.964 -> 0.101.965 (ok)
Mar 26 23:26:33 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.101.964 -> 0.101.965
Mar 26 23:26:33 server02 cib: [4905]: info: write_cib_contents: Wrote version 0.101.965 of the CIB to disk (digest: 4b29679f39cfa464fc1b6cfef3945da1)
Mar 26 23:26:33 server02 crmd: [16606]: info: do_election_count_vote: Updated voted hash for server02 to vote
Mar 26 23:26:33 server02 crmd: [16606]: info: do_election_count_vote: Election ignore: our vote (server02)
Mar 26 23:26:33 server02 crmd: [16606]: info: do_election_check: Still waiting on 1 non-votes (2 total)
Mar 26 23:26:34 server02 crmd: [16606]: info: do_election_count_vote: Updated voted hash for server01 to no-vote
Mar 26 23:26:34 server02 crmd: [16606]: info: do_election_count_vote: Election ignore: no-vote from server01
Mar 26 23:26:34 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Mar 26 23:26:34 server02 crmd: [16606]: info: start_subsystem: Starting sub-system "tengine"
Mar 26 23:26:34 server02 crmd: [16606]: WARN: start_subsystem: Client tengine already running as pid 16988
Mar 26 23:26:34 server02 crmd: [16606]: info: start_subsystem: Starting sub-system "pengine"
Mar 26 23:26:34 server02 crmd: [16606]: WARN: start_subsystem: Client pengine already running as pid 16989
Mar 26 23:26:34 server02 crmd: [16606]: info: do_dc_takeover: Taking over DC status for this partition
Mar 26 23:26:34 server02 cib: [16602]: info: cib_process_readwrite: We are now in R/O mode
Mar 26 23:26:34 server02 crmd: [16606]: info: update_dc: Set DC to <null> (<null>)
Mar 26 23:26:34 server02 crmd: [16606]: info: do_dc_join_offer_all: join-3: Waiting on 2 outstanding join acks
Mar 26 23:26:34 server02 cib: [16602]: info: cib_process_readwrite: We are now in R/W mode
Mar 26 23:26:34 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:51): 0.101.965 -> 0.101.966 (ok)
Mar 26 23:26:34 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.101.965 -> 0.101.966
Mar 26 23:26:34 server02 cib: [4906]: info: write_cib_contents: Wrote version 0.101.966 of the CIB to disk (digest: f6345a73547f8c0b7e7c2b5dcd86866a)
Mar 26 23:26:34 server02 crmd: [16606]: info: update_dc: Set DC to server02 (1.0.7)
Mar 26 23:26:35 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 26 23:26:35 server02 crmd: [16606]: info: do_state_transition: All 2 cluster nodes responded to the join offer.
Mar 26 23:26:35 server02 attrd: [16605]: info: attrd_local_callback: Sending full refresh
Mar 26 23:26:35 server02 cib: [16602]: info: sync_our_cib: Syncing CIB to all peers
Mar 26 23:26:35 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:54): 0.101.966 -> 0.101.967 (ok)
Mar 26 23:26:35 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.101.966 -> 0.101.967
Mar 26 23:26:35 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:55): 0.101.967 -> 0.102.968 (ok)
Mar 26 23:26:35 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_bump): 0.101.967 -> 0.102.968
Mar 26 23:26:35 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:56): 0.102.968 -> 0.102.969 (ok)
Mar 26 23:26:35 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.102.968 -> 0.102.969
Mar 26 23:26:35 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:57): 0.102.969 -> 0.102.970 (ok)
Mar 26 23:26:35 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.102.969 -> 0.102.970
Mar 26 23:26:35 server02 cib: [4938]: info: write_cib_contents: Wrote version 0.102.970 of the CIB to disk (digest: 954b7768896a9fe88bbc66b061b8c8ed)
Mar 26 23:26:35 server02 crmd: [16606]: info: update_dc: Set DC to server02 (1.0.7)
Mar 26 23:26:35 server02 crmd: [16606]: info: append_restart_list: Resource ip_sample01 does not support reloads
Mar 26 23:26:35 server02 crmd: [16606]: info: do_dc_join_ack: join-3: Updating node state to member for server01)
Mar 26 23:26:35 server02 crmd: [16606]: info: do_dc_join_ack: join-3: Updating node state to member for server02)
Mar 26 23:26:35 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:58): 0.102.970 -> 0.102.971 (ok)
Mar 26 23:26:35 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 26 23:26:35 server02 crmd: [16606]: info: do_state_transition: All 2 cluster nodes are eligable to run resources.
Mar 26 23:26:35 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.102.970 -> 0.102.971
Mar 26 23:26:35 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:59): 0.102.971 -> 0.102.972 (ok)
Mar 26 23:26:35 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.102.971 -> 0.102.972
Mar 26 23:26:35 server02 tengine: [16988]: info: process_graph_event: Detected action ip_sample01_monitor_0 from a different transition: 0 vs. 3
Mar 26 23:26:35 server02 tengine: [16988]: info: process_graph_event: Action ip_sample01_start_0 arrived after a completed transition
Mar 26 23:26:35 server02 tengine: [16988]: info: process_graph_event: Action ip_sample01_monitor_10000 arrived after a completed transition
Mar 26 23:26:35 server02 cib: [4939]: info: write_cib_contents: Wrote version 0.102.972 of the CIB to disk (digest: 6b62310714fb54bef84fa87506fb93e8)
Mar 26 23:26:35 server02 pengine: [16989]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" cib_feature_revision="1.3" ignore_dtd="false" num_peers="2" ccm_transition="2" generated="true" dc_uuid="41b08ac0-6f59-4771-ad28-45b18ad47ce4" epoch="102" num_updates="972"/>
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value 'false' for cluster option 'stonith-enabled'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
Mar 26 23:26:35 server02 pengine: [16989]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
Mar 26 23:26:36 server02 pengine: [16989]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
Mar 26 23:26:36 server02 pengine: [16989]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
Mar 26 23:26:36 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
Mar 26 23:26:36 server02 pengine: [16989]: info: determine_online_status: Node server02 is online
Mar 26 23:26:36 server02 pengine: [16989]: info: determine_online_status: Node server01 is online
Mar 26 23:26:36 server02 pengine: [16989]: ERROR: native_add_running: Resource ocf::IPaddr:ip_sample01 appears to be active on 2 nodes.
Mar 26 23:26:36 server02 pengine: [16989]: ERROR: See http://linux-ha.org/v2/faq/resource_too_active for more information.
Mar 26 23:26:36 server02 pengine: [16989]: info: native_print: ip_sample01 (heartbeat::ocf:IPaddr)
Mar 26 23:26:36 server02 pengine: [16989]: info: native_print: 0 : server02
Mar 26 23:26:36 server02 pengine: [16989]: info: native_print: 1 : server01
Mar 26 23:26:36 server02 pengine: [16989]: WARN: native_assign_node: 2 nodes with equal score (+INFINITY) for running the listed resources (chose server02):
Mar 26 23:26:36 server02 pengine: [16989]: ERROR: native_create_actions: Attempting recovery of resource ip_sample01
Mar 26 23:26:36 server02 pengine: [16989]: notice: StopRsc: server02 Stop ip_sample01
Mar 26 23:26:36 server02 pengine: [16989]: notice: StopRsc: server01 Stop ip_sample01
Mar 26 23:26:36 server02 pengine: [16989]: notice: StartRsc: server02 Start ip_sample01
Mar 26 23:26:36 server02 pengine: [16989]: notice: Recurring: server02 ip_sample01_monitor_10000
Mar 26 23:26:36 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Mar 26 23:26:36 server02 pengine: [16989]: ERROR: process_pe_message: Transition 4: ERRORs found during PE processing. PEngine Input stored in: /var/lib/heartbeat/pengine/pe-error-0.bz2
Mar 26 23:26:36 server02 tengine: [16988]: info: unpack_graph: Unpacked transition 4: 5 actions in 5 synapses
Mar 26 23:26:36 server02 tengine: [16988]: info: send_rsc_command: Initiating action 6: ip_sample01_stop_0 on server02
Mar 26 23:26:36 server02 tengine: [16988]: info: send_rsc_command: Initiating action 7: ip_sample01_stop_0 on server01
Mar 26 23:26:36 server02 crmd: [16606]: info: do_lrm_rsc_op: Performing op=ip_sample01_stop_0 key=6:4:38d96444-0d9f-4b9b-bf00-d79d4dfbd3ae)
Mar 26 23:26:36 server02 tengine: [16988]: info: send_rsc_command: Initiating action 4: probe_complete on server02
Mar 26 23:26:36 server02 crmd: [16606]: WARN: process_lrm_event: LRM operation ip_sample01_monitor_10000 (call=4, rc=-2) Cancelled
Mar 26 23:26:36 server02 lrmd: [16603]: info: RA output: (ip_sample01:stop:stderr) SIOCDELRT: No such process
Mar 26 23:26:36 server02 IPaddr[4940]: INFO: /sbin/ifconfig bond0:0 192.168.1.1 down
Mar 26 23:26:36 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:65): 0.102.972 -> 0.102.973 (ok)
Mar 26 23:26:36 server02 crmd: [16606]: info: process_lrm_event: LRM operation ip_sample01_stop_0 (call=6, rc=0) complete
Mar 26 23:26:36 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_modify): 0.102.972 -> 0.102.973
Mar 26 23:26:36 server02 cib: [4956]: info: write_cib_contents: Wrote version 0.102.973 of the CIB to disk (digest: 45bd5fa08447f3982705f4eba9e2941c)
Mar 26 23:26:36 server02 tengine: [16988]: info: extract_event: Aborting on transient_attributes changes for 41b08ac0-6f59-4771-ad28-45b18ad47ce4
Mar 26 23:26:36 server02 cib: [16602]: info: cib_diff_notify: Update (client: 16606, call:66): 0.102.973 -> 0.102.974 (ok)
Mar 26 23:26:36 server02 tengine: [16988]: info: update_abort_priority: Abort priority upgraded to 1000000
Mar 26 23:26:36 server02 cib: [4957]: info: write_cib_contents: Wrote version 0.102.974 of the CIB to disk (digest: 1ed2d46e50cd0bbf24ab1e16060f55a2)
Mar 26 23:26:36 server02 tengine: [16988]: info: update_abort_priority: Abort action 0 superceeded by 2
Mar 26 23:26:36 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.102.973 -> 0.102.974
Mar 26 23:26:36 server02 tengine: [16988]: info: match_graph_event: Action ip_sample01_stop_0 (6) confirmed on 41b08ac0-6f59-4771-ad28-45b18ad47ce4
Mar 26 23:26:37 server02 cib: [16602]: info: cib_diff_notify: Update (client: 31276, call:44): 0.102.974 -> 0.102.975 (ok)
Mar 26 23:26:37 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.102.974 -> 0.102.975
Mar 26 23:26:37 server02 tengine: [16988]: info: match_graph_event: Action ip_sample01_stop_0 (7) confirmed on 1c847fdd-4f55-4d04-ae67-09ea48ffaff5
Mar 26 23:26:37 server02 tengine: [16988]: info: run_graph: ====================================================
Mar 26 23:26:37 server02 tengine: [16988]: notice: run_graph: Transition 4: (Complete=3, Pending=0, Fired=0, Skipped=2, Incomplete=0)
Mar 26 23:26:37 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ]
Mar 26 23:26:37 server02 crmd: [16606]: info: do_state_transition: All 2 cluster nodes are eligable to run resources.
Mar 26 23:26:37 server02 cib: [4960]: info: write_cib_contents: Wrote version 0.102.975 of the CIB to disk (digest: 608655be67928eda3be8b4c422c9cf43)
Mar 26 23:26:37 server02 pengine: [16989]: info: log_data_element: process_pe_message: [generation] <cib admin_epoch="0" have_quorum="true" cib_feature_revision="1.3" ignore_dtd="false" num_peers="2" ccm_transition="2" generated="true" dc_uuid="41b08ac0-6f59-4771-ad28-45b18ad47ce4" epoch="102" num_updates="975"/>
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value 'false' for cluster option 'stonith-enabled'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max'
Mar 26 23:26:37 server02 pengine: [16989]: notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing'
Mar 26 23:26:37 server02 pengine: [16989]: info: determine_online_status: Node server02 is online
Mar 26 23:26:37 server02 pengine: [16989]: info: determine_online_status: Node server01 is online
Mar 26 23:26:37 server02 pengine: [16989]: info: native_print: ip_sample01 (heartbeat::ocf:IPaddr): Stopped
Mar 26 23:26:37 server02 pengine: [16989]: notice: StartRsc: server01 Start ip_sample01
Mar 26 23:26:37 server02 pengine: [16989]: notice: Recurring: server01 ip_sample01_monitor_10000
Mar 26 23:26:38 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Mar 26 23:26:38 server02 pengine: [16989]: info: process_pe_message: Transition 5: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-329.bz2
Mar 26 23:26:38 server02 tengine: [16988]: info: unpack_graph: Unpacked transition 5: 2 actions in 2 synapses
Mar 26 23:26:38 server02 tengine: [16988]: info: send_rsc_command: Initiating action 4: ip_sample01_start_0 on server01
Mar 26 23:26:39 server02 cib: [16602]: info: cib_diff_notify: Update (client: 31276, call:45): 0.102.975 -> 0.102.976 (ok)
Mar 26 23:26:39 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.102.975 -> 0.102.976
Mar 26 23:26:39 server02 tengine: [16988]: info: match_graph_event: Action ip_sample01_start_0 (4) confirmed on 1c847fdd-4f55-4d04-ae67-09ea48ffaff5
Mar 26 23:26:39 server02 tengine: [16988]: info: send_rsc_command: Initiating action 5: ip_sample01_monitor_10000 on server01
Mar 26 23:26:39 server02 cib: [4980]: info: write_cib_contents: Wrote version 0.102.976 of the CIB to disk (digest: 161c849592bfc6262e0db96f3b4e8c36)
Mar 26 23:26:40 server02 cib: [16602]: info: cib_diff_notify: Update (client: 31276, call:46): 0.102.976 -> 0.102.977 (ok)
Mar 26 23:26:40 server02 tengine: [16988]: info: te_update_diff: Processing diff (cib_update): 0.102.976 -> 0.102.977
Mar 26 23:26:40 server02 tengine: [16988]: info: match_graph_event: Action ip_sample01_monitor_10000 (5) confirmed on 1c847fdd-4f55-4d04-ae67-09ea48ffaff5
Mar 26 23:26:40 server02 tengine: [16988]: info: run_graph: Transition 5: (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0)
Mar 26 23:26:40 server02 tengine: [16988]: info: notify_crmd: Transition 5 status: te_complete - <null>
Mar 26 23:26:40 server02 crmd: [16606]: info: do_state_transition: server02: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ]
Mar 26 23:26:40 server02 cib: [4981]: info: write_cib_contents: Wrote version 0.102.977 of the CIB to disk (digest: 38f7498a262c2c842d7de0f6af81236b)
------------------------------------------------------------------------
$B0J>e!"59$7$/$*4j$$CW$7$^$9!#(B
_______________________________________________
Linux-ha-japan mailing list
Linux-ha-japan [at] lists
http://lists.sourceforge.jp/mailman/listinfo/linux-ha-japan
|
