Mailing List Archive: Linux-HA: Users

heartbeat - execute a script on a running node when the other node is back?

Index | Next | Previous | View Threaded

mangoo at wpkg

Nov 15, 2009, 12:09 PM

Post #1 of 10 (1725 views)
Permalink

heartbeat - execute a script on a running node when the other node is back?

I have two nodes, node_1 and node_2.

node_2 was down, but is now up.

How can I execute a custom script on node_1 when it detects that node_2
is back?

--
Tomasz Chmielewski
http://wpkg.org
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

dejanmm at fastmail

Nov 16, 2009, 2:44 AM

Post #2 of 10 (1669 views)
Permalink

Re: heartbeat - execute a script on a running node when the other node is back? [In reply to]

Hi,

On Sun, Nov 15, 2009 at 09:09:53PM +0100, Tomasz Chmielewski wrote:
> I have two nodes, node_1 and node_2.
>
> node_2 was down, but is now up.
>
>
> How can I execute a custom script on node_1 when it detects that node_2
> is back?

That's not possible. What would you want to with that script?

Thanks,

Dejan

>
> --
> Tomasz Chmielewski
> http://wpkg.org
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

tappel at eso

Nov 16, 2009, 2:55 AM

Post #3 of 10 (1663 views)
Permalink

Re: heartbeat - execute a script on a running node when the other node is back? [In reply to]

On 11/15/2009 09:09 PM, Tomasz Chmielewski wrote:
> I have two nodes, node_1 and node_2.
>
> node_2 was down, but is now up.
>
>
> How can I execute a custom script on node_1 when it detects that node_2
> is back?
>
>

This is a little off the heartbeat list I guess, but we use Nagios to
monitor our heartbeat clusters and you can use the eventhandler from
nagios to execute scripts when there is a state change.

I don't think that you would want to install nagios just for this, but
in case you are running Nagios or something similar already you might
try this approach.

Bye,
Tobi
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

mangoo at wpkg

Nov 16, 2009, 4:28 AM

Post #4 of 10 (1652 views)
Permalink

Re: heartbeat - execute a script on a running node when the other node is back? [In reply to]

Dejan Muhamedagic wrote:
> Hi,
>
> On Sun, Nov 15, 2009 at 09:09:53PM +0100, Tomasz Chmielewski wrote:
>> I have two nodes, node_1 and node_2.
>>
>> node_2 was down, but is now up.
>>
>>
>> How can I execute a custom script on node_1 when it detects that node_2
>> is back?
>
> That's not possible. What would you want to with that script?

I have two PostgreSQL servers running; pgpool-ii is started by Heartbeat
to distribute the load (reads) among two servers and to send writes to
both servers.

When one PostgreSQL server fails, the setup will still work fine. When
the failed PostgreSQL instance is back, the data should be first
"synchronized" from the running PostgreSQL server to a server which was
failed a while ago.

It is best if such a script could be started by Heartbeat running on the
active node, as soon as it detects that the other node is back.

--
Tomasz Chmielewski
http://wpkg.org
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

dk at in-telegence

Nov 16, 2009, 5:39 AM

Post #5 of 10 (1657 views)
Permalink

Re: heartbeat - execute a script on a running node when the other node is back? [In reply to]

Tomasz Chmielewski wrote:
> Dejan Muhamedagic wrote:
>> Hi,
>>
>> On Sun, Nov 15, 2009 at 09:09:53PM +0100, Tomasz Chmielewski wrote:
>>> I have two nodes, node_1 and node_2.
>>>
>>> node_2 was down, but is now up.
>>>
>>>
>>> How can I execute a custom script on node_1 when it detects that node_2
>>> is back?
>> That's not possible. What would you want to with that script?
>
> I have two PostgreSQL servers running; pgpool-ii is started by Heartbeat
> to distribute the load (reads) among two servers and to send writes to
> both servers.
>
> When one PostgreSQL server fails, the setup will still work fine. When
> the failed PostgreSQL instance is back, the data should be first
> "synchronized" from the running PostgreSQL server to a server which was
> failed a while ago.
>
> It is best if such a script could be started by Heartbeat running on the
> active node, as soon as it detects that the other node is back.

If you need such thing - I'd personally be most comfortable with not
starting the cluster at boot time. Then you can do whatever you need to
do and then - when you _know_ everything is right, the script is done
etc. - start the cluster software.

Just my personal preference.

Regards
Dominik
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

dejanmm at fastmail

Nov 16, 2009, 9:11 AM

Post #6 of 10 (1649 views)
Permalink

Re: heartbeat - execute a script on a running node when the other node is back? [In reply to]

Hi,

On Mon, Nov 16, 2009 at 02:39:52PM +0100, Dominik Klein wrote:
> Tomasz Chmielewski wrote:
> > Dejan Muhamedagic wrote:
> >> Hi,
> >>
> >> On Sun, Nov 15, 2009 at 09:09:53PM +0100, Tomasz Chmielewski wrote:
> >>> I have two nodes, node_1 and node_2.
> >>>
> >>> node_2 was down, but is now up.
> >>>
> >>>
> >>> How can I execute a custom script on node_1 when it detects that node_2
> >>> is back?
> >> That's not possible. What would you want to with that script?
> >
> > I have two PostgreSQL servers running; pgpool-ii is started by Heartbeat
> > to distribute the load (reads) among two servers and to send writes to
> > both servers.
> >
> > When one PostgreSQL server fails, the setup will still work fine. When
> > the failed PostgreSQL instance is back, the data should be first
> > "synchronized" from the running PostgreSQL server to a server which was
> > failed a while ago.
> >
> > It is best if such a script could be started by Heartbeat running on the
> > active node, as soon as it detects that the other node is back.
>
> If you need such thing - I'd personally be most comfortable with not
> starting the cluster at boot time. Then you can do whatever you need to
> do and then - when you _know_ everything is right, the script is done
> etc. - start the cluster software.
>
> Just my personal preference.

It would be mine too. Nothing wrong to have a script for that,
but best to run it by hand so that you can check the database.
Besides, your nodes shouldn't be disappearing that often.

Thanks,

Dejan

> Regards
> Dominik
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

mangoo at wpkg

Nov 19, 2009, 3:33 AM

Post #7 of 10 (1590 views)
Permalink

Re: heartbeat - execute a script on a running node when the other node is back? [In reply to]

Dejan Muhamedagic wrote:

>>> When one PostgreSQL server fails, the setup will still work fine. When
>>> the failed PostgreSQL instance is back, the data should be first
>>> "synchronized" from the running PostgreSQL server to a server which was
>>> failed a while ago.
>>>
>>> It is best if such a script could be started by Heartbeat running on the
>>> active node, as soon as it detects that the other node is back.
>> If you need such thing - I'd personally be most comfortable with not
>> starting the cluster at boot time. Then you can do whatever you need to
>> do and then - when you _know_ everything is right, the script is done
>> etc. - start the cluster software.
>>
>> Just my personal preference.
>
> It would be mine too. Nothing wrong to have a script for that,
> but best to run it by hand so that you can check the database.
> Besides, your nodes shouldn't be disappearing that often.

OK, then I would like to be at least notified that heartbeat lost
connection with the peer, and that it re-established the connection again.

Can I at least have that with heartbeat?

Monitoring (i.e. nagios) will usually only check the server every 5
minutes or so, so a "spontaneous" reboot (or restart on kernel panic)
may not be detected by that (yes, I can send a mail via startup scripts,
but I'd rather heartbeat told me about its problems).

--
Tomasz Chmielewski
http://wpkg.org
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

dejanmm at fastmail

Nov 19, 2009, 3:45 AM

Post #8 of 10 (1595 views)
Permalink

Re: heartbeat - execute a script on a running node when the other node is back? [In reply to]

Hi,

On Thu, Nov 19, 2009 at 12:33:09PM +0100, Tomasz Chmielewski wrote:
> Dejan Muhamedagic wrote:
>
> >>> When one PostgreSQL server fails, the setup will still work fine. When
> >>> the failed PostgreSQL instance is back, the data should be first
> >>> "synchronized" from the running PostgreSQL server to a server which was
> >>> failed a while ago.
> >>>
> >>> It is best if such a script could be started by Heartbeat running on the
> >>> active node, as soon as it detects that the other node is back.
> >> If you need such thing - I'd personally be most comfortable with not
> >> starting the cluster at boot time. Then you can do whatever you need to
> >> do and then - when you _know_ everything is right, the script is done
> >> etc. - start the cluster software.
> >>
> >> Just my personal preference.
> >
> > It would be mine too. Nothing wrong to have a script for that,
> > but best to run it by hand so that you can check the database.
> > Besides, your nodes shouldn't be disappearing that often.
>
> OK, then I would like to be at least notified that heartbeat lost
> connection with the peer, and that it re-established the connection again.
>
> Can I at least have that with heartbeat?
>

You can get emails or snmp alerts. Take a look at crm_mon.

Thanks,

Dejan
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

alanr at unix

Nov 20, 2009, 4:23 AM

Post #9 of 10 (1543 views)
Permalink

Re: heartbeat - execute a script on a running node when the other node is back? [In reply to]

Actually,

The heartbeat API will allow you to get notified when you lose a
single link or it recovers - not just the whole node.

Quoting Dejan Muhamedagic <dejanmm [at] fastmail>:

> Hi,
>
> On Thu, Nov 19, 2009 at 12:33:09PM +0100, Tomasz Chmielewski wrote:
>> Dejan Muhamedagic wrote:
>>
>> >>> When one PostgreSQL server fails, the setup will still work fine. When
>> >>> the failed PostgreSQL instance is back, the data should be first
>> >>> "synchronized" from the running PostgreSQL server to a server which was
>> >>> failed a while ago.
>> >>>
>> >>> It is best if such a script could be started by Heartbeat running on the
>> >>> active node, as soon as it detects that the other node is back.
>> >> If you need such thing - I'd personally be most comfortable with not
>> >> starting the cluster at boot time. Then you can do whatever you need to
>> >> do and then - when you _know_ everything is right, the script is done
>> >> etc. - start the cluster software.
>> >>
>> >> Just my personal preference.
>> >
>> > It would be mine too. Nothing wrong to have a script for that,
>> > but best to run it by hand so that you can check the database.
>> > Besides, your nodes shouldn't be disappearing that often.
>>
>> OK, then I would like to be at least notified that heartbeat lost
>> connection with the peer, and that it re-established the connection again.
>>
>> Can I at least have that with heartbeat?
>>
>
> You can get emails or snmp alerts. Take a look at crm_mon.
>
> Thanks,
>
> Dejan
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>

_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

foxycode at yahoo

Nov 30, 2009, 8:43 AM

Post #10 of 10 (1291 views)
Permalink

Re: heartbeat - execute a script on a running node when the other node is back? [In reply to]

On Thu, Nov 19, 2009 at 12:33:09PM +0100, Tomasz Chmielewski wrote:
>
> When one PostgreSQL server fails, the setup will still work fine. When
> the failed PostgreSQL instance is back, the data should be first
> "synchronized" from the running PostgreSQL server to a server which was
> failed a while ago.
>

Shouldn't this synchronization be, instead, initiated by the node that came back up ?

_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads