Mailing List Archive: Linux-HA: Users

Failover during server freeze

Index | Next | Previous | View Threaded

lars at larsjohansen

Nov 26, 2009, 5:37 AM

Post #1 of 9 (1322 views)
Permalink

Failover during server freeze

Hi,

I have a 2 cluster setup running heartbeat 2.1.3.

Im running a active/passive setup, where I have mysql in master<-> master
replication, stored on each nodes filesystem, and I have a DRBD drive with
NFS.

Pulling cables, shuttding down server etc. works very well.. the "alive"
server takes over..

However Ive tried to simulate a "server freeze" by create a forkbomb on the
active server, after doing that, I cannot log in with ssh, access nfs or
mysql, ping works, and I guess since the kernel responds, heartbeat wont
switch over.

How can I setup heartbeat so it detects if a server is frozen?

Greetings,

LArs
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

dejanmm at fastmail

Nov 27, 2009, 3:01 AM

Post #2 of 9 (1223 views)
Permalink

Re: Failover during server freeze [In reply to]

Hi,

On Thu, Nov 26, 2009 at 02:37:10PM +0100, Lars Johansen wrote:
> Hi,
>
> I have a 2 cluster setup running heartbeat 2.1.3.
>
> Im running a active/passive setup, where I have mysql in master<-> master
> replication, stored on each nodes filesystem, and I have a DRBD drive with
> NFS.
>
> Pulling cables, shuttding down server etc. works very well.. the "alive"
> server takes over..
>
> However Ive tried to simulate a "server freeze" by create a forkbomb on the
> active server, after doing that, I cannot log in with ssh, access nfs or
> mysql, ping works, and I guess since the kernel responds, heartbeat wont
> switch over.

It's not the kernel, but the heartbeat process which sends
messages. The only difference to other processes is that it runs
at the highest priority, so it may be possible to have
heartbeats, but no services...

> How can I setup heartbeat so it detects if a server is frozen?

... What about resource monitoring? If the node is overwhelmed,
as you say it is, then some monitors should timeout or fail, right?

Thanks,

Dejan

>
> Greetings,
>
> LArs
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

lars at larsjohansen

Nov 27, 2009, 3:59 AM

Post #3 of 9 (1221 views)
Permalink

Re: Failover during server freeze [In reply to]

Hi,

I am not having any monitors, should I use mon for this?

Greetings,

Lars

2009/11/27 Dejan Muhamedagic <dejanmm [at] fastmail>

> Hi,
>
> On Thu, Nov 26, 2009 at 02:37:10PM +0100, Lars Johansen wrote:
> > Hi,
> >
> > I have a 2 cluster setup running heartbeat 2.1.3.
> >
> > Im running a active/passive setup, where I have mysql in master<-> master
> > replication, stored on each nodes filesystem, and I have a DRBD drive
> with
> > NFS.
> >
> > Pulling cables, shuttding down server etc. works very well.. the "alive"
> > server takes over..
> >
> > However Ive tried to simulate a "server freeze" by create a forkbomb on
> the
> > active server, after doing that, I cannot log in with ssh, access nfs or
> > mysql, ping works, and I guess since the kernel responds, heartbeat wont
> > switch over.
>
> It's not the kernel, but the heartbeat process which sends
> messages. The only difference to other processes is that it runs
> at the highest priority, so it may be possible to have
> heartbeats, but no services...
>
> > How can I setup heartbeat so it detects if a server is frozen?
>
> ... What about resource monitoring? If the node is overwhelmed,
> as you say it is, then some monitors should timeout or fail, right?
>
> Thanks,
>
> Dejan
>
>
> >
> > Greetings,
> >
> > LArs
> > _______________________________________________
> > Linux-HA mailing list
> > Linux-HA [at] lists
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > See also: http://linux-ha.org/ReportingProblems
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

dejanmm at fastmail

Nov 27, 2009, 5:44 AM

Post #4 of 9 (1229 views)
Permalink

Re: Failover during server freeze [In reply to]

Hi,

On Fri, Nov 27, 2009 at 12:59:33PM +0100, Lars Johansen wrote:
> Hi,
>
> I am not having any monitors, should I use mon for this?

No, I meant monitor operations on resources, i.e. to use the
resource agents monitor operations which are configured in the
CIB.

Thanks,

Dejan

>
> Greetings,
>
> Lars
>
> 2009/11/27 Dejan Muhamedagic <dejanmm [at] fastmail>
>
> > Hi,
> >
> > On Thu, Nov 26, 2009 at 02:37:10PM +0100, Lars Johansen wrote:
> > > Hi,
> > >
> > > I have a 2 cluster setup running heartbeat 2.1.3.
> > >
> > > Im running a active/passive setup, where I have mysql in master<-> master
> > > replication, stored on each nodes filesystem, and I have a DRBD drive
> > with
> > > NFS.
> > >
> > > Pulling cables, shuttding down server etc. works very well.. the "alive"
> > > server takes over..
> > >
> > > However Ive tried to simulate a "server freeze" by create a forkbomb on
> > the
> > > active server, after doing that, I cannot log in with ssh, access nfs or
> > > mysql, ping works, and I guess since the kernel responds, heartbeat wont
> > > switch over.
> >
> > It's not the kernel, but the heartbeat process which sends
> > messages. The only difference to other processes is that it runs
> > at the highest priority, so it may be possible to have
> > heartbeats, but no services...
> >
> > > How can I setup heartbeat so it detects if a server is frozen?
> >
> > ... What about resource monitoring? If the node is overwhelmed,
> > as you say it is, then some monitors should timeout or fail, right?
> >
> > Thanks,
> >
> > Dejan
> >
> >
> > >
> > > Greetings,
> > >
> > > LArs
> > > _______________________________________________
> > > Linux-HA mailing list
> > > Linux-HA [at] lists
> > > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > > See also: http://linux-ha.org/ReportingProblems
> > _______________________________________________
> > Linux-HA mailing list
> > Linux-HA [at] lists
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > See also: http://linux-ha.org/ReportingProblems
> >
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

michalko.system at a-i-p

Nov 30, 2009, 4:11 AM

Post #5 of 9 (1192 views)
Permalink

Re: Failover during server freeze [In reply to]

Hi Lars,

we have the same HA-version, without mysql though. Could you send me pls
the "forkbomb" to test it on our cluster?

TIA!

Nikita Michalko

Am Donnerstag, 26. November 2009 14:37 schrieb Lars Johansen:
> Hi,
>
> I have a 2 cluster setup running heartbeat 2.1.3.
>
> Im running a active/passive setup, where I have mysql in master<-> master
> replication, stored on each nodes filesystem, and I have a DRBD drive with
> NFS.
>
> Pulling cables, shuttding down server etc. works very well.. the "alive"
> server takes over..
>
> However Ive tried to simulate a "server freeze" by create a forkbomb on the
> active server, after doing that, I cannot log in with ssh, access nfs or
> mysql, ping works, and I guess since the kernel responds, heartbeat wont
> switch over.
>
> How can I setup heartbeat so it detects if a server is frozen?
>
>
> Greetings,
>
> LArs
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Johan.Verrept at able

Nov 30, 2009, 5:11 AM

Post #6 of 9 (1197 views)
Permalink

Re: Failover during server freeze [In reply to]

On Mon, 2009-11-30 at 13:11 +0100, Nikita Michalko wrote:
> Hi Lars,
>
> we have the same HA-version, without mysql though. Could you send me pls
> the "forkbomb" to test it on our cluster?

#!/bin/bash

while (true); do bash $0; done;

_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

michalko.system at a-i-p

Nov 30, 2009, 7:44 AM

Post #7 of 9 (1192 views)
Permalink

Re: Failover during server freeze [In reply to]

Hi Johan,

thanks for that - so easy can one "disable" the server ;-)

Nikita Michalko

Am Montag, 30. November 2009 14:11 schrieb Johan Verrept:
> On Mon, 2009-11-30 at 13:11 +0100, Nikita Michalko wrote:
> > Hi Lars,
> >
> > we have the same HA-version, without mysql though. Could you send me
> > pls the "forkbomb" to test it on our cluster?
>
> #!/bin/bash
>
> while (true); do bash $0; done;
>
>
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

ibnerazi at yahoo

Nov 30, 2009, 7:48 AM

Post #8 of 9 (1199 views)
Permalink

Re: Failover during server freeze [In reply to]

> so easy can one "disable" the server ;-)
Yes and No.

you can prevent fork bomb attacks via configuring 'limits.conf'
see http://www.cyberciti.biz/tips/linux-limiting-user-process.html

--- On Mon, 11/30/09, Nikita Michalko <michalko.system [at] a-i-p> wrote:

From: Nikita Michalko <michalko.system [at] a-i-p>
Subject: Re: [Linux-HA] Failover during server freeze
To: "General Linux-HA mailing list" <linux-ha [at] lists>
Date: Monday, November 30, 2009, 7:44 AM

Hi Johan,

thanks for that - so easy can one "disable" the server ;-)

Nikita Michalko

Am Montag, 30. November 2009 14:11 schrieb Johan Verrept:
> On Mon, 2009-11-30 at 13:11 +0100, Nikita Michalko wrote:
> > Hi Lars,
> >
> >� we have the same HA-version, without mysql� though. Could� you send me
> > pls the "forkbomb" to test it on our cluster?
>
> #!/bin/bash
>
> while (true); do bash $0; done;
>
>
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

lj at ez

Dec 2, 2009, 2:10 AM

Post #9 of 9 (1149 views)
Permalink

Re: Failover during server freeze [In reply to]

Is there a solution to make Heartbeat failover in situations like this?
without using crm?

2009/11/30 Nikita Michalko <michalko.system [at] a-i-p>

> Hi Johan,
>
> thanks for that - so easy can one "disable" the server ;-)
>
>
> Nikita Michalko
>
>
> Am Montag, 30. November 2009 14:11 schrieb Johan Verrept:
> > On Mon, 2009-11-30 at 13:11 +0100, Nikita Michalko wrote:
> > > Hi Lars,
> > >
> > > we have the same HA-version, without mysql though. Could you send me
> > > pls the "forkbomb" to test it on our cluster?
> >
> > #!/bin/bash
> >
> > while (true); do bash $0; done;
> >
> >
> >
> > _______________________________________________
> > Linux-HA mailing list
> > Linux-HA [at] lists
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > See also: http://linux-ha.org/ReportingProblems
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads