Mailing List Archive: Linux-HA: Users

Re: [patch] clock_t wrapped around causingfalse resourcestart failure

Index | Next | Previous | View Threaded

Simon.Tavanyar at stratus

Jul 1, 2009, 7:22 AM

Post #1 of 8 (557 views)
Permalink

Re: [patch] clock_t wrapped around causingfalse resourcestart failure

Aahh ... so it's measured after reboot. That makes sense.
Thanks, Lars.

Should I wait for you to log a bugzilla on this?

-----Original Message-----
From: linux-ha-bounces[at]lists.linux-ha.org
[mailto:linux-ha-bounces[at]lists.linux-ha.org] On Behalf Of Lars Ellenberg
Sent: Wednesday, July 01, 2009 6:28 AM
To: linux-ha[at]lists.linux-ha.org
Subject: Re: [Linux-HA] [patch] clock_t wrapped around causingfalse
resourcestart failure

On Tue, Jun 30, 2009 at 10:03:29AM -0400, Tavanyar, Simon wrote:
> Hi Dejan,
>
> The bug looks like a one-off occurrence. We run hundreds of hours of
> system stress tests in a week, moving resources between main and
standby
> systems, and we haven't seen this error in a couple years. (There was
a
> longclock error back in 2007 found by my colleague Simon Graham).
>
> The longclock wrap occurred within 2:45 of a reboot.
> The apparent coincidence seems to be that we were starting resources
on
> a back-up node around 165 seconds after the node had been rebooted and
> hearbeat restarted. As I expect you know, somewhere between 160 and
175
> seconds after a heartbeat start, the longclock is configured to wrap.

Nonono.
times() is coupled to uptime,
uptime is measured in jiffies,
and initial jiffies in linux kernel is
include/linux/jiffies.h:
#define INITIAL_JIFFIES ((unsigned long)(unsigned int) (-300*HZ))

so the first wrap occurs 5 minutes after boot.
nothing to do with heartbeat start, or longclock.

> The rareness of this makes me think we hit a really obscure window...

yes.
but.

it is totally broken.
some grepping in my mercurial checkouts suggest that you are using
heartbeat, which is using its "plumbing" wrappers.

[skip this paragraph unless you are heartbeat coder]
in particular, the killing is done by
TrackedProcTimeoutFunction(),
and the timeout for that is set by SetTrackedProcTimeouts(),
which calls into Gmain_timeout_add -> Gmain_timeout_add_full
(still in the heartbeat plumbing wrappers),
which creates a new event source of type GTimoutAppend
(also heartbeat plumbing) with methods defined by Gmain_timeout_funcs.

The Gmain_timeout_prepare does a simple check on
"is the next time this event shall trigger in the past?",
and if so, triggers immediately.

Now, that check is broken.
in lib/clplumbing/GSource.c, Gmain_timeout_prepare():
if (cmp_longclock(lnow, append->nexttime) >= 0)
assuming that nexttime was set correctly, and lnow is correct, too,
and further assuming your clock_t is only 32 bit,
longclock_t is defined as unsigned long long,
and that thing becomes:

if ((unsigned long long)(lnow) >= (unsigned long
long)(append-nexttime))

which exactly does _NOT_ care for wrap around :(

cmp_longclock macro and functions are broken. conceivably some other
longclock related macros/functions may be broken as well.

it should probably be more like
(caution, not even compile tested)

diff -r 731f8f7b5450 include/clplumbing/longclock.h
--- a/include/clplumbing/longclock.h Tue Jun 30 12:02:16 2009 +0200
+++ b/include/clplumbing/longclock.h Wed Jul 01 12:27:27 2009 +0200
@@ -127,9 +127,9 @@
((longclock_t)(l) - (longclock_t)(r))

# define cmp_longclock(l,r) \
- (((longclock_t)(l) < (longclock_t)(r)) \
+ ((((long long)(l) - (long long)(r)) < 0) \
? -1 \
- : (((longclock_t)(l) > (longclock_t)(r)) \
+ : ((((long long)(l) - (long long)(r)) > 0) \
? +1 : 0))
#endif

diff -r 731f8f7b5450 lib/clplumbing/longclock.c
--- a/lib/clplumbing/longclock.c Tue Jun 30 12:02:16 2009 +0200
+++ b/lib/clplumbing/longclock.c Wed Jul 01 12:27:27 2009 +0200
@@ -259,12 +259,7 @@
int
cmp_longclock(longclock_t l, longclock_t r)
{
- if (l < r) {
- return -1;
- }
- if (l > r) {
- return 1;
- }
- return 0;
+ return (((long long)(l) - (long long)(r)) < 0) ? -1
+ : (((long long)(l) - (long long)(r)) > 0) ? +1 : 0;
}
#endif /* CLOCK_T_IS_LONG_ENOUGH */

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD(r) and LINBIT(r) are registered trademarks of LINBIT, Austria.
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

lars.ellenberg at linbit

Jul 1, 2009, 7:44 AM

Post #2 of 8 (525 views)
Permalink

Re: [patch] clock_t wrapped around causingfalse resourcestart failure [In reply to]

On Wed, Jul 01, 2009 at 10:22:58AM -0400, Tavanyar, Simon wrote:
> Aahh ... so it's measured after reboot. That makes sense.
> Thanks, Lars.
>
> Should I wait for you to log a bugzilla on this?

bugzilla and me don't go along very well, really ;-)
so please feel free to open that bugzilla,
and attach my proposed patch there.

just to reiterate:
the problem is not the uptime wrap or resulting longclock wrap,
as that is handled correctly in cl_times() and time_longclock.

but the cmp_longclock comparing unsigned values
without checking for wrap.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBDŽ and LINBITŽ are registered trademarks of LINBIT, Austria.
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

dejanmm at fastmail

Jul 1, 2009, 7:59 AM

Post #3 of 8 (527 views)
Permalink

Re: [patch] clock_t wrapped around causingfalse resourcestart failure [In reply to]

Hi Lars,

On Wed, Jul 01, 2009 at 04:44:06PM +0200, Lars Ellenberg wrote:
> On Wed, Jul 01, 2009 at 10:22:58AM -0400, Tavanyar, Simon wrote:
> > Aahh ... so it's measured after reboot. That makes sense.
> > Thanks, Lars.
> >
> > Should I wait for you to log a bugzilla on this?
>
> bugzilla and me don't go along very well, really ;-)
> so please feel free to open that bugzilla,
> and attach my proposed patch there.
>
> just to reiterate:
> the problem is not the uptime wrap or resulting longclock wrap,
> as that is handled correctly in cl_times() and time_longclock.
>
> but the cmp_longclock comparing unsigned values
> without checking for wrap.

The wrap is stuffed into upper bits of the longclock_t variables:

return (lc_wrapcount | (longclock_t)timesval);

Or am I missing something? The only explanation I can think of is
that this is actually wrong:

if (cmp_longclock(lnow, append->nexttime) >= 0) {

i.e. it should be strictly greater. I guess that what happened is
that somehow the prepare function got called fast enough, so that
both timestamps ended equal.

Thanks,

Dejan

>
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
> DRBD? and LINBIT? are registered trademarks of LINBIT, Austria.
> _______________________________________________
> Linux-HA mailing list
> Linux-HA[at]lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

lars.ellenberg at linbit

Jul 1, 2009, 8:13 AM

Post #4 of 8 (525 views)
Permalink

Re: [patch] clock_t wrapped around causingfalse resourcestart failure [In reply to]

On Wed, Jul 01, 2009 at 04:59:24PM +0200, Dejan Muhamedagic wrote:
> Hi Lars,
>
> On Wed, Jul 01, 2009 at 04:44:06PM +0200, Lars Ellenberg wrote:
> > On Wed, Jul 01, 2009 at 10:22:58AM -0400, Tavanyar, Simon wrote:
> > > Aahh ... so it's measured after reboot. That makes sense.
> > > Thanks, Lars.
> > >
> > > Should I wait for you to log a bugzilla on this?
> >
> > bugzilla and me don't go along very well, really ;-)
> > so please feel free to open that bugzilla,
> > and attach my proposed patch there.
> >
> > just to reiterate:
> > the problem is not the uptime wrap or resulting longclock wrap,
> > as that is handled correctly in cl_times() and time_longclock.
> >
> > but the cmp_longclock comparing unsigned values
> > without checking for wrap.
>
> The wrap is stuffed into upper bits of the longclock_t variables:
>
> return (lc_wrapcount | (longclock_t)timesval);
>
> Or am I missing something? The only explanation I can think of is
> that this is actually wrong:

absolutely.

> if (cmp_longclock(lnow, append->nexttime) >= 0) {
>
> i.e. it should be strictly greater. I guess that what happened is
> that somehow the prepare function got called fast enough, so that
> both timestamps ended equal.

absolutely not.
;)

Much worse!
as far as I see, cmp_longclock is broken (does not care for wrap)
even on 64bit archs, and the time window for this to happen is about
as large as the larges action (or else) timeout you set somewhere in
crm/lrm/whatever.

did you have a look at my explanation ealier in this thread,
and the patch? (part of that message pasted here for reference)

if (cmp_longclock(lnow, append->nexttime) >= 0)
assuming that nexttime was set correctly, and lnow is correct, too,
and further assuming your clock_t is only 32 bit,
longclock_t is defined as unsigned long long,
and that thing becomes:

if ((unsigned long long)(lnow) >= (unsigned long long)(append->nexttime))

which exactly does _NOT_ care for wrap around :(

example:
say, you start with a _large_ lnow (e.g. the equivalent of "-15
seconds"), and add an intervall of 30 seconds, resulting
"nexttime" become the equivalent of +15 seconds.
comparing absolute _unsigned_ means that test ends up comparing
if (something verry large >= something very small).
which is always true :(

you need instead cast both values to _signed_, and compare the
_difference_ against zero, as below:

cmp_longclock macro and functions are broken. conceivably some other
longclock related macros/functions may be broken as well.

it should probably be more like
(caution, not even compile tested)
((and this time incompletely copy'n'pasted,
so whitespace will be wrong))

diff -r 731f8f7b5450 include/clplumbing/longclock.h
--- a/include/clplumbing/longclock.h Tue Jun 30 12:02:16 2009 +0200
+++ b/include/clplumbing/longclock.h Wed Jul 01 12:27:27 2009 +0200
@@ -127,9 +127,9 @@
((longclock_t)(l) - (longclock_t)(r))

# define cmp_longclock(l,r) \
- (((longclock_t)(l) < (longclock_t)(r)) \
+ ((((long long)(l) - (long long)(r)) < 0) \
? -1 \
- : (((longclock_t)(l) > (longclock_t)(r)) \
+ : ((((long long)(l) - (long long)(r)) > 0) \
? +1 : 0))

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBDŽ and LINBITŽ are registered trademarks of LINBIT, Austria.
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

dejanmm at fastmail

Jul 2, 2009, 2:00 AM

Post #5 of 8 (512 views)
Permalink

Re: [patch] clock_t wrapped around causingfalse resourcestart failure [In reply to]

On Wed, Jul 01, 2009 at 05:13:42PM +0200, Lars Ellenberg wrote:
> On Wed, Jul 01, 2009 at 04:59:24PM +0200, Dejan Muhamedagic wrote:
> > Hi Lars,
> >
> > On Wed, Jul 01, 2009 at 04:44:06PM +0200, Lars Ellenberg wrote:
> > > On Wed, Jul 01, 2009 at 10:22:58AM -0400, Tavanyar, Simon wrote:
> > > > Aahh ... so it's measured after reboot. That makes sense.
> > > > Thanks, Lars.
> > > >
> > > > Should I wait for you to log a bugzilla on this?
> > >
> > > bugzilla and me don't go along very well, really ;-)
> > > so please feel free to open that bugzilla,
> > > and attach my proposed patch there.
> > >
> > > just to reiterate:
> > > the problem is not the uptime wrap or resulting longclock wrap,
> > > as that is handled correctly in cl_times() and time_longclock.
> > >
> > > but the cmp_longclock comparing unsigned values
> > > without checking for wrap.
> >
> > The wrap is stuffed into upper bits of the longclock_t variables:
> >
> > return (lc_wrapcount | (longclock_t)timesval);
> >
> > Or am I missing something? The only explanation I can think of is
> > that this is actually wrong:
>
> absolutely.

Looking again, this is probably not a problem. The nexttime is
always in the future.

> > if (cmp_longclock(lnow, append->nexttime) >= 0) {
> >
> > i.e. it should be strictly greater. I guess that what happened is
> > that somehow the prepare function got called fast enough, so that
> > both timestamps ended equal.
>
> absolutely not.
> ;)
>
> Much worse!
> as far as I see, cmp_longclock is broken (does not care for wrap)

I still don't understand what do you mean by "does not care for
wrap". In case of 32-bit clock_t the wrapcount is stuffed into
the upper bits of longclock_t which is 64-bit. How can it then
not take it into account? In case of 64-bit clock_t, there is no
wrap. It has enough bits to serve us until the end of the world.

> even on 64bit archs, and the time window for this to happen is about
> as large as the larges action (or else) timeout you set somewhere in
> crm/lrm/whatever.
>
> did you have a look at my explanation ealier in this thread,
> and the patch? (part of that message pasted here for reference)
>
>
> if (cmp_longclock(lnow, append->nexttime) >= 0)
> assuming that nexttime was set correctly, and lnow is correct, too,
> and further assuming your clock_t is only 32 bit,
> longclock_t is defined as unsigned long long,
> and that thing becomes:
>
> if ((unsigned long long)(lnow) >= (unsigned long long)(append->nexttime))
>
> which exactly does _NOT_ care for wrap around :(
>
>
> example:
> say, you start with a _large_ lnow (e.g. the equivalent of "-15

But you can't start with large lnow. lnow is 64-bit and you can't
get that far into the future.

> seconds"), and add an intervall of 30 seconds, resulting
> "nexttime" become the equivalent of +15 seconds.
> comparing absolute _unsigned_ means that test ends up comparing
> if (something verry large >= something very small).
> which is always true :(
>
> you need instead cast both values to _signed_, and compare the
> _difference_ against zero, as below:
>
>
> cmp_longclock macro and functions are broken. conceivably some other
> longclock related macros/functions may be broken as well.
>
> it should probably be more like
> (caution, not even compile tested)
> ((and this time incompletely copy'n'pasted,
> so whitespace will be wrong))
>
> diff -r 731f8f7b5450 include/clplumbing/longclock.h
> --- a/include/clplumbing/longclock.h Tue Jun 30 12:02:16 2009 +0200
> +++ b/include/clplumbing/longclock.h Wed Jul 01 12:27:27 2009 +0200
> @@ -127,9 +127,9 @@
> ((longclock_t)(l) - (longclock_t)(r))
>
> # define cmp_longclock(l,r) \
> - (((longclock_t)(l) < (longclock_t)(r)) \
> + ((((long long)(l) - (long long)(r)) < 0) \
> ? -1 \
> - : (((longclock_t)(l) > (longclock_t)(r)) \
> + : ((((long long)(l) - (long long)(r)) > 0) \
> ? +1 : 0))

This patch may be correct, but for practical reasons it won't
make any difference.

Thanks,

Dejan

> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
> DRBD? and LINBIT? are registered trademarks of LINBIT, Austria.
> _______________________________________________
> Linux-HA mailing list
> Linux-HA[at]lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

lars.ellenberg at linbit

Jul 2, 2009, 5:16 AM

Post #6 of 8 (505 views)
Permalink

Re: [patch] clock_t wrapped around causingfalse resourcestart failure [In reply to]

On Thu, Jul 02, 2009 at 11:00:08AM +0200, Dejan Muhamedagic wrote:
> > if (cmp_longclock(lnow, append->nexttime) >= 0)
> > assuming that nexttime was set correctly, and lnow is correct, too,
> > and further assuming your clock_t is only 32 bit,
> > longclock_t is defined as unsigned long long,
> > and that thing becomes:
> >
> > if ((unsigned long long)(lnow) >= (unsigned long long)(append->nexttime))
> >
> > which exactly does _NOT_ care for wrap around :(
> >
> >
> > example:
> > say, you start with a _large_ lnow (e.g. the equivalent of "-15
>
> But you can't start with large lnow. lnow is 64-bit and you can't
> get that far into the future.

Hm. Right.

Then maybe something else is wrong, like some wrong typecast?
int i = -300000;
unsigned long long l_wrong = i;
unsigned long long l_right = (unsigned long long)(unsigned int)i;

l_wrong is now "very far" into the future,
and my patch would fix the comparison of that.

whereas l_right is just below 2<<32, and will compare fine in 64 bit
wide unsigned long long, with the existing code.

Simon: can you help to track this down? Which exact "platform" is this
on, heartbeat version, configure and compiler flags...
You could also try to start this in a VM, and try to reproduce?

Or shall we just ignore this for now?

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBDŽ and LINBITŽ are registered trademarks of LINBIT, Austria.
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Simon.Tavanyar at stratus

Jul 5, 2009, 3:09 PM

Post #7 of 8 (463 views)
Permalink

Re: [patch] clock_t wrapped aroundcausingfalse resourcestart failure [In reply to]

This looks like a very hard bug to reproduce. I'm reluctantly going to
recommend "Ignore for now".

-----Original Message-----
From: linux-ha-bounces[at]lists.linux-ha.org
[mailto:linux-ha-bounces[at]lists.linux-ha.org] On Behalf Of Lars Ellenberg
Sent: Thursday, July 02, 2009 8:17 AM
To: linux-ha[at]lists.linux-ha.org
Subject: Re: [Linux-HA] [patch] clock_t wrapped aroundcausingfalse
resourcestart failure

On Thu, Jul 02, 2009 at 11:00:08AM +0200, Dejan Muhamedagic wrote:
> > if (cmp_longclock(lnow, append->nexttime) >= 0)
> > assuming that nexttime was set correctly, and lnow is correct, too,
> > and further assuming your clock_t is only 32 bit,
> > longclock_t is defined as unsigned long long,
> > and that thing becomes:
> >
> > if ((unsigned long long)(lnow) >= (unsigned long
long)(append->nexttime))
> >
> > which exactly does _NOT_ care for wrap around :(
> >
> >
> > example:
> > say, you start with a _large_ lnow (e.g. the equivalent of "-15
>
> But you can't start with large lnow. lnow is 64-bit and you can't
> get that far into the future.

Hm. Right.

Then maybe something else is wrong, like some wrong typecast?
int i = -300000;
unsigned long long l_wrong = i;
unsigned long long l_right = (unsigned long long)(unsigned int)i;

l_wrong is now "very far" into the future,
and my patch would fix the comparison of that.

whereas l_right is just below 2<<32, and will compare fine in 64 bit
wide unsigned long long, with the existing code.

Simon: can you help to track this down? Which exact "platform" is this
on, heartbeat version, configure and compiler flags...
You could also try to start this in a VM, and try to reproduce?

Or shall we just ignore this for now?

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD(r) and LINBIT(r) are registered trademarks of LINBIT, Austria.
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

dejanmm at fastmail

Jul 6, 2009, 1:53 AM

Post #8 of 8 (452 views)
Permalink

Re: [patch] clock_t wrapped aroundcausingfalse resourcestart failure [In reply to]

On Sun, Jul 05, 2009 at 06:09:13PM -0400, Tavanyar, Simon wrote:
> This looks like a very hard bug to reproduce.

Yes.

> I'm reluctantly going to
> recommend "Ignore for now".

Can you please use hb_report and file a bugzilla so that we don't
forget about it.

Thanks,

Dejan

> -----Original Message-----
> From: linux-ha-bounces[at]lists.linux-ha.org
> [mailto:linux-ha-bounces[at]lists.linux-ha.org] On Behalf Of Lars Ellenberg
> Sent: Thursday, July 02, 2009 8:17 AM
> To: linux-ha[at]lists.linux-ha.org
> Subject: Re: [Linux-HA] [patch] clock_t wrapped aroundcausingfalse
> resourcestart failure
>
> On Thu, Jul 02, 2009 at 11:00:08AM +0200, Dejan Muhamedagic wrote:
> > > if (cmp_longclock(lnow, append->nexttime) >= 0)
> > > assuming that nexttime was set correctly, and lnow is correct, too,
> > > and further assuming your clock_t is only 32 bit,
> > > longclock_t is defined as unsigned long long,
> > > and that thing becomes:
> > >
> > > if ((unsigned long long)(lnow) >= (unsigned long
> long)(append->nexttime))
> > >
> > > which exactly does _NOT_ care for wrap around :(
> > >
> > >
> > > example:
> > > say, you start with a _large_ lnow (e.g. the equivalent of "-15
> >
> > But you can't start with large lnow. lnow is 64-bit and you can't
> > get that far into the future.
>
> Hm. Right.
>
> Then maybe something else is wrong, like some wrong typecast?
> int i = -300000;
> unsigned long long l_wrong = i;
> unsigned long long l_right = (unsigned long long)(unsigned int)i;
>
> l_wrong is now "very far" into the future,
> and my patch would fix the comparison of that.
>
> whereas l_right is just below 2<<32, and will compare fine in 64 bit
> wide unsigned long long, with the existing code.
>
> Simon: can you help to track this down? Which exact "platform" is this
> on, heartbeat version, configure and compiler flags...
> You could also try to start this in a VM, and try to reproduce?
>
> Or shall we just ignore this for now?
>
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
> DRBD(r) and LINBIT(r) are registered trademarks of LINBIT, Austria.
> _______________________________________________
> Linux-HA mailing list
> Linux-HA[at]lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
> _______________________________________________
> Linux-HA mailing list
> Linux-HA[at]lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA[at]lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com