Mailing List Archive: Linux-HA: Users

OCFS2 DRBD Heartbeat and high availability did not work out so well

Index | Next | Previous | View Threaded

edlinuxguru at gmail

Aug 3, 2007, 8:50 AM

Post #1 of 9 (7171 views)
Permalink

OCFS2 DRBD Heartbeat and high availability did not work out so well

I see a lot of people on this list talking about Linux HA, DRDB, and OCFS2 I
thought I would share some of my experiences with it.

We took two dell 860 servers out of the box. We setup FC5. Because we are
using software striping we ended up creating LVM partitions.
We decided to leave the a huge slice of the disk unallocated/unformattted
for /opt and run DRBD. Our software running and working inside opt can only
exist on one node at a time, but I decided to use DRBD two primaries. This
way I can look at the data on both nodes, and it seemed really cool.

DRDB and OCFS2 did not work correctly on the base FC5 Kernel so I ran 'yum
update' and got the newest one.
Ok so this is the first sticky point. Yes FC5 is a development line, but FC5
is also IMHO just as stable as any other OS. Lets agree that it cant be
'much less stable' if that makes any sense.

I grabbed the lastest DBRD. Finding an OCFS2 rpm for FC5 took searching but
I found it.

I did get the system setup in a testing environment. I created a file on one
node. Then I edited on the other. So the clustered file system works.
Failover worked. All was well. I was suprised with how well things were
going.

I tried to implement the File system in with heartbeat. At the time HA did
not have a DRBD two primary resource RA. I settled on making CLONED
resources out of the /etc/init.d files.

I think this might be a bad idea in an active/active setup. You might be
better making your storage unmanaged. Because of the way I set this up
stopping heartbeat would stop the cluster disk. This freaked out users that
may have been just looking at a file on the passive node while heartbeat was
restarting.

Also I have found that adding resources into and out of groups or changing
the order can cause resources in HA to restart unexpectedly. I felt like my
setup was a minefield. A bug in one init script might cause a casade, or if
something in a colocation died it was going to cause a cluster
wide failover. It was hard for me to enforce the business rules that I
really wanted to. Restarting something in a colocation caused a cascade.
This may be more or less due to a lack of heartbeat knowledge on my side.

Everything was grand in the backroom. BUT
Our production configuration opens more files for logging and our software
opens a lot of files in general. OCFS2 and DRBD have a noticable lag in
starting our application. Could be the time to aquire file locks,etc.
Whatever the case it was slower. This had a bad effect on my timeouts in my
status and monitor functions. Life lesson, timeouts need to be set higher.
Still this caused headaches because I had already handing the system off to
the next implementer and they could not understand why configurations
changes was causing cluster wide failover from node1, node2,node1,node2.....
and repeatedly kicking you out of ssh as the VIP IP address kept failing
back and forth.

My final straws were:

That a system went up to high IO wait it needed a reboot.

Corrupt files that can not be deleted.

Unable to reboot machine after OCFS2 and DRDB failed. This was really
annoying might have been more of an OCFS problem then DRBD. Regardless I
could not unmount the disk in drbd, OCFS2 and o2cb stop, unload , etc would
not do it. Killing processes did not handle it neither did modprobing. Had
to call out datacenter. They rebooted the wrong node.

I finally gave up on the disk cluster. We have a plan to move it all back to
a single server. I shut off one end of the node. It ran for about a
week. The system crashed today I got this message. 'OCFS timeout
accessing DRBD. OCFS2 is sorry to be fencing the system'

Not as sorry as I am.
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

beekhof at gmail

Aug 7, 2007, 1:05 AM

Post #2 of 9 (7095 views)
Permalink

Re: OCFS2 DRBD Heartbeat and high availability did not work out so well [In reply to]

On 8/3/07, Eddie C <edlinuxguru [at] gmail> wrote:
> I see a lot of people on this list talking about Linux HA, DRDB, and OCFS2 I
> thought I would share some of my experiences with it.

Is this even possible?
I didn't think you could mount more than one node with drbd... is this
a new feature in drbd8?
If not, running OCFS2 on top would be guaranteed to fsck your data.

Its also not clear to me that you ran OCFS2 in userspace-heartbeat
mode which is also quite important.

> We took two dell 860 servers out of the box. We setup FC5. Because we are
> using software striping we ended up creating LVM partitions.
> We decided to leave the a huge slice of the disk unallocated/unformattted
> for /opt and run DRBD. Our software running and working inside opt can only
> exist on one node at a time, but I decided to use DRBD two primaries. This
> way I can look at the data on both nodes, and it seemed really cool.
>
> DRDB and OCFS2 did not work correctly on the base FC5 Kernel so I ran 'yum
> update' and got the newest one.
> Ok so this is the first sticky point. Yes FC5 is a development line, but FC5
> is also IMHO just as stable as any other OS. Lets agree that it cant be
> 'much less stable' if that makes any sense.

Sure.
But what versions of drbd/heartbeat/ocfs2 were you running?
Thats the relevant question.

> I grabbed the lastest DBRD. Finding an OCFS2 rpm for FC5 took searching but
> I found it.
>
> I did get the system setup in a testing environment. I created a file on one
> node. Then I edited on the other. So the clustered file system works.
> Failover worked. All was well. I was suprised with how well things were
> going.
>
> I tried to implement the File system in with heartbeat. At the time HA did
> not have a DRBD two primary resource RA. I settled on making CLONED
> resources out of the /etc/init.d files.
>
> I think this might be a bad idea in an active/active setup. You might be
> better making your storage unmanaged. Because of the way I set this up
> stopping heartbeat would stop the cluster disk.

Uh, yeah.
Having cluster resources running on non-cluster members (any node
without fully active/running cluster software is NOT a cluster member)
is _insanely_ dangerous.

> This freaked out users that
> may have been just looking at a file on the passive node while heartbeat was
> restarting.
>
> Also I have found that adding resources into and out of groups or changing
> the order can cause resources in HA to restart unexpectedly. I felt like my
> setup was a minefield. A bug in one init script might cause a casade, or if
> something in a colocation died it was going to cause a cluster
> wide failover.

This is not exactly surprise.
Even in a non-clustered environment, a filesystem failure will cascade
upwards to any databases, web/mail servers etc that may be running on
it.

So if the init script is broken and creates/invents a failure... the
same thing will happen (or at least we'll try to recover them before
they fail).

Likewise for colocation... if you tell us A must run with B and B is
not running...
consider s/A/database/ and s/B/filesystem/

> It was hard for me to enforce the business rules that I
> really wanted to. Restarting something in a colocation caused a cascade.
> This may be more or less due to a lack of heartbeat knowledge on my side.
>
> Everything was grand in the backroom. BUT
> Our production configuration opens more files for logging and our software
> opens a lot of files in general. OCFS2 and DRBD have a noticable lag in
> starting our application. Could be the time to aquire file locks,etc.
> Whatever the case it was slower. This had a bad effect on my timeouts in my
> status and monitor functions. Life lesson, timeouts need to be set higher.
> Still this caused headaches because I had already handing the system off to
> the next implementer and they could not understand why configurations
> changes was causing cluster wide failover from node1, node2,node1,node2.....
> and repeatedly kicking you out of ssh as the VIP IP address kept failing
> back and forth.
>
> My final straws were:
>
> That a system went up to high IO wait it needed a reboot.
>
> Corrupt files that can not be deleted.
>
> Unable to reboot machine after OCFS2 and DRDB failed. This was really
> annoying might have been more of an OCFS problem then DRBD. Regardless I
> could not unmount the disk in drbd, OCFS2 and o2cb stop, unload , etc would
> not do it. Killing processes did not handle it neither did modprobing. Had
> to call out datacenter. They rebooted the wrong node.
>
> I finally gave up on the disk cluster. We have a plan to move it all back to
> a single server. I shut off one end of the node. It ran for about a
> week. The system crashed today I got this message. 'OCFS timeout
> accessing DRBD. OCFS2 is sorry to be fencing the system'
>
> Not as sorry as I am.
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

edlinuxguru at gmail

Aug 7, 2007, 7:19 PM

Post #3 of 9 (7072 views)
Permalink

Re: OCFS2 DRBD Heartbeat and high availability did not work out so well [In reply to]

Yes the DRBD 0.8 supports 2 I think or even three mounts. I want drbd and
ocfs2 as an active/active disk cluster.

I have these versions of things

-rw-r--r-- 1 root root 482489 May 16 11:09 drbd-8.0.3.tar.gz
-rw-r--r-- 1 root root 5126020 May 16 11:08
kernel-devel-2.6.15-1.2054_FC5.x86_
4.rpm
-rw-r--r-- 1 root root 1173119 May 15 16:18 ocfs2-tools-1.2.1-1.x86_64.rpm

"/etc/ocfs2/cluster.conf"
cluster:
node_count = 2
name = input
node:
ip_port = 7777
ip_address = 10.10.10.81
number = 1
name = input2a
cluster = input
node:
ip_port = 7777
ip_address = 10.10.10.82
number = 2
name = input2b
cluster = input

global {
usage-count yes;
}
common { syncer { rate 10M; } }
resource drbd0 {
protocol C;
net {
cram-hmac-alg sha1;
shared-secret "blabla";
allow-two-primaries;

after-sb-0pri discard-least-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
rr-conflict disconnect;
}
on input2a {
device /dev/drbd0;
disk /dev/mapper/VolGroup00-data_vol;
address 10.10.10.81:7789;
meta-disk internal;
}

on input2b {
device /dev/drbd0;
disk /dev/mapper/VolGroup00-data_vol;
address 10.10.10.82:7789;
meta-disk internal;
}
}

I do not know how to answer your OCFS2 userspace comment:

By 'running the file system unmanaged' I meant that having an active/active
disk on a two node clusters adds complexity. I think it is better to start
you an active/active file system from the init scripts rather then from
inside heartbeat, otherwise you are linking the filesystem to HA when it
does not need to be. It is nice that heartbeat is aware of the file system,
but when they are tied together you can not stop heartbeat without stopping
the disk.

My post was more of a rant then anything else. I was looking for 50 people
or so to read my post and say. "You must be doing something wrong. I run
DRBD with OCFS2 multi node actice/active and MySQL and its super fast and
never crashes on two 100 mhz laptops"

The concept is great an active/active disk partition and heartbeat with no
fancy SAN's. It works for stretches 3 or 4 weeks. But then I run into a
weird locked directory that I cant delete or a file owned by '?'. Or the
partition unmounts and the system will not reboot.

On 8/7/07, Andrew Beekhof <beekhof [at] gmail> wrote:
>
> On 8/3/07, Eddie C <edlinuxguru [at] gmail> wrote:
> > I see a lot of people on this list talking about Linux HA, DRDB, and
> OCFS2 I
> > thought I would share some of my experiences with it.
>
> Is this even possible?
> I didn't think you could mount more than one node with drbd... is this
> a new feature in drbd8?
> If not, running OCFS2 on top would be guaranteed to fsck your data.
>
> Its also not clear to me that you ran OCFS2 in userspace-heartbeat
> mode which is also quite important.
>
> > We took two dell 860 servers out of the box. We setup FC5. Because we
> are
> > using software striping we ended up creating LVM partitions.
> > We decided to leave the a huge slice of the disk
> unallocated/unformattted
> > for /opt and run DRBD. Our software running and working inside opt can
> only
> > exist on one node at a time, but I decided to use DRBD two primaries.
> This
> > way I can look at the data on both nodes, and it seemed really cool.
> >
> > DRDB and OCFS2 did not work correctly on the base FC5 Kernel so I ran
> 'yum
> > update' and got the newest one.
> > Ok so this is the first sticky point. Yes FC5 is a development line, but
> FC5
> > is also IMHO just as stable as any other OS. Lets agree that it cant be
> > 'much less stable' if that makes any sense.
>
> Sure.
> But what versions of drbd/heartbeat/ocfs2 were you running?
> Thats the relevant question.
>
> > I grabbed the lastest DBRD. Finding an OCFS2 rpm for FC5 took searching
> but
> > I found it.
> >
> > I did get the system setup in a testing environment. I created a file on
> one
> > node. Then I edited on the other. So the clustered file system works.
> > Failover worked. All was well. I was suprised with how well things were
> > going.
> >
> > I tried to implement the File system in with heartbeat. At the time HA
> did
> > not have a DRBD two primary resource RA. I settled on making CLONED
> > resources out of the /etc/init.d files.
> >
> > I think this might be a bad idea in an active/active setup. You might be
> > better making your storage unmanaged. Because of the way I set this up
> > stopping heartbeat would stop the cluster disk.
>
> Uh, yeah.
> Having cluster resources running on non-cluster members (any node
> without fully active/running cluster software is NOT a cluster member)
> is _insanely_ dangerous.
>
> > This freaked out users that
> > may have been just looking at a file on the passive node while heartbeat
> was
> > restarting.
> >
> > Also I have found that adding resources into and out of groups or
> changing
> > the order can cause resources in HA to restart unexpectedly. I felt like
> my
> > setup was a minefield. A bug in one init script might cause a casade, or
> if
> > something in a colocation died it was going to cause a cluster
> > wide failover.
>
> This is not exactly surprise.
> Even in a non-clustered environment, a filesystem failure will cascade
> upwards to any databases, web/mail servers etc that may be running on
> it.
>
> So if the init script is broken and creates/invents a failure... the
> same thing will happen (or at least we'll try to recover them before
> they fail).
>
> Likewise for colocation... if you tell us A must run with B and B is
> not running...
> consider s/A/database/ and s/B/filesystem/
>
>
> > It was hard for me to enforce the business rules that I
> > really wanted to. Restarting something in a colocation caused a cascade.
> > This may be more or less due to a lack of heartbeat knowledge on my
> side.
> >
> > Everything was grand in the backroom. BUT
> > Our production configuration opens more files for logging and our
> software
> > opens a lot of files in general. OCFS2 and DRBD have a noticable lag in
> > starting our application. Could be the time to aquire file locks,etc.
> > Whatever the case it was slower. This had a bad effect on my timeouts in
> my
> > status and monitor functions. Life lesson, timeouts need to be set
> higher.
> > Still this caused headaches because I had already handing the system off
> to
> > the next implementer and they could not understand why configurations
> > changes was causing cluster wide failover from node1,
> node2,node1,node2.....
> > and repeatedly kicking you out of ssh as the VIP IP address kept failing
> > back and forth.
> >
> > My final straws were:
> >
> > That a system went up to high IO wait it needed a reboot.
> >
> > Corrupt files that can not be deleted.
> >
> > Unable to reboot machine after OCFS2 and DRDB failed. This was really
> > annoying might have been more of an OCFS problem then DRBD. Regardless I
> > could not unmount the disk in drbd, OCFS2 and o2cb stop, unload , etc
> would
> > not do it. Killing processes did not handle it neither did modprobing.
> Had
> > to call out datacenter. They rebooted the wrong node.
> >
> > I finally gave up on the disk cluster. We have a plan to move it all
> back to
> > a single server. I shut off one end of the node. It ran for about a
> > week. The system crashed today I got this message. 'OCFS timeout
> > accessing DRBD. OCFS2 is sorry to be fencing the system'
> >
> > Not as sorry as I am.
> > _______________________________________________
> > Linux-HA mailing list
> > Linux-HA [at] lists
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > See also: http://linux-ha.org/ReportingProblems
> >
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

beekhof at gmail

Aug 8, 2007, 12:09 AM

Post #4 of 9 (7078 views)
Permalink

Re: OCFS2 DRBD Heartbeat and high availability did not work out so well [In reply to]

On 8/8/07, Eddie C <edlinuxguru [at] gmail> wrote:
> Yes the DRBD 0.8 supports 2 I think or even three mounts. I want drbd and
> ocfs2 as an active/active disk cluster.
>
> I have these versions of things
>
> -rw-r--r-- 1 root root 482489 May 16 11:09 drbd-8.0.3.tar.gz
> -rw-r--r-- 1 root root 5126020 May 16 11:08
> kernel-devel-2.6.15-1.2054_FC5.x86_
> 4.rpm
> -rw-r--r-- 1 root root 1173119 May 15 16:18 ocfs2-tools-1.2.1-1.x86_64.rpm

and of heartbeat? This is a _heartbeat_ mailing list remember...

> I do not know how to answer your OCFS2 userspace comment:

http://oss.oracle.com/pipermail/ocfs2-devel/2006-January/000833.html

The concept being that its a bad idea to have OCFS2 and Heartbeat
_not_ share the same membership information (essentially you have two
clusters). With these patches ocfs2 goes into a sort of slave-mode
and takes orders from Heartbeat.

> By 'running the file system unmanaged' I meant that having an active/active
> disk on a two node clusters adds complexity. I think it is better to start
> you an active/active file system from the init scripts rather then from
> inside heartbeat,

generally speaking, thats a terrible idea (for the same reason that
the membership information will differ and non-cluster nodes will be
running cluster resources).

> otherwise you are linking the filesystem to HA when it
> does not need to be. It is nice that heartbeat is aware of the file system,
> but when they are tied together you can not stop heartbeat without stopping
> the disk.

See my comment in the previous email regarding non-cluster nodes
running cluster resources.

> My post was more of a rant then anything else. I was looking for 50 people
> or so to read my post and say. "You must be doing something wrong. I run
> DRBD with OCFS2 multi node actice/active and MySQL and its super fast and
> never crashes on two 100 mhz laptops"

Given how little you seem to be using heartbeat, perhaps the drbd or
ocfs2 lists might be a more appropriate forum.

> The concept is great an active/active disk partition and heartbeat with no
> fancy SAN's. It works for stretches 3 or 4 weeks. But then I run into a
> weird locked directory that I cant delete or a file owned by '?'. Or the
> partition unmounts and the system will not reboot.
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

rawipfel at novell

Aug 8, 2007, 1:12 AM

Post #5 of 9 (7074 views)
Permalink

Re: OCFS2 DRBD Heartbeat and high availability did not work out so well [In reply to]

>>> On Wed, Aug 8, 2007 at 1:09 AM, in message
<26ef5e70708080009q26b0e795r7cdb919d5c3ad2a9 [at] mail>, "Andrew Beekhof"
<beekhof [at] gmail> wrote:
> On 8/8/07, Eddie C <edlinuxguru [at] gmail> wrote:

[...]

>> My post was more of a rant then anything else. I was looking for 50 people
>> or so to read my post and say. "You must be doing something wrong. I run
>> DRBD with OCFS2 multi node actice/active and MySQL and its super fast and
>> never crashes on two 100 mhz laptops"

I know it's a bit off topic, but for this kind of setup (super low cost
nodes), a shared firewire disk actually seems to work quite well. You'll
have to load the firewire module with exclusive_login = 0, and then
both servers will happily share that $150 fw disk ;-) and it actually works
well enough to run Xen VMs off shared disk, using OCFS2 in userspace
heartbeat mode, with Heartbeat2 managing the VMs as resources. There's
some setup info here:
http://www.oracle.com/technology/pub/articles/hunter_rac10gr2.html

> Given how little you seem to be using heartbeat, perhaps the drbd or
> ocfs2 lists might be a more appropriate forum.
>
>> The concept is great an active/active disk partition and heartbeat with no
>> fancy SAN's. It works for stretches 3 or 4 weeks. But then I run into a
>> weird locked directory that I cant delete or a file owned by '?'. Or the
>> partition unmounts and the system will not reboot.

I haven't checked prices recently, but multi-initiator serial attached
shared SCSI is also a recent option, with a number of vendors providing
low cost RAID enclosures, that can be shared across more than the
shared firewire disk limit of two nodes. E.g. Tom's hardware did a good
review http://www.tomshardware.com/2006/04/07/going_the_sas_storage_way/

Hth,
Robert

_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

edlinuxguru at gmail

Aug 8, 2007, 4:17 AM

Post #6 of 9 (7075 views)
Permalink

Re: OCFS2 DRBD Heartbeat and high availability did not work out so well [In reply to]

A shared firewire disk and a shared storage array are both options, but then
you only have a single point of failure. Theoretically a good disk array is
a very resilient single point of failure still is an SPOF.

The great thing about DRBD active/active and OCFS2 is that you eliminate the
SPOF.

Also other options scsi locking/firewire are all related to specific
hardware/sans. This solution would be very generic.

Point taken most of my problems are DRBD, OCFS2 specific problems.

I understand the points about the resource being un-managed not being a good
thing. I know a fair amount about heartbeat colocations,orders in places.
Here is a more technical description of what i was trying to do heartbeat
wise.

Resource 1 VIP IP (used IPADDR2)
Resource 2 IP route (created an RA) for this
Resource 3 Web Scraping utility (used init script)
Resource 4 Process to work with web scraping and usenet data
Resource 5 Usenet Scraping utility
Resource 6 OCFS2 (cloned)
Resource 7 DRBD (cloned)

This was my first design
Order1 - Start 7 before 6
Group1 - Resource1 and Resource2 Process 3,4,5

This worked well. but since everything was grouped a failed resource in
Group1 caused everything to fail and possibly restart/move. Anyone connected
lost connected as the VIP left and came back a few seconds later. This
scenario was deemed unacceptable.

So then i tried writing a bunch of co location rules.
Collocate 45
Collocate 34
Collocate group1 and 4
That had the same effect though as grouping. an item failed it would cause
the collocation to fail, which would take down all the other collocation.

What I really needed was away to say. I need this resource to run wherever
VIP is running. VIP should only be running on a node with the shared disk
running.
PLACE seems only to be able to tell a resource to run on a node.

So I tried that implementation

Resource 1 VIP IP --PLACE node1 100
Resource 2 IP route --PLACE node1 100
Resource 3 Web Scraping utility --PLACE node1 100
Resource 4 Process to work with web scraping and usenet data --PLACE node1
100
Resource 5 Usenet Scraping utility --PLACE node1 100

This worked well because now everything is loosely coupled, and could still
failover, but failing over the VIP and route does not fail over resource 345

So neither place nor collocation can really express I need this resource to
run only where other resource is, but if this resource can not start don't
fail the parent. But if the parent does fail I need the resource to evaluate
that and move with it. A one way dependency.

On 8/8/07, Robert Wipfel <rawipfel [at] novell> wrote:
>
> >>> On Wed, Aug 8, 2007 at 1:09 AM, in message
> <26ef5e70708080009q26b0e795r7cdb919d5c3ad2a9 [at] mail>, "Andrew
> Beekhof"
> <beekhof [at] gmail> wrote:
> > On 8/8/07, Eddie C <edlinuxguru [at] gmail> wrote:
>
> [...]
>
> >> My post was more of a rant then anything else. I was looking for 50
> people
> >> or so to read my post and say. "You must be doing something wrong. I
> run
> >> DRBD with OCFS2 multi node actice/active and MySQL and its super fast
> and
> >> never crashes on two 100 mhz laptops"
>
> I know it's a bit off topic, but for this kind of setup (super low cost
> nodes), a shared firewire disk actually seems to work quite well. You'll
> have to load the firewire module with exclusive_login = 0, and then
> both servers will happily share that $150 fw disk ;-) and it actually
> works
> well enough to run Xen VMs off shared disk, using OCFS2 in userspace
> heartbeat mode, with Heartbeat2 managing the VMs as resources. There's
> some setup info here:
> http://www.oracle.com/technology/pub/articles/hunter_rac10gr2.html
>
> > Given how little you seem to be using heartbeat, perhaps the drbd or
> > ocfs2 lists might be a more appropriate forum.
> >
> >> The concept is great an active/active disk partition and heartbeat with
> no
> >> fancy SAN's. It works for stretches 3 or 4 weeks. But then I run into a
> >> weird locked directory that I cant delete or a file owned by '?'. Or
> the
> >> partition unmounts and the system will not reboot.
>
> I haven't checked prices recently, but multi-initiator serial attached
> shared SCSI is also a recent option, with a number of vendors providing
> low cost RAID enclosures, that can be shared across more than the
> shared firewire disk limit of two nodes. E.g. Tom's hardware did a good
> review http://www.tomshardware.com/2006/04/07/going_the_sas_storage_way/
>
> Hth,
> Robert
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

beekhof at gmail

Aug 8, 2007, 4:28 AM

Post #7 of 9 (7061 views)
Permalink

Re: OCFS2 DRBD Heartbeat and high availability did not work out so well [In reply to]

On 8/8/07, Eddie C <edlinuxguru [at] gmail> wrote:
> A shared firewire disk and a shared storage array are both options, but then
> you only have a single point of failure. Theoretically a good disk array is
> a very resilient single point of failure still is an SPOF.
>
> The great thing about DRBD active/active and OCFS2 is that you eliminate the
> SPOF.
>
> Also other options scsi locking/firewire are all related to specific
> hardware/sans. This solution would be very generic.
>
> Point taken most of my problems are DRBD, OCFS2 specific problems.
>
> I understand the points about the resource being un-managed not being a good
> thing. I know a fair amount about heartbeat colocations,orders in places.
> Here is a more technical description of what i was trying to do heartbeat
> wise.
>
> Resource 1 VIP IP (used IPADDR2)
> Resource 2 IP route (created an RA) for this
> Resource 3 Web Scraping utility (used init script)
> Resource 4 Process to work with web scraping and usenet data
> Resource 5 Usenet Scraping utility
> Resource 6 OCFS2 (cloned)
> Resource 7 DRBD (cloned)
>
> This was my first design
> Order1 - Start 7 before 6
> Group1 - Resource1 and Resource2 Process 3,4,5
>
> This worked well. but since everything was grouped a failed resource in
> Group1 caused everything to fail and possibly restart/move. Anyone connected
> lost connected as the VIP left and came back a few seconds later. This
> scenario was deemed unacceptable.
>
> So then i tried writing a bunch of co location rules.
> Collocate 45
> Collocate 34
> Collocate group1 and 4
> That had the same effect though as grouping. an item failed it would cause
> the collocation to fail, which would take down all the other collocation.
>
> What I really needed was away to say. I need this resource to run wherever
> VIP is running. VIP should only be running on a node with the shared disk
> running.
> PLACE seems only to be able to tell a resource to run on a node.
>
> So I tried that implementation
>
> Resource 1 VIP IP --PLACE node1 100
> Resource 2 IP route --PLACE node1 100
> Resource 3 Web Scraping utility --PLACE node1 100
> Resource 4 Process to work with web scraping and usenet data --PLACE node1
> 100
> Resource 5 Usenet Scraping utility --PLACE node1 100
>
> This worked well because now everything is loosely coupled, and could still
> failover, but failing over the VIP and route does not fail over resource 345
>
> So neither place nor collocation can really express I need this resource to
> run only where other resource is, but if this resource can not start don't
> fail the parent. But if the parent does fail I need the resource to evaluate
> that and move with it. A one way dependency.

finally some clue as to what version you're running!

please update, we've been able to do one-way colocation since 2.0.8

people really do make life hard on themselves when they don't provide
the relevant information to the people they want help from
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

edlinuxguru at gmail

Aug 8, 2007, 5:23 AM

Post #8 of 9 (7062 views)
Permalink

Re: OCFS2 DRBD Heartbeat and high availability did not work out so well [In reply to]

Do not worry I do not need an HA support with rules/cibs etc.
Regardless of what version of HA I use, it is the DRBD and OCFS2 is
where my issues lie at this point.

That is why I called my post 'did not work out so well' instead of a
good technical description of my problem. My main theme of the post is
that the setup is complex and that I am having a lot of multi-layer
issues.

I got started on this track because as I was reading about DRBD they
have a PDF that says what DRBD.8 supports. Active/Active OCFS2 GFS is
checked. I just wanted to share my experiences with the entire
process.

While I think fundamentally all the pieces are sound, troubleshooting,
configuring, and managing DRBD, OCFS2, Heatbeat has been quite an
adventure. We all know that setup is only half the battle.

The real question is does all this setup and management constitute and
an effective strategy. With my experimenting I would have to say no. I
think the loose couple of DRDB, OCFS2, and HA is not very effective.

The system takes far longer to setup. We normally build all our
systems with kickstart. The DRBD, HA, and OCFS parts are a manual
process that have to be done after install.

Then we have to integrate all these parts with Heartbeat. It took
quite a fair amount of play before I found the option that most people
found (clones of the init.d scripts)

After all the setup the system happens to be very unstable. Systems
that will not reboot, high IO wait. DRBD and OCFS2 kernel modules
refusing to load and unload again more of an OCFS/DRBD then an HA
problem.

Uptime is important, but time spend on management and setup is just as
important if not more. Fact is with a decent backup and a cold spare
system and a kickstart file I could probably bring this system back in
a half hour. Yet I have spent days/weeks worth or work/troubleshooting
DRDB/OCFS2 and Heartbeat. I have had to failover and back a number of
times. Also calling the data center for reboots.

On 8/8/07, Andrew Beekhof <beekhof [at] gmail> wrote:
> On 8/8/07, Eddie C <edlinuxguru [at] gmail> wrote:
> > A shared firewire disk and a shared storage array are both options, but then
> > you only have a single point of failure. Theoretically a good disk array is
> > a very resilient single point of failure still is an SPOF.
> >
> > The great thing about DRBD active/active and OCFS2 is that you eliminate the
> > SPOF.
> >
> > Also other options scsi locking/firewire are all related to specific
> > hardware/sans. This solution would be very generic.
> >
> > Point taken most of my problems are DRBD, OCFS2 specific problems.
> >
> > I understand the points about the resource being un-managed not being a good
> > thing. I know a fair amount about heartbeat colocations,orders in places.
> > Here is a more technical description of what i was trying to do heartbeat
> > wise.
> >
> > Resource 1 VIP IP (used IPADDR2)
> > Resource 2 IP route (created an RA) for this
> > Resource 3 Web Scraping utility (used init script)
> > Resource 4 Process to work with web scraping and usenet data
> > Resource 5 Usenet Scraping utility
> > Resource 6 OCFS2 (cloned)
> > Resource 7 DRBD (cloned)
> >
> > This was my first design
> > Order1 - Start 7 before 6
> > Group1 - Resource1 and Resource2 Process 3,4,5
> >
> > This worked well. but since everything was grouped a failed resource in
> > Group1 caused everything to fail and possibly restart/move. Anyone connected
> > lost connected as the VIP left and came back a few seconds later. This
> > scenario was deemed unacceptable.
> >
> > So then i tried writing a bunch of co location rules.
> > Collocate 45
> > Collocate 34
> > Collocate group1 and 4
> > That had the same effect though as grouping. an item failed it would cause
> > the collocation to fail, which would take down all the other collocation.
> >
> > What I really needed was away to say. I need this resource to run wherever
> > VIP is running. VIP should only be running on a node with the shared disk
> > running.
> > PLACE seems only to be able to tell a resource to run on a node.
> >
> > So I tried that implementation
> >
> > Resource 1 VIP IP --PLACE node1 100
> > Resource 2 IP route --PLACE node1 100
> > Resource 3 Web Scraping utility --PLACE node1 100
> > Resource 4 Process to work with web scraping and usenet data --PLACE node1
> > 100
> > Resource 5 Usenet Scraping utility --PLACE node1 100
> >
> > This worked well because now everything is loosely coupled, and could still
> > failover, but failing over the VIP and route does not fail over resource 345
> >
> > So neither place nor collocation can really express I need this resource to
> > run only where other resource is, but if this resource can not start don't
> > fail the parent. But if the parent does fail I need the resource to evaluate
> > that and move with it. A one way dependency.
>
> finally some clue as to what version you're running!
>
> please update, we've been able to do one-way colocation since 2.0.8
>
> people really do make life hard on themselves when they don't provide
> the relevant information to the people they want help from
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

beekhof at gmail

Aug 8, 2007, 8:04 AM

Post #9 of 9 (7077 views)
Permalink

Re: OCFS2 DRBD Heartbeat and high availability did not work out so well [In reply to]

On 8/8/07, Eddie C <edlinuxguru [at] gmail> wrote:
> Do not worry I do not need an HA support with rules/cibs etc.
> Regardless of what version of HA I use, it is the DRBD and OCFS2 is
> where my issues lie at this point.

Then, I must say, I find your choice of mailing list confusing :-)

Seriously though - please consider getting a more recent version of
heartbeat... you'll find many things have improved.
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads