Mailing List Archive: Linux-HA: Users

Perpeptual Newbie Question - gfs2 active/active

Index | Next | Previous | View Threaded

plarsen at famlarsen

Sep 14, 2010, 1:41 PM

Post #1 of 8 (653 views)
Permalink

Perpeptual Newbie Question - gfs2 active/active

Ok, I ventured into the linux-ha from using OCFS and a few other
commercial solutions a few years ago. I realized it's been a few years
since I last played with clustered file systems so I started from
scratch with "Clusters from Scratch".

My platform is two VMs running Fedora 13. While the guide is written for
F12 most of the stuff mentioned didn't change, except drbd is included
with F13. In general, the guide is very informative, but there are quite
a few misleading/errors in there - in particular in the DRBD and gfs2
sections.

I have a shared disk setup between my two VMs. Since my history is
databases that's how most of my clusters have worked: shared disks
between multiple active nodes. The guide basically uses gfs2 on top of
drbd which confuses me (why would you EVER want to do that?). And while
it briefly mentions a shared disk setup, it doesn't say HOW.

I've had a few pointers (thank you very much) on #linux-ha. But I'm
stuck in how to get cl2 (the second node) to mount and serve the gfs2
shared partition (/dev/vdb1 in my example). Right now, things mount and
looks right on cl1 (the first node). But I don't see the mount point
mounted on cl2.

Since my ultimate goal here is learning, I was wondering if anyone knew
of a guide that would talk to the active/active setup of GFS2?

Here's my current configuration. It's basic/primitive I know. Any ideas
on how I can get the last few things up and working?

crm(live)# configure show
node cl1 \
attributes standby="off"
node cl2
primitive ClusterIP ocf:heartbeat:IPaddr2 \
params ip="192.168.199.210" cidr_netmask="32" \
op monitor interval="30s"
primitive WebFS ocf:heartbeat:Filesystem \
params device="/dev/vdb1" directory="/var/www/html"
fstype="gfs2"
primitive WebSite ocf:heartbeat:apache \
params configfile="/etc/httpd/conf/httpd.conf" \
op monitor interval="60s"
primitive dlm ocf:pacemaker:controld \
op monitor interval="120s"
primitive gfs-control ocf:pacemaker:controld \
params daemon="gfs_controld.pcmk" args="-g 0" \
op monitor interval="120s"
clone dlm-clone dlm \
meta interleave="true"
clone gfs-clone gfs-control \
meta interleave="true"
colocation WebSite-with-WebFS inf: WebSite WebFS
colocation gfs-with-dlm inf: gfs-clone dlm-clone
colocation website-with-ip inf: WebSite ClusterIP
order WebSite-after-WebFS inf: WebFS WebSite
order apache-after-ip inf: ClusterIP WebSite
order start-gfs-after-dlm inf: dlm-clone gfs-clone
property $id="cib-bootstrap-options" \
dc-version="1.1.1-972b9a5f68606f632893fceed658efa085062f55" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no_quorum-policy="ignore" \
no-quorum-policy="ignore" \
last-lrm-refresh="1284509691"

--
Best Regards
Peter Larsen

Wise words of the day:
The grand leap of the whale up the Fall of Niagara is esteemed, by all
who have seen it, as one of the finest spectacles in nature.
-- Benjamin Franklin.

Attachments:

signature.asc (0.19 KB)

lars.ellenberg at linbit

Sep 14, 2010, 3:09 PM

Post #2 of 8 (634 views)
Permalink

Re: Perpeptual Newbie Question - gfs2 active/active [In reply to]

On Tue, Sep 14, 2010 at 04:41:11PM -0400, Peter Larsen wrote:
> Ok, I ventured into the linux-ha from using OCFS and a few other
> commercial solutions a few years ago. I realized it's been a few years
> since I last played with clustered file systems so I started from
> scratch with "Clusters from Scratch".
>
> My platform is two VMs running Fedora 13. While the guide is written for
> F12 most of the stuff mentioned didn't change, except drbd is included
> with F13. In general, the guide is very informative, but there are quite
> a few misleading/errors in there - in particular in the DRBD and gfs2
> sections.
>
> I have a shared disk setup between my two VMs. Since my history is
> databases that's how most of my clusters have worked: shared disks
> between multiple active nodes. The guide basically uses gfs2 on top of
> drbd which confuses me (why would you EVER want to do that?). And while
> it briefly mentions a shared disk setup, it doesn't say HOW.
>
> I've had a few pointers (thank you very much) on #linux-ha. But I'm
> stuck in how to get cl2 (the second node) to mount and serve the gfs2
> shared partition (/dev/vdb1 in my example). Right now, things mount and
> looks right on cl1 (the first node). But I don't see the mount point
> mounted on cl2.
>
> Since my ultimate goal here is learning, I was wondering if anyone knew
> of a guide that would talk to the active/active setup of GFS2?

Use that clusters-from-scratch guide you talked about earlier,
I'm referring to http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf

Skip the drbd stuff.

But don't skip the cloning of relevant primitives ;-)

Fast forward to "Reconfigure Pacemaker for Active/Active",

There ClusterIP will be cloned,
(you won't need that for GFS2, but may need that for apache without
explicit loadbalancer)

On Page 69, first line,

The WebFS will be cloned!
(That's the step you are desperately missing)

And WebSite will be cloned (you may want to clone something else
instead, if you are not doing apache)

So basically all you are missing for the GFS2 to be mounted on all
nodes is
configure clone WebFSClone WebFS

hth,

> Here's my current configuration. It's basic/primitive I know. Any ideas
> on how I can get the last few things up and working?
>
> crm(live)# configure show
> node cl1 \
> attributes standby="off"
> node cl2
> primitive ClusterIP ocf:heartbeat:IPaddr2 \
> params ip="192.168.199.210" cidr_netmask="32" \
> op monitor interval="30s"
> primitive WebFS ocf:heartbeat:Filesystem \
> params device="/dev/vdb1" directory="/var/www/html"
> fstype="gfs2"
> primitive WebSite ocf:heartbeat:apache \
> params configfile="/etc/httpd/conf/httpd.conf" \
> op monitor interval="60s"
> primitive dlm ocf:pacemaker:controld \
> op monitor interval="120s"
> primitive gfs-control ocf:pacemaker:controld \
> params daemon="gfs_controld.pcmk" args="-g 0" \
> op monitor interval="120s"
> clone dlm-clone dlm \
> meta interleave="true"
> clone gfs-clone gfs-control \
> meta interleave="true"
> colocation WebSite-with-WebFS inf: WebSite WebFS
> colocation gfs-with-dlm inf: gfs-clone dlm-clone
> colocation website-with-ip inf: WebSite ClusterIP
> order WebSite-after-WebFS inf: WebFS WebSite
> order apache-after-ip inf: ClusterIP WebSite
> order start-gfs-after-dlm inf: dlm-clone gfs-clone
> property $id="cib-bootstrap-options" \
> dc-version="1.1.1-972b9a5f68606f632893fceed658efa085062f55" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="2" \
> stonith-enabled="false" \
> no_quorum-policy="ignore" \
> no-quorum-policy="ignore" \
> last-lrm-refresh="1284509691"

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD� and LINBIT� are registered trademarks of LINBIT, Austria.
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

eric.schoeller at colorado

Sep 14, 2010, 7:17 PM

Post #3 of 8 (629 views)
Permalink

Re: Perpeptual Newbie Question - gfs2 active/active [In reply to]

I think there are plenty of uses for gfs2 ontop of drbd... but I could
be wrong. For example, if you're looking for an active/active
configuration but don't have access to a SAN for shared storage (say,
perhaps due to cost). Using DRBD allows you to simply use the internal
disks on both nodes for your shared storage. Then you obviously need a
clustered file system to ride on top of that, and gfs2 fits the bill.

On 09/14/10 14:41, Peter Larsen wrote:
> Ok, I ventured into the linux-ha from using OCFS and a few other
> commercial solutions a few years ago. I realized it's been a few years
> since I last played with clustered file systems so I started from
> scratch with "Clusters from Scratch".
>
> My platform is two VMs running Fedora 13. While the guide is written for
> F12 most of the stuff mentioned didn't change, except drbd is included
> with F13. In general, the guide is very informative, but there are quite
> a few misleading/errors in there - in particular in the DRBD and gfs2
> sections.
>
> I have a shared disk setup between my two VMs. Since my history is
> databases that's how most of my clusters have worked: shared disks
> between multiple active nodes. The guide basically uses gfs2 on top of
> drbd which confuses me (why would you EVER want to do that?). And while
> it briefly mentions a shared disk setup, it doesn't say HOW.
>
> I've had a few pointers (thank you very much) on #linux-ha. But I'm
> stuck in how to get cl2 (the second node) to mount and serve the gfs2
> shared partition (/dev/vdb1 in my example). Right now, things mount and
> looks right on cl1 (the first node). But I don't see the mount point
> mounted on cl2.
>
> Since my ultimate goal here is learning, I was wondering if anyone knew
> of a guide that would talk to the active/active setup of GFS2?
>
> Here's my current configuration. It's basic/primitive I know. Any ideas
> on how I can get the last few things up and working?
>
> crm(live)# configure show
> node cl1 \
> attributes standby="off"
> node cl2
> primitive ClusterIP ocf:heartbeat:IPaddr2 \
> params ip="192.168.199.210" cidr_netmask="32" \
> op monitor interval="30s"
> primitive WebFS ocf:heartbeat:Filesystem \
> params device="/dev/vdb1" directory="/var/www/html"
> fstype="gfs2"
> primitive WebSite ocf:heartbeat:apache \
> params configfile="/etc/httpd/conf/httpd.conf" \
> op monitor interval="60s"
> primitive dlm ocf:pacemaker:controld \
> op monitor interval="120s"
> primitive gfs-control ocf:pacemaker:controld \
> params daemon="gfs_controld.pcmk" args="-g 0" \
> op monitor interval="120s"
> clone dlm-clone dlm \
> meta interleave="true"
> clone gfs-clone gfs-control \
> meta interleave="true"
> colocation WebSite-with-WebFS inf: WebSite WebFS
> colocation gfs-with-dlm inf: gfs-clone dlm-clone
> colocation website-with-ip inf: WebSite ClusterIP
> order WebSite-after-WebFS inf: WebFS WebSite
> order apache-after-ip inf: ClusterIP WebSite
> order start-gfs-after-dlm inf: dlm-clone gfs-clone
> property $id="cib-bootstrap-options" \
> dc-version="1.1.1-972b9a5f68606f632893fceed658efa085062f55" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="2" \
> stonith-enabled="false" \
> no_quorum-policy="ignore" \
> no-quorum-policy="ignore" \
> last-lrm-refresh="1284509691"
>
>
>
>
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA [at] lists
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

plarsen at famlarsen

Sep 14, 2010, 11:12 PM

Post #4 of 8 (628 views)
Permalink

Re: Perpeptual Newbie Question - gfs2 active/active [In reply to]

On Wed, 2010-09-15 at 00:09 +0200, Lars Ellenberg wrote:
>
> Use that clusters-from-scratch guide you talked about earlier,
> I'm referring to
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>
> Skip the drbd stuff.

I did - except it's the part that holds the max_nodes configuration. I
was looking for where to put that. And I think I managed to solve it.

> But don't skip the cloning of relevant primitives ;-)
>
> Fast forward to "Reconfigure Pacemaker for Active/Active",
>
> There ClusterIP will be cloned,
> (you won't need that for GFS2, but may need that for apache without
> explicit loadbalancer)

Understood. Apache is to me just a primitive tool to prove a point. My
ultimate goal is session based systems like databases such as mysql and
maybe even a session based web-application over multiple nodes.

> On Page 69, first line,
>
> The WebFS will be cloned!
> (That's the step you are desperately missing)

I also had to add a ms (whcih created the clone) to set "master-max".
Ending up with the following - and seemingly working - configuration:

# crm configure show
node cl1 \
attributes standby="off"
node cl2
node cl3
primitive ClusterIP ocf:heartbeat:IPaddr2 \
params ip="192.168.199.210" cidr_netmask="32"
clusterip_hash="sourceip" \
op monitor interval="30s"
primitive WebFS ocf:heartbeat:Filesystem \
params device="/dev/vdb1" directory="/var/www/html"
fstype="gfs2"
primitive WebSite ocf:heartbeat:apache \
params configfile="/etc/httpd/conf/httpd.conf" \
op monitor interval="60s"
primitive dlm ocf:pacemaker:controld \
op monitor interval="120s"
primitive gfs-control ocf:pacemaker:controld \
params daemon="gfs_controld.pcmk" args="-g 0" \
op monitor interval="120s"
ms WebFSClone WebFS \
meta master-max="2"
clone WebIP ClusterIP \
meta globally-unique="true" clone-max="3" clone-node-max="3"
clone WebSiteClone WebSite
clone dlm-clone dlm \
meta interleave="true"
clone gfs-clone gfs-control \
meta interleave="true"
colocation WebSite-with-WebFS inf: WebSiteClone WebFSClone
colocation gfs-with-dlm inf: gfs-clone dlm-clone
colocation website-with-ip inf: WebSiteClone WebIP
order WebSite-after-WebFS inf: WebFSClone WebSiteClone
order apache-after-ip inf: WebIP WebSiteClone
order start-gfs-after-dlm inf: dlm-clone gfs-clone
property $id="cib-bootstrap-options" \
dc-version="1.1.1-972b9a5f68606f632893fceed658efa085062f55" \
cluster-infrastructure="openais" \
expected-quorum-votes="3" \
stonith-enabled="false" \
no_quorum-policy="ignore" \
no-quorum-policy="ignore" \
last-lrm-refresh="1284524815"

I also added a 3rd node just to see how that was done. That wasn't too
hard.

> And WebSite will be cloned (you may want to clone something else
> instead, if you are not doing apache)

Well, I need master-max specified and the challenge was to find out
where that was done.

> So basically all you are missing for the GFS2 to be mounted on all
> nodes is
> configure clone WebFSClone WebFS

Well, I had to do a few more things but yes, that was the primary driver
of what I did.

Thanks for feedback!

--
Best Regards
Peter Larsen

Wise words of the day:
Dijkstra probably hates me.
-- Linus Torvalds, in kernel/sched.c

Attachments:

signature.asc (0.19 KB)

plarsen at famlarsen

Sep 14, 2010, 11:17 PM

Post #5 of 8 (627 views)
Permalink

Re: Perpeptual Newbie Question - gfs2 active/active [In reply to]

On Tue, 2010-09-14 at 20:17 -0600, Eric Schoeller wrote:
> I think there are plenty of uses for gfs2 ontop of drbd... but I
> could
> be wrong. For example, if you're looking for an active/active
> configuration but don't have access to a SAN for shared storage (say,
> perhaps due to cost). Using DRBD allows you to simply use the
> internal
> disks on both nodes for your shared storage. Then you obviously need
> a
> clustered file system to ride on top of that, and gfs2 fits the bill.

Never in an active/active configuration. drbd makes a lot of sense if
you have a single active host but with an active/active solution you
have no way of dealing with distributed conflicts? drbd starts by
keeping two copies of everything on separate nodes and asyncronly
exchange the data between the nodes to keep the logical device
consistent. For an application that runs on two separate nodes
independently this just makes no sense. Not only are we tripling the
writes needed but you have no way to dealing with two master copies in
conflict.

My main concern is the huge overhead with the solution. drbd looks like
the way to go in passive/active solutions though.

--
Best Regards
Peter Larsen

Wise words of the day:
Linux: because a PC is a terrible thing to waste
-- ksh [at] cis put this on Tshirts in '93

Attachments:

signature.asc (0.19 KB)

andrew at beekhof

Sep 15, 2010, 12:00 AM

Post #6 of 8 (623 views)
Permalink

Re: Perpeptual Newbie Question - gfs2 active/active [In reply to]

On Wed, Sep 15, 2010 at 8:12 AM, Peter Larsen
<plarsen [at] famlarsen> wrote:
> On Wed, 2010-09-15 at 00:09 +0200, Lars Ellenberg wrote:
>>
>> Use that clusters-from-scratch guide you talked about earlier,
>> I'm referring to
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>
>> Skip the drbd stuff.
>
> I did - except it's the part that holds the max_nodes configuration. I
> was looking for where to put that. And I think I managed to solve it.
>
>> But don't skip the cloning of relevant primitives ;-)
>>
>> Fast forward to "Reconfigure Pacemaker for Active/Active",
>>
>> There ClusterIP will be cloned,
>> (you won't need that for GFS2, but may need that for apache without
>> explicit loadbalancer)
>
> Understood. Apache is to me just a primitive tool to prove a point. My
> ultimate goal is session based systems like databases such as mysql and
> maybe even a session based web-application over multiple nodes.
>
>> On Page 69, first line,
>>
>> � � � � The WebFS will be cloned!
>> � � � � (That's the step you are desperately missing)
>
> I also had to add a ms (whcih created the clone) to set "master-max".
> Ending up with the following - and seemingly working - configuration:

There is no need for that.

"clone WebFSClone WebFS" would have been sufficient.
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

lars.ellenberg at linbit

Sep 15, 2010, 5:25 AM

Post #7 of 8 (617 views)
Permalink

Re: Perpeptual Newbie Question - gfs2 active/active [In reply to]

On Wed, Sep 15, 2010 at 02:17:09AM -0400, Peter Larsen wrote:
> On Tue, 2010-09-14 at 20:17 -0600, Eric Schoeller wrote:
> > I think there are plenty of uses for gfs2 ontop of drbd... but I
> > could
> > be wrong. For example, if you're looking for an active/active
> > configuration but don't have access to a SAN for shared storage (say,
> > perhaps due to cost). Using DRBD allows you to simply use the
> > internal
> > disks on both nodes for your shared storage. Then you obviously need
> > a
> > clustered file system to ride on top of that, and gfs2 fits the bill.
>
> Never in an active/active configuration. drbd makes a lot of sense if
> you have a single active host but with an active/active solution you
> have no way of dealing with distributed conflicts? drbd starts by
> keeping two copies of everything on separate nodes and asyncronly

Nope. For obvious reasons,
we allow dual-primary mode only in _synchronous_ replication mode.

> exchange the data between the nodes to keep the logical device
> consistent. For an application that runs on two separate nodes
> independently this just makes no sense. Not only are we tripling the
> writes needed but you have no way to dealing with two master copies in
> conflict.

But yes, dual-primary is more complex to handle, as most replication link
breakage currently necessitates an immediate reboot of one of the servers.
So you better make sure your replication link is redundant and stable in
itself. We try to improve the handling.

> My main concern is the huge overhead with the solution. drbd looks like
> the way to go in passive/active solutions though.

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD� and LINBIT� are registered trademarks of LINBIT, Austria.
_______________________________________________
Linux-HA mailing list
Linux-HA [at] lists
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

plarsen at famlarsen

Sep 15, 2010, 7:01 AM

Post #8 of 8 (616 views)
Permalink

Re: Perpeptual Newbie Question - gfs2 active/active [In reply to]

On Wed, 2010-09-15 at 14:25 +0200, Lars Ellenberg wrote:
> On Wed, Sep 15, 2010 at 02:17:09AM -0400, Peter Larsen wrote:
> > Never in an active/active configuration. drbd makes a lot of sense if
> > you have a single active host but with an active/active solution you
> > have no way of dealing with distributed conflicts? drbd starts by
> > keeping two copies of everything on separate nodes and asyncronly
>
> Nope. For obvious reasons,
> we allow dual-primary mode only in _synchronous_ replication mode.

Wasn't aware that DRBD allowed for synchronous replication. I'm still
trying to grasp how multiple local volumes being updated across a
network helps? Not only do you have 3 IOs for each logical IO; you now
have journals too? I would like to learn a bit more about the
architecture being attempted with sync replication mode here - so I'll
start looking for some docs. I presume http://www.drbd.org/docs/about/
is the place to go to learn the details?

> > exchange the data between the nodes to keep the logical device
> > consistent. For an application that runs on two separate nodes
> > independently this just makes no sense. Not only are we tripling the
> > writes needed but you have no way to dealing with two master copies in
> > conflict.
>
> But yes, dual-primary is more complex to handle, as most replication link
> breakage currently necessitates an immediate reboot of one of the servers.
> So you better make sure your replication link is redundant and stable in
> itself. We try to improve the handling.

No kidding! It's a lot more complex - you have chosen a very impressive
challenge! I'm trying to hold DRBD up against the complexity of a
NAS/SAN configuration. Would you have any thoughts on comparing the
"decentralized storage" approach of DRBD with a centralized storage like
a NAS?

> > My main concern is the huge overhead with the solution. drbd looks like
> > the way to go in passive/active solutions though.

I'm still partial to that opinion. But I also should disclose that I
just learned about drbd a week ago. I wouldn't exactly call myself a
expert in that architecture ( ...... yet).

--
Best Regards
Peter Larsen

Wise words of the day:
I could dance with you till the cows come home. On second thought, I'd rather
dance with the cows till you come home.
-- Groucho Marx

Attachments:

signature.asc (0.19 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads