Mailing List Archive: Zope: Dev

HA storage for zope

Index | Next | Previous | View Threaded

miles at jamkit

Jun 2, 2009, 4:04 PM

Post #1 of 13 (1067 views)
Permalink

HA storage for zope

Hi,

I'm looking at a HA setup for a project, and was wondering what the current
best way forward would be. There seem to be a few potential options around
for the storage end of the setup, and I'm wondering what's now considered
the current "best practice" for this sort of setup. Specifically, I've come
up with three options:

1. RelStorage
Using this, I think I can then take care of replication/mirroring as I have
access to a database that is already clustered in a HA environment. My
questions are:
+ Are the connections opened only when zope is started? Say I unplugged a
network cable and then plugged it back in again (breaking the database
connection) - will it be re-opened?
+ How does RelStorage take care of the blob storage?
+ Are there any details of big sites out there that use RelStorage
(particularly on Oracle)?

2. ZeoRAID
I could use ZeoRAID to write to maintain several independent storages
subject to stopping the ZeoRAID server being the single point of failure (I
posted some questions about this separately).

3. ZRS
If I'm right, ZRS still has a limitation in that there is a single server
for writes. Also, budget may be an issue. Having read the factsheet, I'm a
bit unsure as to how it is functionally different from ZeoRAID - can anyone
explain?

I also thought of using DirectoryStorage to take care of
replication/mirroring at the hardware level, for example using a clustered
SAN, but I think this would not work because the actual directory tree can
only be writeable by one zope at a time. Am I right about this?

Is there anything else worth considering?

I'd be grateful if anyone operating a HA-setup for their zope site, could
post your setup and experiences? To me, out of the above options,
RelStorage looks to be the most promising, simply because it will involve
the least setup and the database clustering and tools are pretty robust and
well-understood by the sql experts here.

Thanks in advance for your help, I will post a write-up once it's done,

Miles
_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

lists at zopyx

Jun 2, 2009, 8:56 PM

Post #2 of 13 (1010 views)
Permalink

Re: HA storage for zope [In reply to]

On 03.06.09 01:04, Miles Waller wrote:
> Is there anything else worth considering?

Dedicated HA-hardware. We are using a SUN Cluster for all of our
backend service like ZEO, Oracle, Postgres and LDAP.

-aj

Attachments:

lists.vcf (0.31 KB)

miles at jamkit

Jun 2, 2009, 11:14 PM

Post #3 of 13 (1010 views)
Permalink

Re: HA storage for zope [In reply to]

>> Is there anything else worth considering?
>
> Dedicated HA-hardware. We are using a SUN Cluster for all of our
> backend service like ZEO, Oracle, Postgres and LDAP.

I'm interested in how that works specifically with zeo. how does it
ensure the clustered Zeos don't try to write conflicting transactions
at the same time? Or is it set up for another service to start on
failover, rather than a load-balanced arrangement?

Thanks

Miles

_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

jens at bluedynamics

Jun 3, 2009, 12:08 AM

Post #4 of 13 (1008 views)
Permalink

Re: HA storage for zope [In reply to]

Am Wed, 03 Jun 2009 00:04:12 +0100 schrieb Miles Waller:

> Hi,
>
> I'm looking at a HA setup for a project, and was wondering what the
> current best way forward would be. There seem to be a few potential
> options around for the storage end of the setup, and I'm wondering
> what's now considered the current "best practice" for this sort of
> setup.
[...]
> Is there anything else worth considering?

the classics: DRBD and Heartbeat (for the zeo-server).

regards Jensens
--
Jens W. Klein - Klein & Partner KEG - BlueDynamics Alliance

_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

wichert at wiggy

Jun 3, 2009, 12:52 AM

Post #5 of 13 (1010 views)
Permalink

Re: HA storage for zope [In reply to]

Previously Miles Waller wrote:
> Hi,
>
> I'm looking at a HA setup for a project, and was wondering what the current
> best way forward would be. There seem to be a few potential options around
> for the storage end of the setup, and I'm wondering what's now considered
> the current "best practice" for this sort of setup. Specifically, I've come
> up with three options:
>
> 1. RelStorage
> Using this, I think I can then take care of replication/mirroring as I have
> access to a database that is already clustered in a HA environment. My
> questions are:
> + Are the connections opened only when zope is started? Say I unplugged a
> network cable and then plugged it back in again (breaking the database
> connection) - will it be re-opened?
> + How does RelStorage take care of the blob storage?
> + Are there any details of big sites out there that use RelStorage
> (particularly on Oracle)?

The initial RelStorage development was sponsored by Jarn and Elkj�p for
exactly that purpose: Elkj�p (a chain of Scandinavian electronics
stores) uses an Oracle cluster for storage of all their Zope data.

> 2. ZeoRAID
> I could use ZeoRAID to write to maintain several independent storages
> subject to stopping the ZeoRAID server being the single point of failure (I
> posted some questions about this separately).

If I remember correctly the ZeoRAID server is stateless, so you can
use a standard redundant setup to remove the single point of failure.

> 3. ZRS
> If I'm right, ZRS still has a limitation in that there is a single server
> for writes. Also, budget may be an issue. Having read the factsheet, I'm a
> bit unsure as to how it is functionally different from ZeoRAID - can anyone
> explain?

>From what I hear ZRS is also fairly expensive, especially compared to
the other two options which are free.

Wichert.

--
Wichert Akkerman <wichert[at]wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

hannosch at hannosch

Jun 3, 2009, 2:46 AM

Post #6 of 13 (1009 views)
Permalink

Re: HA storage for zope [In reply to]

Wichert Akkerman wrote:
> Previously Miles Waller wrote:
>> 1. RelStorage
>> Using this, I think I can then take care of replication/mirroring as I have
>> access to a database that is already clustered in a HA environment. My
>> questions are:
>> + Are the connections opened only when zope is started? Say I unplugged a
>> network cable and then plugged it back in again (breaking the database
>> connection) - will it be re-opened?

Connections will be re-opened should they be lost for any reason. I
remember only one instance of non-recoverable connection loss in an
entire year now. This was caused by maintenance tasks on the Oracle
cluster. The changes in 1.1.3 might have fixed even this rare problem.

>> + How does RelStorage take care of the blob storage?

RelStorage has no special blob storage support. You can wrap a
file-system based blob storage around a RelStorage. We don't use blobs
for this project at Jarn but store all information directly in Oracle.
Depending on the kind of data you need to store this can be inefficient.

>> + Are there any details of big sites out there that use RelStorage
>> (particularly on Oracle)?
>
> The initial RelStorage development was sponsored by Jarn and Elkjøp for
> exactly that purpose: Elkjøp (a chain of Scandinavian electronics
> stores) uses an Oracle cluster for storage of all their Zope data.

Exported to a regular Data.fs the data size is about 20gb. We have 20
Zope clients connected to the storage in the setup.

The only problem we found with this setup is that the Oracle Cluster has
quite a high latency, so reading and writing lots of objects to the
database can be slow. We aren't using the "cache-servers" option yet,
which should help with this specific problem.

One additional advice: You want to increase your temporary table space
to a much larger value. If you pack the database, the tempspace should
be able to contain your entire data set.

If you decide for RelStorage, Shane offers commercial support and
consultancy for it, which might be worth considering depending on the
criticality of the application.

Hanno

_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

mj at zopatista

Jun 3, 2009, 3:42 AM

Post #7 of 13 (1007 views)
Permalink

Re: HA storage for zope [In reply to]

On Wed, Jun 3, 2009 at 01:04, Miles Waller <miles[at]jamkit.com> wrote:
> 1. RelStorage
> Using this, I think I can then take care of replication/mirroring as I have
> access to a database that is already clustered in a HA environment. My
> questions are:
> + Are the connections opened only when zope is started? Say I unplugged a
> network cable and then plugged it back in again (breaking the database
> connection) - will it be re-opened?
> + How does RelStorage take care of the blob storage?

Hanno answered these 2 already.

> + Are there any details of big sites out there that use RelStorage
> (particularly on Oracle)?

The Elkjøp intranet cluster currently runs on 4 machines with 4
clients each, plus a maintenance client on a 5th machine. Varnish and
fail-over rigged load balancers complete the cluster. Currently there
are 100k+ content objects and 7k active users using this intranet
(plus another 3k inactive).

--
Martijn Pieters
_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

miles at jamkit

Jun 3, 2009, 10:41 AM

Post #8 of 13 (1008 views)
Permalink

Re: HA storage for zope [In reply to]

Hi Jens,
*
*>>* I'm looking at a HA setup for a project, and was wondering what the
*>>* current best way forward would be. There seem to be a few potential
*>>* options around for the storage end of the setup, and I'm wondering
*>>* what's now considered the current "best practice" for this sort of
*>>* setup.
*>>[...]
>>
>>* Is there anything else worth considering?
*>
>the classics: DRBD and Heartbeat (for the zeo-server).
>

Are you using this at the moment? My only worry would be that it can
take zeo a while to start up. do you ever have any problems when it
fails over, as the instances can't connect to anything? (in the past
we've had some issues when error messages like these make it into the
cache layer for some reason)

Thanks,

Miles

shane at hathawaymix

Jun 3, 2009, 2:10 PM

Post #9 of 13 (1001 views)
Permalink

Re: HA storage for zope [In reply to]

Hanno Schlichting wrote:
> Wichert Akkerman wrote:
>> [lots of great info about RelStorage]

I couldn't have said it better. :-) Thanks guys.

Shane

_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

jim at zope

Jun 11, 2009, 11:56 AM

Post #10 of 13 (844 views)
Permalink

Re: HA storage for zope [In reply to]

On Jun 2, 2009, at 7:04 PM, Miles Waller wrote:
...
> 3. ZRS
> If I'm right, ZRS still has a limitation in that there is a single
> server
> for writes.

Yup.

...

> Having read the factsheet, I'm a
> bit unsure as to how it is functionally different from ZeoRAID - can
> anyone
> explain?

I'm not very familiar with ZEORaid, but, I think ZEORaid replicates
during transactions. That is, it writes to each replica at the same
time it writes to the main storage. In ZRS, replication is
asynchronous from writes. This means that transactions only have to
wait for writes to be completed on the main storage.
ZRS also support chaining replication, meaning a secondary can
replicate from another secondary. For example, we typically have off-
site secondaries that replicate from an on-site secondary.

Note that ZRS has been used for years in large-scale critical
environments.

Also note that ZRS 2, which we've been using in production for more
than 2 years doesn't use spread and is much easier to configure and
more flexible than earlier versions.

See http://www.zope.com/products/zope_replication_services.html for
more information.

Jim

--
Jim Fulton
Zope Corporation

_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

chris at simplistix

Jun 12, 2009, 6:44 AM

Post #11 of 13 (829 views)
Permalink

Re: HA storage for zope [In reply to]

Jim Fulton wrote:
> Note that ZRS has been used for years in large-scale critical
> environments.
>
> Also note that ZRS 2, which we've been using in production for more
> than 2 years doesn't use spread and is much easier to configure and
> more flexible than earlier versions.
>
> See http://www.zope.com/products/zope_replication_services.html for
> more information.

It's a shame that the 2 or 3 times I've tried to organise purchase of
ZRS for customers, the Zope Corp sales process hasn't succeeded in
delivering anything :-(

(I'm not 100% on the details, but it may just have been that the prices
were extortionate...)

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk
_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

shane at hathawaymix

Jun 12, 2009, 1:35 PM

Post #12 of 13 (828 views)
Permalink

Re: HA storage for zope [In reply to]

Chris Withers wrote:
> It's a shame that the 2 or 3 times I've tried to organise purchase of
> ZRS for customers, the Zope Corp sales process hasn't succeeded in
> delivering anything :-(
>
> (I'm not 100% on the details, but it may just have been that the prices
> were extortionate...)

I disagree. The kind of fault tolerance ZRS provides is hard to achieve
without a lot of either time or money. I suspect ZRS is less expensive
than a new replicated Oracle setup, or paying a developer to invent
something new.

(I only recommend Oracle + RelStorage to people who have already
invested deeply in Oracle.)

Shane

_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

chris at simplistix

Jun 16, 2009, 5:27 AM

Post #13 of 13 (750 views)
Permalink

Re: HA storage for zope [In reply to]

Shane Hathaway wrote:
> Chris Withers wrote:
>> It's a shame that the 2 or 3 times I've tried to organise purchase of
>> ZRS for customers, the Zope Corp sales process hasn't succeeded in
>> delivering anything :-(
>>
>> (I'm not 100% on the details, but it may just have been that the
>> prices were extortionate...)
>
> I disagree.

You disagree that I failed to purchase ZRS from Zope Corp both times I
tried? ;-)

> The kind of fault tolerance ZRS provides is hard to achieve
> without a lot of either time or money.

Indeed, but when it's easier to get two hearbeat-failover'd servers and
some SAN to put the storage on, or go for clustered-MySQL-as-a-service
and RelStorage or ZEORAID, it's a bit of a moot point...

> I suspect ZRS is less expensive
> than a new replicated Oracle setup, or paying a developer to invent
> something new.

I'm not necessarily disagreeing with you, I wish getting hold of ZRS was
easier (not necessarily cheaper!)...

Chris
_______________________________________________
Zope-Dev maillist - Zope-Dev[at]zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com