Mailing List Archive: Zope: Users

Making Zope stop storing old object versions

Index | Next | Previous | View Threaded

D.Sinang at spi-bpo

May 7, 2008, 5:08 PM

Post #1 of 8 (467 views)
Permalink

Making Zope stop storing old object versions

Hello,

We've got lots of Catalog reads and writes and our Data.fs is growing at
an immense rate.

Is there a way we can tell Zope to stop storing old object versions to
limit the growth and not have to pack that often ?

Regards,
Danny

tino at wildenhain

May 7, 2008, 7:54 PM

Post #2 of 8 (452 views)
Permalink

Re: Making Zope stop storing old object versions [In reply to]

Hi,

Sinang, Danny wrote:
> Hello,
>
> We've got lots of Catalog reads and writes and our Data.fs is growing
> at an immense rate.
>
> Is there a way we can tell Zope to stop storing old object
> versions to limit the growth and not have to pack that often ?

Well in fact it (Filestorage) does only "store" new object versions.
What it does not is removing the old version - this is done during packing.

When you have many Catalog writes or other parts of your object tree
experiences many writes, try to mount an extra ZODB (and storage) there.

You can also try some of the other storages like relstorage (where it
depends on the RDBMS how it handles writes)

Cheers
Tino

Attachments:

smime.p7s (3.17 KB)

dev101 at magma

May 8, 2008, 4:16 AM

Post #3 of 8 (445 views)
Permalink

Re: Making Zope stop storing old object versions [In reply to]

>> Sinang, Danny wrote:
>> Hello,
>>
>> We've got lots of Catalog reads and writes and our Data.fs is growing
>> at an immense rate.
>>
>> Is there a way we can tell Zope to stop storing old object
>> versions to limit the growth and not have to pack that often ?
>
>Well in fact it (Filestorage) does only "store" new object versions.
>What it does not is removing the old version - this is done during packing.
>
>When you have many Catalog writes or other parts of your object tree
>experiences many writes, try to mount an extra ZODB (and storage) there.
>
>You can also try some of the other storages like relstorage (where it
>depends on the RDBMS how it handles writes)

Alternatively, you could just replace your Catalog with an external db such
as MySQL (using zmysql database connections).

Jonathan

_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

D.Sinang at spi-bpo

May 8, 2008, 7:03 AM

Post #4 of 8 (442 views)
Permalink

RE: Making Zope stop storing old object versions [In reply to]

How do I go about replacing a Catalog using MySQL ? This is the first
time I've heard this.

-----Original Message-----
From: Jonathan [mailto:dev101[at]magma.ca]
Sent: Thursday, May 08, 2008 7:16 PM
To: Sinang, Danny
Cc: zope[at]zope.org
Subject: Re: [Zope] Making Zope stop storing old object versions

>> Sinang, Danny wrote:
>> Hello,
>>
>> We've got lots of Catalog reads and writes and our Data.fs is growing

>> at an immense rate.
>>
>> Is there a way we can tell Zope to stop storing old object versions
>> to limit the growth and not have to pack that often ?
>
>Well in fact it (Filestorage) does only "store" new object versions.
>What it does not is removing the old version - this is done during
packing.
>
>When you have many Catalog writes or other parts of your object tree
>experiences many writes, try to mount an extra ZODB (and storage)
there.
>
>You can also try some of the other storages like relstorage (where it
>depends on the RDBMS how it handles writes)

Alternatively, you could just replace your Catalog with an external db
such as MySQL (using zmysql database connections).

Jonathan

_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

dev101 at magma

May 8, 2008, 7:08 AM

Post #5 of 8 (440 views)
Permalink

Re: Making Zope stop storing old object versions [In reply to]

<snip>
----- Original Message -----
From: "Sinang, Danny" <D.Sinang[at]spi-bpo.com>
To: "Jonathan" <dev101[at]magma.ca>
Cc: <zope[at]zope.org>
Sent: Thursday, May 08, 2008 10:03 AM
Subject: RE: [Zope] Making Zope stop storing old object versions

How do I go about replacing a Catalog using MySQL ? This is the first
time I've heard this.
</snip>

Here's the relevant chapter in the Zope Book:
http://plope.com/Books/2_7Edition/RelationalDatabases.stx

Jonathan

_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

D.Sinang at spi-bpo

May 8, 2008, 7:15 AM

Post #6 of 8 (445 views)
Permalink

RE: Making Zope stop storing old object versions [In reply to]

Thanks Jonathan. But I thought you were referring to a way to make
Catalog writes transparently write to MySQL.

The article you're pointing to teaches how to use use ZSQLMethods, which
we're already doing.

But the product we're using to do workflows (OpenFlow) uses the
Zcatalog. So we either rewrite it to use MySQL, or we find a way to make
Catalogs write to MySQL.

Regards,
Danny

-----Original Message-----
From: Jonathan [mailto:dev101[at]magma.ca]
Sent: Thursday, May 08, 2008 10:08 PM
To: Sinang, Danny
Cc: zope[at]zope.org
Subject: Re: [Zope] Making Zope stop storing old object versions

<snip>
----- Original Message -----
From: "Sinang, Danny" <D.Sinang[at]spi-bpo.com>
To: "Jonathan" <dev101[at]magma.ca>
Cc: <zope[at]zope.org>
Sent: Thursday, May 08, 2008 10:03 AM
Subject: RE: [Zope] Making Zope stop storing old object versions

How do I go about replacing a Catalog using MySQL ? This is the first
time I've heard this.
</snip>

Here's the relevant chapter in the Zope Book:
http://plope.com/Books/2_7Edition/RelationalDatabases.stx

Jonathan

_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

zopelist at betabug

May 9, 2008, 2:46 AM

Post #7 of 8 (424 views)
Permalink

Re: Making Zope stop storing old object versions [In reply to]

(Thu, May 08, 2008 at 08:08:28AM +0800) Sinang, Danny
wrote/schrieb/egrapse:
> We've got lots of Catalog reads and writes and our Data.fs is growing
> at an immense rate.

Sometimes you have to look at your product code and figure out what it's
exactly doing. For example many times not very optimal products will
change all the instance when only a small attribute changes, resulting
in a zodb write that is much larger then really needed.

I documented an example of a hunt for a case like that in
"Quasi-Normal in Numbers": http://betabug.ch/blogs/ch-athens/708

Regards,

Sascha

_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

tino at wildenhain

May 9, 2008, 5:56 AM

Post #8 of 8 (426 views)
Permalink

Re: Making Zope stop storing old object versions [In reply to]

Hi,

Jonathan wrote:
>>> Sinang, Danny wrote:
>>> Hello,
...
>> You can also try some of the other storages like relstorage (where it
>> depends on the RDBMS how it handles writes)
>
> Alternatively, you could just replace your Catalog with an external db
> such as MySQL (using zmysql database connections).

well if MySQL is the right tool and does not introduce other problems.
It should be pointed out that along the powerful open source DBs there
is for example also postgresql which is extreamly good at handling
concurrent loads.

Cheers
Tino

Attachments:

smime.p7s (3.17 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com