Mailing List Archive: Zope: Users

Zope threads and the GIL

Index | Next | Previous | View Threaded

mva.led at gmail

May 4, 2008, 6:44 AM

Post #1 of 13 (724 views)
Permalink

Zope threads and the GIL

Hi all, How does the GIL affect Zope threads?

Best regards,
Manuel.
_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

lists at zopyx

May 4, 2008, 6:51 AM

Post #2 of 13 (703 views)
Permalink

Re: Zope threads and the GIL [In reply to]

--On 4. Mai 2008 09:44:17 -0400 Manuel Vazquez Acosta <mva.led[at]gmail.com>
wrote:

> Hi all, How does the GIL affect Zope threads?

Zope threads are Python threads and therefore a thread switch is not
possible is some thread is holding the GIL.

-aj

mva.led at gmail

May 4, 2008, 8:12 AM

Post #3 of 13 (707 views)
Permalink

Re: Zope threads and the GIL [In reply to]

Andreas Jung wrote:
>
>
> --On 4. Mai 2008 09:44:17 -0400 Manuel Vazquez Acosta
> <mva.led[at]gmail.com> wrote:
>
>> Hi all, How does the GIL affect Zope threads?
>
> Zope threads are Python threads and therefore a thread switch is not
> possible is some thread is holding the GIL.
>
> -aj
>

That would mean that the only place where real multitasking occurs is
when fetching objects from ZEO and other I/O bounded tasks, isn't?

Best regards,
Manuel
_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

python at zeomega

May 4, 2008, 5:52 PM

Post #4 of 13 (698 views)
Permalink

Re: Zope threads and the GIL [In reply to]

another scenario is if you are using GIL aware Database adapters like mxODBC to access an RDBMS

sam
----- Original Message -----
From: "Manuel Vazquez Acosta" <mva.led[at]gmail.com>
To: "Andreas Jung" <lists[at]zopyx.com>
Cc: "Zope user list" <zope[at]zope.org>
Sent: Sunday, May 4, 2008 9:12:16 AM (GMT-0600) America/Guatemala
Subject: Re: [Zope] Zope threads and the GIL

Andreas Jung wrote:
>
>
> --On 4. Mai 2008 09:44:17 -0400 Manuel Vazquez Acosta
> <mva.led[at]gmail.com> wrote:
>
>> Hi all, How does the GIL affect Zope threads?
>
> Zope threads are Python threads and therefore a thread switch is not
> possible is some thread is holding the GIL.
>
> -aj
>

That would mean that the only place where real multitasking occurs is
when fetching objects from ZEO and other I/O bounded tasks, isn't?

Best regards,
Manuel
_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

lists at zopyx

May 4, 2008, 6:36 PM

Post #5 of 13 (696 views)
Permalink

Re: Zope threads and the GIL [In reply to]

--On 4. Mai 2008 19:52:38 -0500 sam <python[at]zeomega.com> wrote:

> another scenario is if you are using GIL aware Database adapters like
> mxODBC to access an RDBMS
>

What means GIL-aware in this context?

-aj

python at zeomega

May 4, 2008, 7:05 PM

Post #6 of 13 (695 views)
Permalink

Re: Zope threads and the GIL [In reply to]

Basically mxODBC is a third party extension and is GIL "aware" as it yields the GIL before accessing the DB

----- Original Message -----
From: "Andreas Jung" <lists[at]zopyx.com>
To: "sam" <python[at]zeomega.com>, "Manuel Vazquez Acosta" <mva.led[at]gmail.com>
Cc: "Zope user list" <zope[at]zope.org>
Sent: Sunday, May 4, 2008 7:36:08 PM (GMT-0600) America/Guatemala
Subject: Re: [Zope] Zope threads and the GIL

--On 4. Mai 2008 19:52:38 -0500 sam <python[at]zeomega.com> wrote:

> another scenario is if you are using GIL aware Database adapters like
> mxODBC to access an RDBMS
>

What means GIL-aware in this context?

-aj
_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

lists at zopyx

May 4, 2008, 7:34 PM

Post #7 of 13 (691 views)
Permalink

Re: Zope threads and the GIL [In reply to]

--On 4. Mai 2008 21:05:55 -0500 sam <python[at]zeomega.com> wrote:

> Basically mxODBC is a third party extension and is GIL "aware" as it
> yields the GIL before accessing the DB

Then it should play nicely with Python. But keep in mind that thread
switches can also be blocked by some thread without having the GIL locked.
We've seen such situations were a thread had to perform complex regular
expression matching operations. Obviously code in Python C extension is
also able to lock up the Python interpreter until the result is being
passed back to the Python interpreter.

Andreas

mva.led at gmail

May 5, 2008, 7:11 AM

Post #8 of 13 (684 views)
Permalink

Re: Zope threads and the GIL [In reply to]

Andreas Jung wrote:
> Then it should play nicely with Python. But keep in mind that thread
> switches can also be blocked by some thread without having the GIL locked.
> We've seen such situations were a thread had to perform complex regular
> expression matching operations. Obviously code in Python C extension is
> also able to lock up the Python interpreter until the result is being
> passed back to the Python interpreter.
>

Another Q: Is AccessControl GIL aware? I mean, this is probably the most
used component in Zope. Maybe a performance boost is experienced if
those checks can be made simultaneously.

Thinking harder, I realize that a tradeoff is in place: Objects can
change its security status (the security policy -- i.e, the security
object -- can be changed also, but is a weird use case, isn't it?). So
maybe write-lock is needed.

On the other hand, for objects with a high demand rate, which are kept
in memory, good results are to be expected.

CacheFu deals OK with Anonymous users, but a boost is to be expected for
authenticated users.

Best regards,
Manuel.
_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

lists at zopyx

May 5, 2008, 8:26 AM

Post #9 of 13 (683 views)
Permalink

Re: Zope threads and the GIL [In reply to]

--On 5. Mai 2008 10:11:32 -0400 Manuel Vazquez Acosta <mva.led[at]gmail.com>
wrote:

> Andreas Jung wrote:
>> Then it should play nicely with Python. But keep in mind that thread
>> switches can also be blocked by some thread without having the GIL
>> locked. We've seen such situations were a thread had to perform complex
>> regular expression matching operations. Obviously code in Python C
>> extension is also able to lock up the Python interpreter until the
>> result is being passed back to the Python interpreter.
>>
>
> Another Q: Is AccessControl GIL aware? I mean, this is probably the most
> used component in Zope. Maybe a performance boost is experienced if
> those checks can be made simultaneously.

Better tell us what your _real_ question/problem is. I am not getting the
point what you are really interested in.

-aj

mva.led at gmail

May 5, 2008, 8:47 AM

Post #10 of 13 (683 views)
Permalink

Re: Zope threads and the GIL [In reply to]

Andreas Jung wrote:
>
>
> --On 5. Mai 2008 10:11:32 -0400 Manuel Vazquez Acosta
> <mva.led[at]gmail.com> wrote:
>
>> Andreas Jung wrote:
>>> Then it should play nicely with Python. But keep in mind that thread
>>> switches can also be blocked by some thread without having the GIL
>>> locked. We've seen such situations were a thread had to perform complex
>>> regular expression matching operations. Obviously code in Python C
>>> extension is also able to lock up the Python interpreter until the
>>> result is being passed back to the Python interpreter.
>>>
>>
>> Another Q: Is AccessControl GIL aware? I mean, this is probably the most
>> used component in Zope. Maybe a performance boost is experienced if
>> those checks can be made simultaneously.
>
> Better tell us what your _real_ question/problem is. I am not getting
> the point what you are really interested in.
>
> -aj

I'm thinking in CPU-bounded parts of a request: Reindexing, Clustering,
Automatic Content Extraction, and the like...

We have implemented k-means clustering algorithm on top ZCTextIndex (now
plan to port to zc.catalog), although this is batch operation, some
other are per-query: Summarizing is one of those: Per cluster, and per
document. (Although this is maybe a question for the OTS list).

So I'm a little bit worried about how the resource utilization.

Best regards,
Manuel.

_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

dieter at handshake

May 5, 2008, 12:34 PM

Post #11 of 13 (676 views)
Permalink

Re: Zope threads and the GIL [In reply to]

Manuel Vazquez Acosta wrote at 2008-5-4 11:12 -0400:
> ...
>That would mean that the only place where real multitasking occurs is
>when fetching objects from ZEO and other I/O bounded tasks, isn't?

I depends what you mean with "real multitasking".

If you mean by this that several CPUs are concurrently active for your
Zope process, then you are near. You can, however, also write code
for expensive time consuming operations in a language different from
Python and then release the GIL before you active such code. Then,
this code, too, can occupy a CPU in addition to the Python activity.

--
Dieter
_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

mva.led at gmail

May 5, 2008, 3:57 PM

Post #12 of 13 (672 views)
Permalink

Re: Zope threads and the GIL [In reply to]

Dieter Maurer wrote:
> Manuel Vazquez Acosta wrote at 2008-5-4 11:12 -0400:
>> ...
>> That would mean that the only place where real multitasking occurs is
>> when fetching objects from ZEO and other I/O bounded tasks, isn't?
>
> I depends what you mean with "real multitasking".
>
> If you mean by this that several CPUs are concurrently active for your
> Zope process, then you are near.

Exactly. Several CPUs, I would like to use.

> You can, however, also write code
> for expensive time consuming operations in a language different from
> Python and then release the GIL before you active such code. Then,
> this code, too, can occupy a CPU in addition to the Python activity.
>

I'm very new to Python, and don't grasp yet the C API. Also I would like
to implement almost everything in Python: I need this, b/c my of my
team's skills in C are None.

Best regards,
Manuel.

_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

tseaver at palladion

May 5, 2008, 7:02 PM

Post #13 of 13 (665 views)
Permalink

Re: Zope threads and the GIL [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Manuel Vazquez Acosta wrote:
> Dieter Maurer wrote:
>> Manuel Vazquez Acosta wrote at 2008-5-4 11:12 -0400:
>>> ...
>>> That would mean that the only place where real multitasking occurs is
>>> when fetching objects from ZEO and other I/O bounded tasks, isn't?
>> I depends what you mean with "real multitasking".
>>
>> If you mean by this that several CPUs are concurrently active for your
>> Zope process, then you are near.
>
> Exactly. Several CPUs, I would like to use.

Generally, the rule-of-thumb for scaling Zope on a multi-CPU box is to
run an appserver instance per CPU, each talking to a shared storage
(ZEO, usually, but RelStorage would do as well).

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver[at]palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIH7wa+gerLs4ltQ4RAndLAJwM+qs7w5hQy0tA9ZY33mJBGPeTBgCeN8fi
J50GVau5mZLxpFnRyJLYCIc=
=09EA
-----END PGP SIGNATURE-----

_______________________________________________
Zope maillist - Zope[at]zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com