Mailing List Archive: Python: Dev

Reworking the GIL

1 2 3

View All

Index | Next | Previous | View Threaded

solipsis at pitrou

Nov 7, 2009, 11:56 AM

Post #51 of 57 (103 views)
Permalink

Re: Reworking the GIL [In reply to]

Hello again,

> It shows that, on my platform for this specific benchmark:
> * newgil manage to leverage a significant amount of parallelism
> (1.7) where python 3.1 does not (3.1 is 80% slower)

I think you are mistaken:

-j0 (main thread only)
newgil: 47.483s, 47.605s, 47.512s
-j4 (4 consumer threads, main thread producing/waiting)
newgil: 48.428s, 49.217s

The runtimes are actually the same, so newgil doesn't leverage anything.
However, it doesn't degrade performance like 2.x/3.1 does :-)

> * newgil as a small impact on non multi-threaded execution
> (~1-2%) [may be worth investigating]

It goes from very slightly slower to very slightly faster and it is
likely to be caused by variations in generated output from the compiler.

> * 3.1 is more than 2 times slower than python 2.6 on this
> benchmark

That's the most worrying outcome I'd say. Are you sure the benchmark
really does the same thing? Under 2.6, you should add re.UNICODE to the
regular expression flags so as to match the 3.x semantics.

> [if I understood correctly in 3.x regex release the GIL].

Unless I've missed something it doesn't, no.
This could be a separate opportunity for optimization, if someone wants
to take a look at it.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

baptiste.lepilleur at gmail

Nov 8, 2009, 6:07 AM

Post #52 of 57 (101 views)
Permalink

Re: Reworking the GIL [In reply to]

2009/11/7 Antoine Pitrou <solipsis[at]pitrou.net>

>
> Hello again,
>
> > It shows that, on my platform for this specific benchmark:
> > * newgil manage to leverage a significant amount of parallelism
> > (1.7) where python 3.1 does not (3.1 is 80% slower)
>
> I think you are mistaken:
>
> -j0 (main thread only)
> newgil: 47.483s, 47.605s, 47.512s
> -j4 (4 consumer threads, main thread producing/waiting)
> newgil: 48.428s, 49.217s
>
> The runtimes are actually the same, so newgil doesn't leverage anything.
> However, it doesn't degrade performance like 2.x/3.1 does :-)
>

Ooops, I was comparing to 3.1 -j4 times which make no sense. One would think
I wanted to see that result since I though the GIL was released :/. This
greatly reduce the interest of this benchmark...

> > * 3.1 is more than 2 times slower than python 2.6 on this
> > benchmark
>
> That's the most worrying outcome I'd say. Are you sure the benchmark
> really does the same thing? Under 2.6, you should add re.UNICODE to the
> regular expression flags so as to match the 3.x semantics.
>

I've tried, but there is no change in result (the regexp does not use \w &
co but specify a lot unicode ranges). All strings are already of unicode
type in 2.6.

> > [if I understood correctly in 3.x regex release the GIL].
>
> Unless I've missed something it doesn't, no.
>
Hmmm, I was confusing with other modules (bzip2 & hashlib?). Looking back at
the result of your benchmark it's obvious. Is there a place where the list
of functions releasing the GIL is available? I did not see anything in
bz2.compress documentation.

solipsis at pitrou

Nov 8, 2009, 6:17 AM

Post #53 of 57 (101 views)
Permalink

Re: Reworking the GIL [In reply to]

Baptiste Lepilleur <baptiste.lepilleur <at> gmail.com> writes:
>
> I've tried, but there is no change in result (the regexp does not use \w &
> co but specify a lot unicode ranges). All strings are already of unicode
> type in 2.6.

No they aren't. You should add "from __future__ import unicode_literals" at the
start of your script and run it again.

> Hmmm, I was confusing with other modules (bzip2 & hashlib?). Looking back at
> the result of your benchmark it's obvious. Is there a place where the list of
> functions releasing the GIL is available? I did not see anything in
> bz2.compress documentation.

No, there isn't. You'd have to test, or read the source code.
But bz2 and zlib, for example, do release the GIL.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Nov 10, 2009, 1:26 PM

Post #54 of 57 (88 views)
Permalink

Re: Reworking the GIL [In reply to]

Hello again,

I've now removed priority requests, tried to improve the internal doc a
bit, and merged the changes into py3k.

Afterwards, the new Windows 7 buildbot has hung in test_multiprocessing,
but I don't know whether it's related.

Regards

Antoine.

Guido van Rossum <guido <at> python.org> writes:
>
>
> I would remove them -- if and when we find there's a need for
> something like them I suspect it won't look like what you currently
> have, so it's better not to complexificate your current patch with
> them. (I would remove all traces, including the conditions passed into
> the end macros.)
>
> >> My only suggestion so far: the description could use more explicit
> >> documentation on the various variables and macros and how they
> >> combine.
> >
> > Is it before or after
> > http://mail.python.org/pipermail/python-checkins/2009-
November/087482.html ?
>
> After. While that is already really helpful, not all the code is
> easily linked back to paragraphs in that comment block, and some
> variables are not mentioned by name in the block.
>

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

s.r at visotech

Nov 23, 2009, 12:54 AM

Post #55 of 57 (24 views)
Permalink

Re: Reworking the GIL [In reply to]

Hello,

I built something very similar for my company last year, and it’s been running
flawlessly in production at a few customer sites since, with avg. CPU usage ~50%
around the clock. I even posted about it on the Python mailing list [1] where
there was almost no resonance at that time. I never posted code, though --
nobody seemed to be too interested.

I am well aware that your current work is a lot more far-reaching than what I’ve
done, which is basically just a FIFO scheduler. I even added scheduling
priorities later which don’t work too great because the amount of time used for
a "tick" can vary by several orders of magnitude, as you know.

Thought you might be interested.

Regards
Stefan

[1] http://mail.python.org/pipermail/python-dev/2008-March/077814.html
[2] http://www.bestinclass.dk/index.php/2009/10/python-vs-clojure-evolving/
[3] www.dabeaz.com/python/GIL.pdf

PS On a slightly different note, I came across some Python bashing [2] yesterday
and somehow from there to David Beazley’s presentation about the GIL [3]. While
I don’t mind the bashing, the observations about the GIL seem quite unfair to me
because David’s measurements have been made on Mac OS X with its horribly slow
pthreads functions. I was not able to measure any slowdown on Linux.

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Nov 23, 2009, 2:29 AM

Post #56 of 57 (19 views)
Permalink

Re: Reworking the GIL [In reply to]

Stefan Ring wrote:
> [2] http://www.bestinclass.dk/index.php/2009/10/python-vs-clojure-evolving/
> [3] www.dabeaz.com/python/GIL.pdf
>
> PS On a slightly different note, I came across some Python bashing [2] yesterday
> and somehow from there to David Beazley’s presentation about the GIL [3]. While
> I don’t mind the bashing, the observations about the GIL seem quite unfair to me
> because David’s measurements have been made on Mac OS X with its horribly slow
> pthreads functions. I was not able to measure any slowdown on Linux.

We care about Mac OS X though, so even if the contention wasn't as bad
on a different OS, the Mac downsides matter.

With the GIL updates in place, it would be interesting to see that
analysis redone for 2.7/3.2 though.

Regards,
Nick.

P.S. As far as interest in the idea goes, the GIL is one of those areas
where it takes a fairly rare combination of interest, expertise and
established credibility to propose a change and get assent to it. You'll
notice that even Antoine had to resort to the "if nobody objects soon,
I'm checking this in" tactic to garner any responses. It's an area where
even those with relevant expertise still have to put aside a fair chunk
of time in order to properly review a proposed change :)

--
Nick Coghlan | ncoghlan[at]gmail.com | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

s.r at visotech

Nov 23, 2009, 3:25 AM

Post #57 of 57 (19 views)
Permalink

Re: Reworking the GIL [In reply to]

> I built something very similar for my company last year, and it’s been running
> flawlessly in production at a few customer sites since, with avg. CPU usage ~50%
> around the clock. I even posted about it on the Python mailing list [1] where
> there was almost no resonance at that time. I never posted code, though --
> nobody seemed to be too interested.

I've never bothered to make this tidy and nice, especially the
function naming (PySpecial_*) leaves some things to be desired. It's
not too bad, though; it just doesn't have commit-ready quality. I
don't worry about this anymore, so I just post what I have. Maybe
someone can make use of it.

Attachments:

py252-scheduling.patch (7.33 KB)

1 2 3

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com