Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

asyncore fixes in Python 2.6 broke Zope's version of medusa

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


sidnei at enfoldsystems

Oct 8, 2008, 10:15 AM

Post #1 of 8 (283 views)
Permalink
asyncore fixes in Python 2.6 broke Zope's version of medusa

I am working on getting Zope to run (or at least, start) with Python
2.6. It actually starts right now after applying some patches, which
is amazing on itself, but it dies right away due to changes in
asyncore that break Zope's internal version of medusa.

I've opened a bug against Zope on Launchpad, but someone suggested
that it might actually be a bug in python, in the sense that it
changed asyncore in a backwards-incompatible way. I wouldn't go that
far, since I think it's more likely that Zope's version of medusa is
poking into asyncore internals instead:

https://bugs.edge.launchpad.net/zope2/+bug/280020

I suspect a patch similar to this might be needed:

http://codereview.appspot.com/744/diff/1/23

Anyone cares to comment?

--
Sidnei da Silva
Enfold Systems http://enfoldsystems.com
Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


janssen at parc

Oct 8, 2008, 10:30 AM

Post #2 of 8 (268 views)
Permalink
Re: asyncore fixes in Python 2.6 broke Zope's version of medusa [In reply to]

Sidnei da Silva <sidnei[at]enfoldsystems.com> wrote:

> https://bugs.edge.launchpad.net/zope2/+bug/280020

I think there are real issues here with both asynchat and Medusa.
Asynchat has been heavily re-written, and the "ac_out_buffer" has
apparently disappeared. But "ac_out_buffer_size" is still there. That
strikes me as odd, and probably means that asynchat.py needs more
changes. However, Medusa (basically just an application layer on top
of asyncore/asynchat) also needs to be re-written to take account of
the changes in asynchat.

Bill

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


josiah.carlson at gmail

Oct 8, 2008, 2:26 PM

Post #3 of 8 (268 views)
Permalink
Re: asyncore fixes in Python 2.6 broke Zope's version of medusa [In reply to]

On Wed, Oct 8, 2008 at 10:30 AM, Bill Janssen <janssen[at]parc.com> wrote:
> Sidnei da Silva <sidnei[at]enfoldsystems.com> wrote:
>
>> https://bugs.edge.launchpad.net/zope2/+bug/280020
>
> I think there are real issues here with both asynchat and Medusa.
> Asynchat has been heavily re-written, and the "ac_out_buffer" has
> apparently disappeared. But "ac_out_buffer_size" is still there. That
> strikes me as odd, and probably means that asynchat.py needs more
> changes. However, Medusa (basically just an application layer on top
> of asyncore/asynchat) also needs to be re-written to take account of
> the changes in asynchat.

ac_out_buffer was removed because it is unneeded (we have a deque; why
rely on an extra attribute?).
ac_out_buffer_size still remains as a blocksize in which to pre-split
outgoing data (if you have a 100k string you want to send, repeatedly
slicing it as you are able to send pieces is slow, but pre-slicing
chunks is fast, and generally results in being able to pass full chunk
to the underlying TCP/IP stack).

But yes, zope needs to be changed to reflect the updated
asyncore/asynchat semantics. Trust me; it's faster, cleaner, and
easier to use now.

- Josiah
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


sidnei at enfoldsystems

Oct 8, 2008, 2:31 PM

Post #4 of 8 (268 views)
Permalink
Re: asyncore fixes in Python 2.6 broke Zope's version of medusa [In reply to]

On Wed, Oct 8, 2008 at 6:26 PM, Josiah Carlson <josiah.carlson[at]gmail.com> wrote:
> ac_out_buffer was removed because it is unneeded (we have a deque; why
> rely on an extra attribute?).
> ac_out_buffer_size still remains as a blocksize in which to pre-split
> outgoing data (if you have a 100k string you want to send, repeatedly
> slicing it as you are able to send pieces is slow, but pre-slicing
> chunks is fast, and generally results in being able to pass full chunk
> to the underlying TCP/IP stack).
>
> But yes, zope needs to be changed to reflect the updated
> asyncore/asynchat semantics. Trust me; it's faster, cleaner, and
> easier to use now.

Great to know. So medusa should be changed similarly to the changes
made to asynchat? Your suggestion on the bugtracker is even better: to
subclass asynchat as much as possible.

--
Sidnei da Silva
Enfold Systems http://enfoldsystems.com
Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


janssen at parc

Oct 8, 2008, 2:49 PM

Post #5 of 8 (268 views)
Permalink
Re: asyncore fixes in Python 2.6 broke Zope's version of medusa [In reply to]

Sidnei da Silva <sidnei[at]enfoldsystems.com> wrote:

> Great to know. So medusa should be changed similarly to the changes
> made to asynchat?

Hmmm. Been a long time since a Medusa release, but I need a working
Medusa, too.

> Your suggestion on the bugtracker is even better: to
> subclass asynchat as much as possible.

That's basically what Medusa is, a set of subclasses.

Bill
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


janssen at parc

Oct 8, 2008, 6:39 PM

Post #6 of 8 (258 views)
Permalink
Re: asyncore fixes in Python 2.6 broke Zope's version of medusa [In reply to]

Josiah Carlson <josiah.carlson[at]gmail.com> wrote:

> But yes, zope needs to be changed to reflect the updated
> asyncore/asynchat semantics. Trust me; it's faster, cleaner, and
> easier to use now.

Just for completeness, I built a fresh 2.6, installed Medusa (from
http://www.amk.ca/python/code/medusa.html), and it runs just fine (well,
as well as it does on 2.5, anyway). I think this is a Zope issue.

Bill
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


guido at python

Oct 8, 2008, 7:18 PM

Post #7 of 8 (258 views)
Permalink
Re: asyncore fixes in Python 2.6 broke Zope's version of medusa [In reply to]

On Wed, Oct 8, 2008 at 6:39 PM, Bill Janssen <janssen[at]parc.com> wrote:
> Josiah Carlson <josiah.carlson[at]gmail.com> wrote:
>
>> But yes, zope needs to be changed to reflect the updated
>> asyncore/asynchat semantics. Trust me; it's faster, cleaner, and
>> easier to use now.
>
> Just for completeness, I built a fresh 2.6, installed Medusa (from
> http://www.amk.ca/python/code/medusa.html), and it runs just fine (well,
> as well as it does on 2.5, anyway). I think this is a Zope issue.

Way back in the day, Zope monkeypatched whole parts of asyncore,
replacing them with some of its own code. If that's still the case,
this breakage should be no surprise.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


tseaver at palladion

Oct 9, 2008, 9:08 AM

Post #8 of 8 (250 views)
Permalink
Re: asyncore fixes in Python 2.6 broke Zope's version of medusa [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Guido van Rossum wrote:
> On Wed, Oct 8, 2008 at 6:39 PM, Bill Janssen <janssen[at]parc.com> wrote:
>> Josiah Carlson <josiah.carlson[at]gmail.com> wrote:
>>
>>> But yes, zope needs to be changed to reflect the updated
>>> asyncore/asynchat semantics. Trust me; it's faster, cleaner, and
>>> easier to use now.
>> Just for completeness, I built a fresh 2.6, installed Medusa (from
>> http://www.amk.ca/python/code/medusa.html), and it runs just fine (well,
>> as well as it does on 2.5, anyway). I think this is a Zope issue.
>
> Way back in the day, Zope monkeypatched whole parts of asyncore,
> replacing them with some of its own code. If that's still the case,
> this breakage should be no surprise.

Zope has been hooking the 'asyncore.loop' function (wrapping it in order
to add a "clean shutdown' flog) since 2001 at least (the 2.5 branch is
the earliest checkout I have, and it was branched in early January 2002).

Zope also began patched asyncore's logging functions in 2.7, and later
updated that patch to make the logger use the stdlib 'logging' module.

I think the change we need to make (in ZServer/medusa/http_server.py) is
to emulate the new 'writeable' implementation in asynchat, at least when
running under 2.6.

Zope's 'ZServer.medusa.http_server.http_request.writable()'
implementation is::

def writable (self):
# this is just the normal async_chat 'writable',
# here for comparison
return self.ac_out_buffer or len(self.producer_fifo)


The Python 2.5 asynchat.asynchat.writable does::

def writable (self):
"predicate for inclusion in the writable for select()"
# return len(self.ac_out_buffer) or len(self.producer_fifo) or
# (not self.connected)
# this is about twice as fast, though not as clear.
return not (
(self.ac_out_buffer == '') and
self.producer_fifo.is_empty() and
self.connected
)

The Python 2.6 asynchat.asynchat.writable now does::

def writable (self):
"predicate for inclusion in the writable for select()"
return self.producer_fifo or (not self.connected)


medusa 0.5.4's medusa.http_server.http_request doesn't override
'writable', but it does have a broken 'writeable_for_proxy':


def writable_for_proxy (self):
# this version of writable supports the idea of a 'stalled'
# producer
# [i.e., it's not ready to produce any output yet] This is
# needed by
# the proxy, which will be waiting for the magic combination of
# 1) hostname resolved
# 2) connection made
# 3) data available.
if self.ac_out_buffer:
return 1
elif len(self.producer_fifo):
p = self.producer_fifo.first()
if hasattr (p, 'stalled'):
return not p.stalled()
else:
return 1


Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tseaver[at]palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFI7ix3+gerLs4ltQ4RAldnAKC/QLJHmdE9dxInkWuIGja0gtSXYwCcCJcH
6XooEwW/AkJ1ntmGyxi8urM=
=1kps
-----END PGP SIGNATURE-----

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact lists@gossamer-threads.com
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.