Mailing List Archive: Python: Bugs

[issue14684] zlib set dictionary support inflateSetDictionary

Index | Next | Previous | View Threaded

report at bugs

Apr 27, 2012, 1:49 PM

Post #1 of 17 (117 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary

New submission from Sam Rushing <rushing.sam [at] gmail>:

Google's SPDY protocol requires the use of a pre-defined compression dictionary. The current zlib module doesn't expose the two functions for setting the dictionary.

This patch is minimal in the sense that it only exposes the two functions, but unfortunately the sequence of zlib calls required is clumsy: a call to inflate() must fail first (with an error of Z_NEED_DICT):

import zlib
zdict = b"thequickbrownfoxjumped\x00"
c = zlib.compressobj()
c.set_dictionary (zdict)
cd = c.compress (b"the quick brown fox jumped over the candlestick")
cd += c.flush()
d = zlib.decompressobj()
try:
print (d.decompress (cd))
except zlib.error as what:
if what.args[0].startswith ('Error 2 '):
d.set_dictionary (zdict)
print (d.flush())

Obviously a better way to catch/match Z_NEED_DICT would be nice.
A much nicer solution would allow you to associate the dictionary at creation time, rather than having to wait for an error condition. I can only guess that the zlib authors designed it that way for a reason?

----------
components: Extension Modules
files: zlib_set_dictionary.patch
keywords: patch
messages: 159492
nosy: Sam.Rushing
priority: normal
severity: normal
status: open
title: zlib set dictionary support inflateSetDictionary
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file25387/zlib_set_dictionary.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 12:31 PM

Post #2 of 17 (104 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Jesús Cea Avión <jcea [at] jcea> added the comment:

A dictionary could be provided an init time. Then, the Z_NEED_DICT could be intercepted in the binding and automatically inject the dictionary provided in the init.

Anyway, for a patch to be approved, we need a test too.

PS: Why is this NOT targeted to 3.3?. We have time, yet.

----------
nosy: +jcea

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 4:04 PM

Post #3 of 17 (109 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Sam Rushing <rushing.sam [at] gmail> added the comment:

I'm currently reworking this so that the dictionaries are provided in the constructor, and inflateSetDictionary() is called automatically. I've gone over the zlib RFC's and zlibmodule.c, and I'm fairly certain that whatever usage mode might involve multiple calls to SetDictionary() couldn't be supported by the zlib object anyway.

r.e. 3.3/3.4 - I merely chose the highest version number I saw, since this diff is against HEAD (as recommended by the docs). It's been quite a few years since I've submitted a patch to CPython!

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 4:38 PM

Post #4 of 17 (110 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Jesús Cea Avión <jcea [at] jcea> added the comment:

Retargetting to python 3.3.

If you hurry a bit and I find your patch acceptable (remember the tests!), I will try to integrate it.

----------
assignee: -> serwy
nosy: +serwy
versions: +Python 3.3 -Python 3.4

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 4:39 PM

Post #5 of 17 (107 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Changes by Jesús Cea Avión <jcea [at] jcea>:

----------
assignee: serwy ->
nosy: -serwy

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 4:48 PM

Post #6 of 17 (103 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Sam Rushing <rushing.sam [at] gmail> added the comment:

Ok, here's the patch. It has a single short test. For use with SPDY, it's necessary to test that the following stream data also correctly decompresses, I'll attach that to the next comment.

----------
Added file: http://bugs.python.org/file25446/zlib_set_dictionary_2.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 4:50 PM

Post #7 of 17 (103 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Sam Rushing <rushing.sam [at] gmail> added the comment:

This test is rather large, since it includes the predefined SPDY draft 2 dictionary, and some real-world data. Not sure what the policy is on including so much data in a test. If there's enough time I could make a smaller test that also verifies the correct behavior on a stream...

----------
Added file: http://bugs.python.org/file25447/tz2.py

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 5:00 PM

Post #8 of 17 (105 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Sam Rushing <rushing.sam [at] gmail> added the comment:

Argh, probably need to add the 'dict' field to the copy() method.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 5:50 PM

Post #9 of 17 (102 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Sam Rushing <rushing.sam [at] gmail> added the comment:

Updated version of the patch: extends the test, including a test of the streaming behavior needed for SPDY (both compression and decompression).

Also wik: copy()/uncopy() are aware of the 'dict' attribute.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 5:51 PM

Post #10 of 17 (102 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Changes by Sam Rushing <rushing.sam [at] gmail>:

Added file: http://bugs.python.org/file25449/zlib_set_dictionary_3.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 3, 2012, 8:33 PM

Post #11 of 17 (102 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Éric Araujo <merwok [at] netwok> added the comment:

Added a few comments on Rietveld.

----------
nosy: +eric.araujo

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 5, 2012, 3:01 AM

Post #12 of 17 (102 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Changes by Nadeem Vawda <nadeem.vawda [at] gmail>:

----------
nosy: +nadeem.vawda

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 5, 2012, 9:43 AM

Post #13 of 17 (101 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Nadeem Vawda <nadeem.vawda [at] gmail> added the comment:

I've posted a review on Rietveld.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 5, 2012, 12:33 PM

Post #14 of 17 (105 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Sam Rushing <rushing.sam [at] gmail> added the comment:

renames dict->zdict, splits the test, adds BEGIN/END around inflate call.

----------
Added file: http://bugs.python.org/file25472/zlib_set_dictionary_4.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 18, 2012, 12:42 PM

Post #15 of 17 (84 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Jesús Cea Avión <jcea [at] jcea> added the comment:

Status of this feature?. Ready to integrate?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 18, 2012, 12:58 PM

Post #16 of 17 (86 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Changes by Éric Araujo <merwok [at] netwok>:

----------
keywords: +needs review
stage: -> patch review

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 18, 2012, 11:08 PM

Post #17 of 17 (84 views)
Permalink

[issue14684] zlib set dictionary support inflateSetDictionary [In reply to]

Nadeem Vawda <nadeem.vawda [at] gmail> added the comment:

The code should be changed to use the buffer API (instead of accepting
only bytes objects). Other than that, I think it's ready for integration.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14684>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads