Mailing List Archive: Python: Bugs

[issue13959] Re-implement parts of imp in pure Python

1 2 3

View All

Index | Next | Previous | View Threaded

report at bugs

Apr 14, 2012, 11:29 AM

Post #1 of 56 (194 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python

Antoine Pitrou <pitrou [at] free> added the comment:

I think this should be a blocker for 3.3.

----------
nosy: +pitrou
priority: normal -> release blocker

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 15, 2012, 12:57 PM

Post #2 of 56 (185 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Brett Cannon <brett [at] python> added the comment:

Just because I was thinking about it, I wonder if necessarily all the frozen stuff really needs to stay in import.c. I mean a frozen module is really just an entry in an array of structs that has a name of an char*[]. I don't see why one couldn't simply have a get_frozen_bytes() method to convert that char*[] into a bytes object and use that to construct a module in pure Python code.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 15, 2012, 1:09 PM

Post #3 of 56 (184 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset d777f854a66e by Brett Cannon in branch 'default':
Issue #13959: Rename imp to _imp and add Lib/imp.py and begin
http://hg.python.org/cpython/rev/d777f854a66e

----------
nosy: +python-dev

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 15, 2012, 1:12 PM

Post #4 of 56 (184 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Brett Cannon <brett [at] python> added the comment:

OK, so I have started to check this stuff in, but I think it's best to do it piecemeal. Going forward I would like to commit in units of functions being replaced, and prioritize stuff that cuts out C code (e.g. the load_*() methods, find_module(), etc.). That way it's clear that progress is being made. Obviously the best way to tell if code is hanging on just because of imp is to comment out the interface code and see what your compiler complains about in terms of dead code (or at least clang is good at this).

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 15, 2012, 2:52 PM

Post #5 of 56 (186 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Brett Cannon <brett [at] python> added the comment:

It looks like in order to get a clear sign of what it will take to remove various parts of import.c, imp.load_module() needs to go along with imp.load_package() (since they call each other in the C code). You also have to take care of imp.reload(), but I am simplifying the C code greatly and will take care of referencing other code in the module.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 15, 2012, 2:56 PM

Post #6 of 56 (184 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset 4dce3afc392c by Brett Cannon in branch 'default':
Issue #13959: Simplify imp.reload() by relying on a module's
http://hg.python.org/cpython/rev/4dce3afc392c

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 15, 2012, 3:34 PM

Post #7 of 56 (184 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Brett Cannon <brett [at] python> added the comment:

I am seeing how this is going to go down. the load_dynamic, load_source, etc. family of functions are simply dispatched to by load_module(). So to keep some semblance of backwards-compatibility, each of those modules need to be implemented and then have load_module() simply dispatch to them based on the "type" of module it is.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 15, 2012, 5:25 PM

Post #8 of 56 (184 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset 2df37938b8e1 by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.load_module() in imp.py.
http://hg.python.org/cpython/rev/2df37938b8e1

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 15, 2012, 7:29 PM

Post #9 of 56 (186 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset 4256df44023b by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.load_package() in imp.py.
http://hg.python.org/cpython/rev/4256df44023b

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 16, 2012, 3:47 AM

Post #10 of 56 (183 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Changes by Berker Peksag <berker.peksag [at] gmail>:

----------
nosy: +berker.peksag

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 16, 2012, 8:48 AM

Post #11 of 56 (183 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Brett Cannon <brett [at] python> added the comment:

>From Eric Smith on python-dev:

> + suffix, mode, type_ = details
> + if mode and (not mode.startswith(('r', 'U'))) or '+' in mode:
> + raise ValueError('invalid file open mode {!r}'.format(mode))

Should this be:
if mode and (not mode.startswith(('r', 'U')) or '+' in mode):

to match:

> - if (*mode) {
...
> - if (!(*mode == 'r' || *mode == 'U') || strchr(mode, '+')) {
> - PyErr_Format(PyExc_ValueError,
> - "invalid file open mode %.200s", mode);

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 16, 2012, 7:11 PM

Post #12 of 56 (184 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset 3b5b4b4bb43c by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.load_source() in imp.py.
http://hg.python.org/cpython/rev/3b5b4b4bb43c

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 16, 2012, 10:57 PM

Post #13 of 56 (182 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Eric Snow <ericsnowcurrently [at] gmail> added the comment:

This is a mostly-working sketch of find_module() in imp.py. I've run out of time tonight, but wanted to get it up in case it's useful to anyone else (a.k.a Brett). If nobody's touched it before then, I'll finish it up tomorrow.

----------
Added file: http://bugs.python.org/file25250/imp_find_module.diff

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 17, 2012, 9:08 AM

Post #14 of 56 (182 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Brett Cannon <brett [at] python> added the comment:

I doubt I will beat you to it, Eric, but I did want to say that your overall design was what I had in my head when I was thinking about how to re-implement the function, so keep at it!

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 17, 2012, 4:14 PM

Post #15 of 56 (183 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset 66bd85bcf916 by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.load_compiled() in imp.py.
http://hg.python.org/cpython/rev/66bd85bcf916

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 17, 2012, 10:51 PM

Post #16 of 56 (182 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Eric Snow <ericsnowcurrently [at] gmail> added the comment:

2 "problems" with that last patch:

* the types of the loaders that get returned by _bootstrap._find_module() are not the classes in _bootstrap (e.g. _frozen_importlib._SourceFileLoader). That doesn't smell right.
* tokenize.get_encoding? is saying that Lib/test/badsyntax_pep3120.py has an encoding of utf-8, when test_find_module_encoding is expecting it to not. So does PyTokenizer_FindEncodingFilename (which the old import code used) behaving differently than tokenize.get_encoding()? FYI, tokenize.get_encoding() is what importlib uses...

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 17, 2012, 10:54 PM

Post #17 of 56 (185 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Eric Snow <ericsnowcurrently [at] gmail> added the comment:

rather, tokenize.detect_encoding()

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 18, 2012, 7:25 AM

Post #18 of 56 (182 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Brett Cannon <brett [at] python> added the comment:

You could change Lib/imp.py to have ``import _frozen_importlib as _bootstrap`` if you want the same modules coming from imp, but I would argue against changing importlib itself being changed as that complicates development since if you screw up importlib._bootstrap when you compile it becomes a major pain to revert the importlib.h change, recompile, and continue to do that until you get it right. Plus you would only care about this if you are doing isinstance() checks on what kind of loader you have which you shouldn't care about since we have clearly defined ABCs to test against.

As for Lib/test/badsyntax_pep3120.py, it *does* have a source encoding of UTF-8 since it does not explicitly specify an encoding. Based on the name I'm assuming the file itself has bad UTF-8 characters, but that doesn't mean that the file is not supposed to be interpreted as UTF-8.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 18, 2012, 7:49 AM

Post #19 of 56 (183 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Eric Snow <ericsnowcurrently [at] gmail> added the comment:

On the _frozen_importlib point, I'm fine with that. It was just counter-intuitive that the importlib._bootstrap functions were returning loaders whose classes weren't also defined there.

I'll have another look at that test tonight and run the patch through the full test suite, but otherwise I think find_module() is basically done.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 19, 2012, 12:53 AM

Post #20 of 56 (182 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Eric Snow <ericsnowcurrently [at] gmail> added the comment:

Looking it over, I'm confident that tokenizer.detect_encoding() does not raise a SyntaxError where PyTokenizer_FindEncodingFilename() does. I've run out of time tonight, but I'll look at it more tomorrow.

Once find_module() is done, I'd like to move on to reload(), which I expect will be pretty straightforward at this point. Then the feasibility of issue14618 should be clear.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 20, 2012, 3:04 PM

Post #21 of 56 (183 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset c820aa9c0c00 by Brett Cannon in branch 'default':
Issue #13959: Keep imp.get_magic() in C code, but cache in importlib
http://hg.python.org/cpython/rev/c820aa9c0c00

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 20, 2012, 7:07 PM

Post #22 of 56 (186 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Eric Snow <ericsnowcurrently [at] gmail> added the comment:

I'd still like to consider this a bit more. When you're trying to understand imports, having one place to look (Lib/importlib/_bootstrap.py) is better than two, especially when the one is pure Python code. So it still may be worth it to pull in the odds and ends that play into that.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 21, 2012, 3:53 PM

Post #23 of 56 (160 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset b773a751c2e7 by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.cache_from_source() in Lib/imp.py.
http://hg.python.org/cpython/rev/b773a751c2e7

New changeset ea46ebba8a0f by Brett Cannon in branch 'default':
Issue #13959: Re-implement imp.source_from_cache() in Lib/imp.py.
http://hg.python.org/cpython/rev/ea46ebba8a0f

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 21, 2012, 4:02 PM

Post #24 of 56 (160 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Brett Cannon <brett [at] python> added the comment:

Time for a recap!

I am personally not bothering with moving imp.get_tag() and imp.get_magic() to importlib._bootstrap as the amount of C code required to support the public C API is not worth it. If someone else once to do the work then by all means attach a patch.

I still need to port imp.find_module() (and the various constants) and will base it off of Eric's code.

NullImporter will get ported once issue #14605 (exposing the import machinery) lands.

get_suffixes() will also get ported, but that is a little bit more involved as I need to change how _DynLoadFiletab works by only storing the file extensions and not the other fluff (which is the same for all OSs).

After all of that is done then I will expose some API in importlib to replace find_module() and load_*() functions and then deprecate them (see issue #14551 for an ongoing discussion the possible API).

----------
assignee: -> brett.cannon
dependencies: +Make import machinery explicit
stage: -> needs patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Apr 21, 2012, 5:58 PM

Post #25 of 56 (159 views)
Permalink

[issue13959] Re-implement parts of imp in pure Python [In reply to]

Eric Snow <ericsnowcurrently [at] gmail> added the comment:

Yeah, I'm hoping to keep pressing those odds and ends forward. I have one lingering, oddball bug in find_module, but that patch is pretty much standing on its own.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13959>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

1 2 3

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads