Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

Removal of intobject.h in 3.1

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


mal at egenix

Nov 21, 2009, 7:06 AM

Post #1 of 11 (1567 views)
Permalink
Removal of intobject.h in 3.1

The wiki page for porting to 3.x says:

http://wiki.python.org/moin/PortingExtensionModulesToPy3k
"""
long/int Unification

In Python 3.0, there is only one integer type. It is called int on the Python level, but actually
corresponds to 2.x's long type. In the C-API, PyInt_* functions are replaced by their PyLong_*
neighbors. The best course of action here is using the PyInt_* functions aliased to PyLong_* found
in intobject.h.
"""

However, intobject.h is no longer available. The checkin message
for the removal says:

"""
r71697 | mark.dickinson | 2009-04-18 12:12:16 +0200 (Sat, 18 Apr 2009) | 3 lines

The comments at the top of intobject.h say that it will be removed in 3.1.
Make it so.
"""

Since package developers are just starting to port things to 3.x and
many appear to be considering supporting both 2.7 and 3.1 (including
myself), I find it a bit strange that such an import aliasing header
was removed in 3.1.

IMHO, that's not really a good way to encourage people to try to provide
a smooth upgrade to the 3.x branch. Much to the contrary. 3.x should make
it easier for developers by providing more standard helpers like
the removed intobject.h header file.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Nov 21 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


eric at trueblade

Nov 21, 2009, 8:37 AM

Post #2 of 11 (1514 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

M.-A. Lemburg wrote:
> Since package developers are just starting to port things to 3.x and
> many appear to be considering supporting both 2.7 and 3.1 (including
> myself), I find it a bit strange that such an import aliasing header
> was removed in 3.1.

There's some discussion of this at http://bugs.python.org/issue7353

You might want to comment there.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Nov 21, 2009, 11:05 AM

Post #3 of 11 (1509 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

> IMHO, that's not really a good way to encourage people to try to provide
> a smooth upgrade to the 3.x branch. Much to the contrary. 3.x should make
> it easier for developers by providing more standard helpers like
> the removed intobject.h header file.

I think it's better than it sounds. The macros (unfortunately) allowed
to make non-obvious mistakes. Now that they are gone, people need to
really think of what precisely they want to do.

For example, consider

if (PyInt_Check(o)){
long val = PyInt_AsLong(o);
// process
} else if (PyLong_Check(o)) {
long long val = PyLong_AsLongLong(o);
// check for overflow
// process
}

With intobject.h, this code would continue to compile, but work
incorrectly, as the second case will never be executed. It would
be better to port this as

#if Py2.x
if (PyInt_Check(o)){
long val = PyInt_AsLong(o);
// process
} else
#endif
if (PyLong_Check(o)) {

i.e. eliminating the int case altogether. For another example,

long foo = PyInt_AsLong(Foo);

has a hidden error in 3.x, with intobject: PyLong_AsLong might
overflow, which the 2.x case doesn't.

So eliminating intobject.h likely helps avoiding subtle errors.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


casevh at gmail

Nov 21, 2009, 12:00 PM

Post #4 of 11 (1509 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

On Sat, Nov 21, 2009 at 11:05 AM, "Martin v. Löwis" <martin [at] v> wrote:
>> IMHO, that's not really a good way to encourage people to try to provide
>> a smooth upgrade to the 3.x branch. Much to the contrary. 3.x should make
>> it easier for developers by providing more standard helpers like
>> the removed intobject.h header file.
>
> I think it's better than it sounds. The macros (unfortunately) allowed
> to make non-obvious mistakes. Now that they are gone, people need to
> really think of what precisely they want to do.
>
> For example, consider
>
> if (PyInt_Check(o)){
>  long val = PyInt_AsLong(o);
>  // process
> } else if (PyLong_Check(o)) {
>  long long val = PyLong_AsLongLong(o);
>  // check for overflow
>  // process
> }
>
> With intobject.h, this code would continue to compile, but work
> incorrectly, as the second case will never be executed. It would
> be better to port this as
>
> #if Py2.x
> if (PyInt_Check(o)){
>  long val = PyInt_AsLong(o);
>  // process
> } else
> #endif
> if (PyLong_Check(o)) {
>
> i.e. eliminating the int case altogether. For another example,
>
> long foo = PyInt_AsLong(Foo);
>
> has a hidden error in 3.x, with intobject: PyLong_AsLong might
> overflow, which the 2.x case doesn't.
>
> So eliminating intobject.h likely helps avoiding subtle errors.

FWIW, I ported gmpy to Python 3.x without using intobject.h. I'm now
using the #if Py2.x ... #endif approach (almost) everywhere. The
same source compiles successfully with Python 2.4 to 3.2.

Case
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/casevh%40gmail.com
>
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mal at egenix

Nov 23, 2009, 2:44 AM

Post #5 of 11 (1482 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

Eric Smith wrote:
> M.-A. Lemburg wrote:
>> Since package developers are just starting to port things to 3.x and
>> many appear to be considering supporting both 2.7 and 3.1 (including
>> myself), I find it a bit strange that such an import aliasing header
>> was removed in 3.1.
>
> There's some discussion of this at http://bugs.python.org/issue7353
>
> You might want to comment there.

Thanks for pointing me to the that ticket.

Looks like Guido already commented on this, so intobject.h could
be revived in some form.

Since this file would only be used by extension modules and possibly
also include other helper macros, perhaps it'd be better to to rename
the file to py2compat.h or something along those lines ?!

We could then also have a py2compat.c to hold corresponding
C code, e.g. to provide compatibility wrappers of new APIs that
implement different semantics in 3.x.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Nov 23 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mal at egenix

Nov 23, 2009, 3:05 AM

Post #6 of 11 (1475 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

"Martin v. Löwis" wrote:
>> IMHO, that's not really a good way to encourage people to try to provide
>> a smooth upgrade to the 3.x branch. Much to the contrary. 3.x should make
>> it easier for developers by providing more standard helpers like
>> the removed intobject.h header file.
>
> I think it's better than it sounds. The macros (unfortunately) allowed
> to make non-obvious mistakes. Now that they are gone, people need to
> really think of what precisely they want to do.
>
> For example, consider
>
> if (PyInt_Check(o)){
> long val = PyInt_AsLong(o);
> // process
> } else if (PyLong_Check(o)) {
> long long val = PyLong_AsLongLong(o);
> // check for overflow
> // process
> }
>
> With intobject.h, this code would continue to compile, but work
> incorrectly, as the second case will never be executed. It would
> be better to port this as
>
> #if Py2.x
> if (PyInt_Check(o)){
> long val = PyInt_AsLong(o);
> // process
> } else
> #endif
> if (PyLong_Check(o)) {
>
> i.e. eliminating the int case altogether.

Sure, but that assumes that the original code already had support
for Python longs, which a lot of code doesn't.

In an ideal world, developers would add that code to their
extensions right away. In the real world, where developers only
have limited resources available, you'll get more 3.x ports
by making such ports as painless as possible while at the
same time not forcing them to alienate their 2.x user base.

The long support could then be added in later releases
of the extensions, giving the developers more time adapt.

> For another example,
>
> long foo = PyInt_AsLong(Foo);
>
> has a hidden error in 3.x, with intobject: PyLong_AsLong might
> overflow, which the 2.x case doesn't.

That's not quite true: PyInt_AsLong(obj) will try the
nb_int slot on non-integer objects which can return errors
(it returns -1 and sets the error message).

> So eliminating intobject.h likely helps avoiding subtle errors.

In the long run, yes. In the short run, other criteria are
more important, IMHO.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Nov 23 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Nov 23, 2009, 4:46 AM

Post #7 of 11 (1482 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

> In an ideal world, developers would add that code to their
> extensions right away. In the real world, where developers only
> have limited resources available, you'll get more 3.x ports
> by making such ports as painless as possible while at the
> same time not forcing them to alienate their 2.x user base.

Unfortunately, such 3.x code would be full of bugs.

>> For another example,
>>
>> long foo = PyInt_AsLong(Foo);
>>
>> has a hidden error in 3.x, with intobject: PyLong_AsLong might
>> overflow, which the 2.x case doesn't.
>
> That's not quite true: PyInt_AsLong(obj) will try the
> nb_int slot on non-integer objects which can return errors
> (it returns -1 and sets the error message).

However, the 2.x code will typically assume that the object
is an int object, in which case PyInt_AsLong can never fail.
So with intobject.h, the code will happily compile, but then
fail to detect an exception at run-time - causing other
difficult-to-find bugs.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


dickinsm at gmail

Nov 23, 2009, 9:12 AM

Post #8 of 11 (1479 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

On Mon, Nov 23, 2009 at 10:44 AM, M.-A. Lemburg <mal [at] egenix> wrote:
> Thanks for pointing me to the that ticket.
>
> Looks like Guido already commented on this, so intobject.h could
> be revived in some form.

I'm wondering how a resurrected intobject.h should be used: would
Linux distributors (for example) package up intobject.h with the rest
of the python-devel RPM/package/whatever so that Python extension
modules could just include it directly, or would it be better to
encourage the extension writers to make a copy of intobject.h to add
to the source for their extension?

In the first case, intobject.h would become a requirement for those
extensions, so we'd presumably end up committed to either
maintaining intobject.h for the lifetime of Python 3.x, or causing
some pain when it does eventually get deleted.

I quite like Benjamin's suggestion (in the issue tracker) of putting
it in Doc/includes.

> Since this file would only be used by extension modules and possibly
> also include other helper macros, perhaps it'd be better to to rename
> the file to py2compat.h or something along those lines ?!
>
> We could then also have a py2compat.c to hold corresponding
> C code, e.g. to provide compatibility wrappers of new APIs that
> implement different semantics in 3.x.

This is also an interesting idea.

Mark
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mal at egenix

Nov 23, 2009, 10:27 AM

Post #9 of 11 (1468 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

Mark Dickinson wrote:
> On Mon, Nov 23, 2009 at 10:44 AM, M.-A. Lemburg <mal [at] egenix> wrote:
>> Thanks for pointing me to the that ticket.
>>
>> Looks like Guido already commented on this, so intobject.h could
>> be revived in some form.
>
> I'm wondering how a resurrected intobject.h should be used: would
> Linux distributors (for example) package up intobject.h with the rest
> of the python-devel RPM/package/whatever so that Python extension
> modules could just include it directly, or would it be better to
> encourage the extension writers to make a copy of intobject.h to add
> to the source for their extension?
>
> In the first case, intobject.h would become a requirement for those
> extensions, so we'd presumably end up committed to either
> maintaining intobject.h for the lifetime of Python 3.x, or causing
> some pain when it does eventually get deleted.
>
> I quite like Benjamin's suggestion (in the issue tracker) of putting
> it in Doc/includes.

I'm not sure whether Doc/ is a good location for such things.

In the past, we've put such files in Misc/, e.g. the old Makefile.pre.in
based extension mechanism was shipped that way:

Extension writers would simply copy the file to their extension
directory, create a Setup.in file and then have the users
run "make -f Makefile.pre.in boot" to have a Makefile created
which then built the extension.

However, I'm not sure whether using the copy mechanism really
works out that well: it's easier to just have the file(s) in
the standard Python include file locations and then have
extension writers use

#include "py2compat.h"

where necessary. Extension writer can then still use a private
copy if needed, but most of the time they are probably better
served by using the Python provided standard one.

>> Since this file would only be used by extension modules and possibly
>> also include other helper macros, perhaps it'd be better to to rename
>> the file to py2compat.h or something along those lines ?!
>>
>> We could then also have a py2compat.c to hold corresponding
>> C code, e.g. to provide compatibility wrappers of new APIs that
>> implement different semantics in 3.x.
>
> This is also an interesting idea.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Nov 23 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


solipsis at pitrou

Nov 23, 2009, 10:47 AM

Post #10 of 11 (1475 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

M.-A. Lemburg <mal <at> egenix.com> writes:
>
> We could then also have a py2compat.c to hold corresponding
> C code, e.g. to provide compatibility wrappers of new APIs that
> implement different semantics in 3.x.

If the semantic differences are embodied in the builtin object types I wonder
how you can make a compatibility wrapper (short of reimplementing the 2.x
types). An int is not a long, a bytes is not a str is not a unicode.

I'm not sure concealing the differences between 2.x and 3.x behind such a
wrapper is a good idea. It would be better if people became aware of / learnt
about the new semantics.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Nov 23, 2009, 12:12 PM

Post #11 of 11 (1476 views)
Permalink
Re: Removal of intobject.h in 3.1 [In reply to]

> I'm not sure concealing the differences between 2.x and 3.x behind such a
> wrapper is a good idea. It would be better if people became aware of / learnt
> about the new semantics.

+1. When porting a larger code base, I always came up with a custom set
of macros, specific to the way the modules were written in the package I
ported. In particular for PyString_, some of the code meant to deal with
bytes, other code with text. There was no way to tell, short of reading
and understanding the code. As you'll have to review the complete code,
anyway, the savings you gain by using macros somebody else wrote are
really minor.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.