Mailing List Archive: Python: Bugs

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2

Index | Next | Previous | View Threaded

report at bugs

Feb 6, 2012, 6:11 AM

Post #1 of 9 (43 views)
Permalink

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2

Changes by Éric Araujo <merwok [at] netwok>:

----------
title: exension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2 -> extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13590>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Feb 6, 2012, 8:48 PM

Post #2 of 9 (38 views)
Permalink

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2 [In reply to]

Ned Deily <nad [at] acm> added the comment:

When building and installing C extension modules, distutils attempts to use the same compiler and some compiler options as Python itself was built with. For the current (3.2.2 and 2.7.2) 64-bit/32-bit python installers, the standard Apple-supplied gcc-4.2 in Xcode 3.2.x on OS X 10.6 was used. As of Xcode 4.2, now standard for OS X 10.7 and optional for OS X 10.6, Apple no longer ships gcc-4.2 in Xcode, in favor of clang and the transitional llvm-gcc. Moving the entire Python build to another compiler is a major undertaking, requiring careful testing, which is underway. A compiler change may well be deemed too risky for a bug-fix release.

In any case, we can no longer assume that the same build compiler will be available on most user systems. While it is possible for the user to manually override the distutils defaults by setting the CC and LDSHARED environment variables properly, setting the latter is tedious. For example, to manually override to clang, the following is currently needed:

CC=clang
LDSHARED='clang -bundle -undefined dynamic_lookup \
-arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -g'
python setup.py ...

Attached is a patch that attempts to "do the right thing" for OS X. First, if the default build compiler is gcc-4.2 and the compiler has not been explicitly overridden by defining CC, distutils will check for the presence of gcc-4.2 on the default path. If gcc-4.2 is not found and if clang is found, it will automatically substitute clang as the build compiler. Second, if CC is used to override the compiler selection but LDSHARED is not defined, distutils will substitute the CC value into the default LDSHARED value. This allows simple one-line compiler overrides, like:

CC=llvm-gcc python setup.py ...

To minimize the risk of unintended side effects, these changes would apply to OS X only.

I propose applying this patch to 3.2 (for 3.2.3) and 2.7 (for 2.7.3) as well as provisionally to default for 3.3; a second patch will be needed with similar changes to packaging. After the evaluation of compiler alternatives is complete and we decide what to do for 3.3, this approach might change.

----------
keywords: +patch
stage: -> patch review
Added file: http://bugs.python.org/file24442/issue13590_distutils.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13590>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Feb 8, 2012, 8:42 AM

Post #3 of 9 (38 views)
Permalink

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2 [In reply to]

Éric Araujo <merwok [at] netwok> added the comment:

Would you think it acceptable to judge that the problem is Apple’s and that we could do only a doc change with the not-so-hard envvar override?

I understand that you’d like to fix the problem for all OS X users in one go, but the new behavior seems a bit too magical for me.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13590>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Feb 8, 2012, 8:50 AM

Post #4 of 9 (38 views)
Permalink

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2 [In reply to]

K Richard Pixley <rich [at] noir> added the comment:

I think a better solution that declaring it to be apple's bug would be to release one binary for pre-10.7, (or maybe 10.6 with the current xcode), and a different binary for post-10.7.

This isn't an apple "bug" in the sense that there's anything wrong nor in the sense that they would ever "fix" it. It's simply a difference between xcode versions. So the choices would seem to be a) code around it or b) release different binaries.

I'm ok with either solution. I'm not sure what would be best as I'm not sure I know all of the concerns involved.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13590>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Feb 8, 2012, 9:01 AM

Post #5 of 9 (38 views)
Permalink

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2 [In reply to]

Ned Deily <nad [at] acm> added the comment:

Éric, no, the problem is not Apple's in the sense that we enforce the use of the build compiler. Without a fix along this line would mean that essentially *every* user of python.org Pythons on the latest releases of OS X would have to ensure that the environment variable override is in place for every Distutils install of an extension module. That strikes me as unacceptable. Distutils already does equally magical things. I think this is the best solution at the moment. Ronald, what's your opinion?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13590>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Feb 8, 2012, 9:13 AM

Post #6 of 9 (38 views)
Permalink

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2 [In reply to]

Éric Araujo <merwok [at] netwok> added the comment:

> the problem is not Apple's in the sense that we enforce the use of the build compiler.
Well, yes, and this is a known limitation, so we could argue that Apple broke distutils. But...

> Without a fix along this line would mean that essentially *every* user of python.org Pythons on
> the latest releases of OS X would have to ensure that the environment variable override is in
> place for every Distutils install of an extension module. That strikes me as unacceptable.
> Distutils already does equally magical things.
Given that we did similar changes to support Debian multiarch instead of having people use environment variables, I change my position and support your patch. (Comments on Rietveld.)

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13590>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Feb 9, 2012, 3:37 AM

Post #7 of 9 (38 views)
Permalink

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2 [In reply to]

Ronald Oussoren <ronaldoussoren [at] mac> added the comment:

distutils would not only have to check for gcc-4.2 vs. clang, but also needs to strip "-arch ppc" from the compiler flags when it is present and it cannot use gcc-4.2. Otherwise you cannot build extensions for the 32-bit python installer on OSX 10.7 with Xcode 4.2.

It is sadly enough not possible to create a binary installer that creates a working installation including building of extensions on all supported versions of OSX without adding special-case code to distutils/packaging.

Note that there already is some special-case code to support the universal binary builds on OSX 10.3.9 (which does not have a compiler that can build universal binaries).

W.r.t. the patch: is it really necessary to use a subshell to look for the compiler? I'd either walk os.environ['PATH'] myself or use xcodebuild to locate binaries.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13590>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Feb 10, 2012, 4:09 AM

Post #8 of 9 (38 views)
Permalink

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2 [In reply to]

Roundup Robot <devnull [at] psf> added the comment:

New changeset 29507a2acdb5 by Ned Deily in branch '2.7':
Issue #13590: On OS X 10.7 and 10.6 with Xcode 4.2, building
http://hg.python.org/cpython/rev/29507a2acdb5

New changeset 5c784b0f263d by Ned Deily in branch '3.2':
Issue #13590: On OS X 10.7 and 10.6 with Xcode 4.2, building
http://hg.python.org/cpython/rev/5c784b0f263d

New changeset 35bd40b16a91 by Ned Deily in branch 'default':
Issue #13590: merge
http://hg.python.org/cpython/rev/35bd40b16a91

----------
nosy: +python-dev

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13590>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Feb 10, 2012, 4:26 AM

Post #9 of 9 (38 views)
Permalink

[issue13590] extension module builds fail with python.org OS X installers on OS X 10.7 and 10.6 with Xcode 4.2 [In reply to]

Ned Deily <nad [at] acm> added the comment:

Éric: I've replied to your review comments in Rietveld. Thanks.

Ronald: "distutils would not only have to check for gcc-4.2 vs. clang, but also needs to strip "-arch ppc" from the compiler flags when it is present and it cannot use gcc-4.2. Otherwise you cannot build extensions for the 32-bit python installer on OSX 10.7 with Xcode 4.2."

The 32-bit-only installers uses gcc-4.0, not -4.2 and it uses the 10.4u SDK, neither of which is available with Xcode 4.*. The patch does not currently attempt to support building extensions for the 32-bit-only installers. Besides stripping arch ppc, it would also have to substitute an available SDK (10.6 is the oldest SDK shipping with Xcode 4.2). CFLAGS and LDFLAGS would also have to be edited to remove ppc and the use of 10.3 as a deployment target results in warning messages:
#warning Building for Intel with Mac OS X Deployment Target < 10.4 is invalid
even though for simple cases it seems to work.

But I think trying to automatically support the 32-bit-only configuration is too intrusive and unnecessary for most users. There are disadvantages to trying to use the old 32-bit-only configuration on newer systems, such as using the older, more broken, or less feature-rich system APIs. One reasonable use case I can think of is for app developers who want to distribute Python-based apps on OS X that run on a range of systems, say 10.4 to 10.7. For those presumably more sophisticated users, documenting the unsupported environment variable settings to override when attempting to build with Xcode 4 should be sufficient. (The safer options of building with Xcode 3 on 10.6 (or earlier) or building their own custom Pythons remain, of course.)

"W.r.t. the patch: is it really necessary to use a subshell to look for the compiler? I'd either walk os.environ['PATH'] myself or use xcodebuild to locate binaries."

I suppose that could be done. I've tried to minimize the performance impact by only performing the check the first time it is needed and caching the result so it's only done once per Python invocation. I'm concerned about the added complexity of getting the PATH parsing and semantics right; I don't know of any Python code in the standard library that does this. My immediate concern is for the imminent code cutoffs for Python 3.2.3 and 2.7.3 so I've committed the patch, updated for other review comments, for them.

I'm deferring looking at doing a PATH walk for the 3.3 version of the code which is needed along with corresponding changes to packaging. There also should be some OS X installer README and/or other documentation of this and other 10.7 and 10.6 issues. I'm considering how to handle that.

----------
priority: release blocker -> deferred blocker
stage: patch review -> needs patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue13590>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads