Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Bugs

[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC

  

 
Python bugs RSS feed   Index | Next | Previous | View Threaded


report at bugs

Oct 7, 2009, 1:09 PM

Post #1 of 11 (481 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Marc-Andre Lemburg <mal [at] egenix> added the comment:

Christoph Gohlke wrote:
>
> Christoph Gohlke <cgohlke [at] uci> added the comment:
>
> The attached patch uses a regular expression.

Much better, thanks.

----------
title: Do not embed manifest files in *.pyd when compiling with MSVC -> Do not embed manifest files in *.pyd when compiling with MSVC

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 12, 2009, 3:24 PM

Post #2 of 11 (444 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Marc-Andre Lemburg <mal [at] egenix> added the comment:

Looking at the description of manifest files, it appears that just
removing the assemblyIdentity-element results in an invalid manifest file:

http://msdn.microsoft.com/en-us/library/aa374219(VS.85).aspx

It appears that the entire dependency-element referencing the MS VC90
CRT DLL has to be removed in order to make the manifest correct again.

Looking at the schema, it's enough to just remove the
dependentAssembly-element:

http://msdn.microsoft.com/en-us/library/aa375635(VS.85).aspx

Reading up on the manifest trouble-shooting page at (near the end of the
page):

http://msdn.microsoft.com/en-us/library/ms235342.aspx

it may actually be enough to just place the Microsoft.VC90.CRT.manifest
file into the Python folder (the one with python.exe and the CRT DLLs):

"""
If the operating system fails to find the CRT or any other assembly as a
shared assembly, it starts looking for the assembly as a private
assembly. It searches for private assemblies in the following order:

1. Check the application local folder for a manifest file with name
<assemblyName>.manifest. In this example, the loader tries to find
Microsoft.VC90.CRT.manifest in the same folder as appl.exe. If the
manifest is found, the loader loads the CRT DLL from the application
folder. If the CRT DLL is not found, load fails.
"""

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 12, 2009, 5:51 PM

Post #3 of 11 (443 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Christoph Gohlke <cgohlke [at] uci> added the comment:

This patch also removes empty dependentAssembly elements after removing
the VC.CRT assemblyIdentity element.

It seems not enough to just place the Microsoft.VC90.CRT.manifest and VC
runtime DLL files into the Python folder. On a system without the VC
runtime installed in winsxs, trying to import a pyd module build without
the patch fails with "ImportError: DLL load failed: ...". Import works
when 1) the manifest and VC runtime DLLs are also placed in the same
folder as the pyd file, or 2) the Microsoft Visual C++ 2008
redistributable package is installed, or 3) the pyd file is build with
the patch installed. Tested with Windows XP SP3, Python 2.6.3,
matplotlib 0.99.1, Visual Studio 2008 SP1.

----------
Added file: http://bugs.python.org/file15113/msvc9compiler_stripruntimes_regexp2.diff

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 13, 2009, 2:46 PM

Post #4 of 11 (436 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Koen van de Sande <koen [at] tibed> added the comment:

> It may actually be enough to just place the
Microsoft.VC90.CRT.manifest file into the Python folder (the one with
python.exe and the CRT DLLs).

I concur with what Christoph says, that is how the embedded
installation works. However, the .pyd files are not in the Python main
folder. And therefore they shouldn't have a manifest for the runtimes
(because it will look in the current folder relative to the DLL/pyd),
because then the runtimes of the main python.exe will be used (which
can then be in the same folder as the 'embedded' MSVCR location).

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 16, 2009, 9:26 AM

Post #5 of 11 (415 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Nick Touran <nick [at] partofthething> added the comment:

Just to share my recent experience with this issue: I was attempting to
get Python 2.6 working with py4mpi, matplotlib, and pymssql on a 64-bit
Windows Vista based HPC cluster via x-copy deployment without the charm
of administrative privileges on the server and on my workstation (which
is XP). It wasn't easy, but I got it working.

I attempted to apply the patch listed here and recompile the packages
that complained about Side-by-side configurations, but got in too deep
while compiling matplotlib on windows (recall I don't have admin
privileges and can't install the prereqs) so I dug deeper.

I found this: http://blog.kalmbachnet.de/?postid=80 which suggests
removing the publicKeyToken attribute from the manifests to force local
DLLs. This would possibly give the same effect as not embedding the
manifests in the first place using the patch. So I went in, using
VS2008, and removed this attribute from the embedded manifests in
python26.dll, python.exe, and the manifest file in C:\python26. I also
copied msvcm90.dll, msvcp90.dll, and msvcr90.dll from the 9.0.21022.8
folder in c:\Windows\WinSxS into c:\Python26. With that, matplotlib
started working, but pymssql still did not.

I then renamed _mssql.pyd to _mssql.dll so that VS2008 could recognize
the manifest, removed the publicKeyToken attribute, and renamed it back
to _mssql.pyd, but it still complained. Finally, using depends.exe from
the internet, I noticed msvcr71.dll was required on my local machine, so
I tried copying it over to the remote machine, into the site-packages
folder. Bam. It worked. So while I don't really consider this a
solution, it definitely worked for what I needed. Any other 3rd party
modules that complain in the future in my weird xcopy-deployment will
undergo manifest-stripping via VS2008 and other dependency checking via
depends.exe. Yay.

By the way, the pymssql was compiled with VS2005, which is why it
requires msvcr71.dll. If I could rebuild it with VS2008, I imagine that
step wouldn't be necessary. While I do have VS2008 on hand, I don't have
MS SQL 2020 Super Awesome Developer Edition, or whatever 3GB package
installs the required header.

----------
nosy: +ntouran

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 17, 2009, 10:56 PM

Post #6 of 11 (410 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Martin v. Löwis <martin [at] v> added the comment:

What's a procedure for testing this patch (please be as precise as
possible)?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 18, 2009, 12:29 PM

Post #7 of 11 (409 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Christoph Gohlke <cgohlke [at] uci> added the comment:

There are two levels of testing.

First, on a Windows development system with Visual Studio 2008, Python
2.6.2+, and the msvc9compiler_stripruntimes_regexp2.diff patch applied,
verify that the embedded manifest in distutil generated PYD extension
files is stripped of the WinSxS VC.CRT dependency and that the
extensions can still be imported. The attached script (testpyd.py)
builds a minimal extension (testpyd.pyd), searches for the WinSxS VC.CRT
dependency within the pyd file, and imports the extension. The script
should run without exception.

Second, verify that distutil generated PYD extensions can also be
imported on a test system, which has no Visual Studio 2008 runtime
installed in %WINDIR%\WinSxS:
Install Windows XP SP3, preferably in a virtual machine. Do not install
any additional programs or patches, which might install the Visual
Studio 2008 runtime. Then install Python 2.6.2+ using the "Install just
for me" option. Try import the testpyd extension built on the
development system: run "python.exe -c 'import testpyd'" in a directory
containing the testpyd.pyd file (not the Python installation directory
containing python.exe). This should work without exception.

Importing the testpyd extension built without the patch fails on the
test system:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: DLL load failed: This application has failed to start
because the application configuration is incorrect. Reinstalling the
application may fix this problem.

Tested with Python 2.6.3 for Windows 32-bit.

----------
Added file: http://bugs.python.org/file15161/testpyd.py

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 28, 2009, 7:07 PM

Post #8 of 11 (375 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Christoph Gohlke <cgohlke [at] uci> added the comment:

Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
for MinGW users. The following C program is using the Python C API to
import the testpyd extension generated by testpyd.py. When compiled with
MinGW, the program fails with "ImportError: DLL load failed:..." if the
PYD extension is compiled with MSVC9 and the patch is applied. The
program works if 1) it is compiled with MSVC9, or 2) the testpyd
extension is build without the patch, or 3) the files
Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
executable and the manifest is also embedded into the executable (e.g.
using mt.exe).


/* Import the testpyd.pyd module. */
#include <Python.h>
int main(void) {
Py_Initialize();
{
PyObject *p = PyImport_ImportModule("testpyd");
}
Py_Finalize();
return 0;
}

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 29, 2009, 5:16 AM

Post #9 of 11 (376 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Marc-Andre Lemburg <mal [at] egenix> added the comment:

Christoph Gohlke wrote:
>
> Christoph Gohlke <cgohlke [at] uci> added the comment:
>
> Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
> for MinGW users. The following C program is using the Python C API to
> import the testpyd extension generated by testpyd.py. When compiled with
> MinGW, the program fails with "ImportError: DLL load failed:..." if the
> PYD extension is compiled with MSVC9 and the patch is applied. The
> program works if 1) it is compiled with MSVC9, or 2) the testpyd
> extension is build without the patch, or 3) the files
> Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
> executable and the manifest is also embedded into the executable (e.g.
> using mt.exe).
>
>
> /* Import the testpyd.pyd module. */
> #include <Python.h>
> int main(void) {
> Py_Initialize();
> {
> PyObject *p = PyImport_ImportModule("testpyd");
> }
> Py_Finalize();
> return 0;
> }

I'm not sure whether this is related to the problem in question.
Does MinGW also embed a manifest in the generated executable ?

If it doesn't, then this is more likely a problem with how
MinGW works per default, than with Python or distutils.

The fact that the above does work without the patch applied
suggests that MinGW does not embed the manifest in the executable
(since the Windows linker then uses the one from the PYD file).

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 29, 2009, 7:40 AM

Post #10 of 11 (372 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Christoph Gohlke <cgohlke [at] uci> added the comment:

My last comment was merely reporting that this patch can break MinGW
based build processes. It did surprise some developers that their
programs embedding matplotlib, which is now build with the MSVC9 patch,
stopped working.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Oct 29, 2009, 8:44 AM

Post #11 of 11 (372 views)
Permalink
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC [In reply to]
Koen van de Sande <koen [at] tibed> added the comment:

The MinGW breakage probably comes from the same "issue" as encountered
in http://bugs.python.org/issue4120#msg80908 and #msg80909: the main
application, which embeds Matplotlib, does not have a manifest for the
MSVCR9 runtimes? That's a problem which should be addressed by adding a
manifest in the main application (and manifests can also be separate
files called app.exe.manifest if you do not want to embed it).

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue4120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Python bugs RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.