Mailing List Archive: Perl: porters

eval "require Foo" with binary-incompatible XS modules

Index | Next | Previous | View Threaded

ntyni at debian

May 15, 2008, 11:55 AM

Post #1 of 11 (157 views)
Permalink

eval "require Foo" with binary-incompatible XS modules

[.Resending because the original still hasn't reached the list after six days
or so. Apologies if this ends up as a duplicate.]

Hi p5p,

we're currently doing the 5.8.8 -> 5.10.0 transition in Debian, and the
binary incompatibility in the XS module interface is biting us in an
unexpected way.

In a nutshell, it's possible during the upgrade to temporarily end up
in a state where perl is still 5.8.8, but some XS modules are already
the new ones (ie. built for 5.10.0).

The package scripts try to fail gracefully in situations like this by
doing effectively an 'eval "require Locale::gettext"' or the like.
Unfortunately this doesn't work: the require succeeds but the dynamic
linker kills the script off immediately afterwards because of a
symbol lookup error.

This can be circumvented by setting $ENV{PERL_DL_NONLAZY}=1 before
the eval, so it's arguably a feature. I'm wondering if it would make
sense for Perl to do this automatically inside all eval statements.
After all, the user is using the eval because he wants to catch
exceptions immediately.

If this is obviously too intrusive or otherwise broken, please hit me
with a cluebat. In that case, I think the eval entry in perlfunc.pod
could mention the issue.

I can take a shot at a patch for either case, I'd just like to know
which is the right thing to do.

Details on the Debian issue can be found at <http://bugs.debian.org/479711>,
cc'd as 479711[at]bugs.debian.org .

# perl -e 'local $ENV{PERL_DL_NONLAZY}=0; eval "require Locale::gettext"; print "got: $@" if $@; exit 0'; echo $?
perl: symbol lookup error: /usr/lib/perl5/auto/Locale/gettext/gettext.so: undefined symbol: Perl_Istack_sp_ptr
127

# perl -e 'local $ENV{PERL_DL_NONLAZY}=1; eval "require Locale::gettext"; print "got: $@" if $@; exit 0'; echo $?
got: Can't load '/usr/lib/perl5/auto/Locale/gettext/gettext.so' for module Locale::gettext: /usr/lib/perl5/auto/Locale/gettext/gettext.so: undefined symbol: Perl_Imarkstack_ptr_ptr at /usr/lib/perl/5.8/DynaLoader.pm line 225.
at (eval 1) line 3
Compilation failed in require at (eval 1) line 3.
0

Cheers,
--
Niko Tyni ntyni[at]debian.org

rgarciasuarez at gmail

May 16, 2008, 1:12 AM

Post #2 of 11 (146 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

2008/5/15 Niko Tyni <ntyni[at]debian.org>:
> In a nutshell, it's possible during the upgrade to temporarily end up
> in a state where perl is still 5.8.8, but some XS modules are already
> the new ones (ie. built for 5.10.0).

(That sounds to me as a dependency problem with your packages, but I do know
that dependency solving is a difficult problem.)

> The package scripts try to fail gracefully in situations like this by
> doing effectively an 'eval "require Locale::gettext"' or the like.
> Unfortunately this doesn't work: the require succeeds but the dynamic
> linker kills the script off immediately afterwards because of a
> symbol lookup error.
>
> This can be circumvented by setting $ENV{PERL_DL_NONLAZY}=1 before
> the eval, so it's arguably a feature. I'm wondering if it would make
> sense for Perl to do this automatically inside all eval statements.
> After all, the user is using the eval because he wants to catch
> exceptions immediately.

The objection that comes to mind is that the eval can be several levels
higher than the require. I think we can narrow down the case to
(PL_in_eval & (EVAL_INEVAL|EVAL_INREQUIRE)) == (EVAL_INEVAL|EVAL_INREQUIRE),
to include only require/use statements in evals (if that works -- tests
are needed)

> If this is obviously too intrusive or otherwise broken, please hit me
> with a cluebat. In that case, I think the eval entry in perlfunc.pod
> could mention the issue.
>
> I can take a shot at a patch for either case, I'd just like to know
> which is the right thing to do.

I think the patch should go in ext/DynaLoader/dl_dlopen.xs.

gisle at activestate

May 16, 2008, 4:51 AM

Post #3 of 11 (141 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

On May 16, 2008, at 10:12, Rafael Garcia-Suarez wrote:

> 2008/5/15 Niko Tyni <ntyni[at]debian.org>:
>> In a nutshell, it's possible during the upgrade to temporarily end up
>> in a state where perl is still 5.8.8, but some XS modules are already
>> the new ones (ie. built for 5.10.0).
>
> (That sounds to me as a dependency problem with your packages, but
> I do know
> that dependency solving is a difficult problem.)
>
>> The package scripts try to fail gracefully in situations like this by
>> doing effectively an 'eval "require Locale::gettext"' or the like.
>> Unfortunately this doesn't work: the require succeeds but the dynamic
>> linker kills the script off immediately afterwards because of a
>> symbol lookup error.
>>
>> This can be circumvented by setting $ENV{PERL_DL_NONLAZY}=1 before
>> the eval, so it's arguably a feature. I'm wondering if it would make
>> sense for Perl to do this automatically inside all eval statements.
>> After all, the user is using the eval because he wants to catch
>> exceptions immediately.
>
> The objection that comes to mind is that the eval can be several
> levels
> higher than the require. I think we can narrow down the case to
> (PL_in_eval & (EVAL_INEVAL|EVAL_INREQUIRE)) == (EVAL_INEVAL|
> EVAL_INREQUIRE),
> to include only require/use statements in evals (if that works --
> tests
> are needed)

Another objection is that you might simply want to load the module
and use some function that does resolve. Making require always fail
if some functions doesn't resolve prevent this for working at all. I
think the user better set PERL_DL_NONLAZY themselves if that's what
they want.

Is there a way of making the call to a function that doesn't resolve
simply raise an exception (die in perl) instead of just killing the
process?

--Gisle

pagaltzis at gmx

May 16, 2008, 5:14 AM

Post #4 of 11 (141 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

* Gisle Aas <gisle[at]activestate.com> [2008-05-16 13:55]:
> Is there a way of making the call to a function that doesn't
> resolve simply raise an exception (die in perl) instead of just
> killing the process?

++

If that’s possible, it seems like the right approach.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

doughera at lafayette

May 16, 2008, 5:31 AM

Post #5 of 11 (141 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

On Thu, 15 May 2008, Niko Tyni wrote:

> Hi p5p,
>
> we're currently doing the 5.8.8 -> 5.10.0 transition in Debian, and the
> binary incompatibility in the XS module interface is biting us in an
> unexpected way.
>
> In a nutshell, it's possible during the upgrade to temporarily end up
> in a state where perl is still 5.8.8, but some XS modules are already
> the new ones (ie. built for 5.10.0).

The default perl installation was deliberately designed to avoid just this
problem by including a version-specific directory-path in
$Config{vendorarch}. That way perl-5.8.8 would not try to load
incompatible perl-5.10.0 modules. Debian's build script deliberately does
not include a version-specific $Config{vendorarch}. It might be worth
reviewing that decision.

Changing 5.10.0's $Config{vendorarch} to include the string "5.10" (much
as is done for $Config{archlib} aleady) might solve this problem. Or at
least it might cause the failure to appear earlier and in a way that is
easier to trap gracefully.

> Details on the Debian issue can be found at <http://bugs.debian.org/479711>,

--
Andy Dougherty doughera[at]lafayette.edu

rgarciasuarez at gmail

May 16, 2008, 6:04 AM

Post #6 of 11 (141 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

2008/5/16 Gisle Aas <gisle[at]activestate.com>:
> Another objection is that you might simply want to load the module and use
> some function that does resolve. Making require always fail if some
> functions doesn't resolve prevent this for working at all. I think the user
> better set PERL_DL_NONLAZY themselves if that's what they want.

Good point.

> Is there a way of making the call to a function that doesn't resolve simply
> raise an exception (die in perl) instead of just killing the process?

There might be. At least on glibc systems. I'm not sure about the
lifecycle of an XSUB. How can I write a test case for this ?

nick at ccl4

May 16, 2008, 4:02 PM

Post #7 of 11 (135 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

On Fri, May 16, 2008 at 03:04:21PM +0200, Rafael Garcia-Suarez wrote:
> 2008/5/16 Gisle Aas <gisle[at]activestate.com>:
> > Another objection is that you might simply want to load the module and use
> > some function that does resolve. Making require always fail if some
> > functions doesn't resolve prevent this for working at all. I think the user
> > better set PERL_DL_NONLAZY themselves if that's what they want.
>
> Good point.

Changing this doesn't feel right. I suspect that Gisle's reasoning is part of
my gut feeling on this. Lazy has been the default since 5.002 (1996), and
prior to that was the only option. I never remember this being a problem.

On Fri, May 16, 2008 at 08:31:24AM -0400, Andy Dougherty wrote:
> On Thu, 15 May 2008, Niko Tyni wrote:
>
> > Hi p5p,
> >
> > we're currently doing the 5.8.8 -> 5.10.0 transition in Debian, and the
> > binary incompatibility in the XS module interface is biting us in an
> > unexpected way.
> >
> > In a nutshell, it's possible during the upgrade to temporarily end up
> > in a state where perl is still 5.8.8, but some XS modules are already
> > the new ones (ie. built for 5.10.0).
>
> The default perl installation was deliberately designed to avoid just this
> problem by including a version-specific directory-path in
> $Config{vendorarch}. That way perl-5.8.8 would not try to load
> incompatible perl-5.10.0 modules. Debian's build script deliberately does
> not include a version-specific $Config{vendorarch}. It might be worth
> reviewing that decision.

Perl 5.6.x and Perl 5.8.x are not binary compatible either, yet Debian
upgraded from 5.6.x to 5.8.x without hitting this issue. What changed?
DynaLoader certainly didn't, hence why I'm highly suspicious that changing
it is not the right solution here.

Nicholas Clark

nick at ccl4

May 16, 2008, 4:36 PM

Post #8 of 11 (130 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

On Sat, May 17, 2008 at 12:02:31AM +0100, Nicholas Clark wrote:

> Perl 5.6.x and Perl 5.8.x are not binary compatible either, yet Debian
> upgraded from 5.6.x to 5.8.x without hitting this issue. What changed?
> DynaLoader certainly didn't, hence why I'm highly suspicious that changing
> it is not the right solution here.

I think that there's one "not" too many in there. Oops.

Nicholas Clark

h.m.brand at xs4all

May 17, 2008, 6:30 AM

Post #9 of 11 (121 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

On Sat, 17 May 2008 00:02:31 +0100, Nicholas Clark <nick[at]ccl4.org>
wrote:

> On Fri, May 16, 2008 at 03:04:21PM +0200, Rafael Garcia-Suarez wrote:
> > 2008/5/16 Gisle Aas <gisle[at]activestate.com>:
> > > Another objection is that you might simply want to load the module and use
> > > some function that does resolve. Making require always fail if some
> > > functions doesn't resolve prevent this for working at all. I think the user
> > > better set PERL_DL_NONLAZY themselves if that's what they want.
> >
> > Good point.
>
> Changing this doesn't feel right. I suspect that Gisle's reasoning is part of
> my gut feeling on this. Lazy has been the default since 5.002 (1996), and
> prior to that was the only option. I never remember this being a problem.

The PERL_DL_NOLAZY _thing_ is either good or bad. I have seen many
occasions, esp in DBD testing procedures, where I automatically remove
these PERL_DL_NOLAZY from Makefile, because I know the tests won't pass
anyway when I leave it in. Most likely are the `other' OS's like HP-UX
and AIX where I hit these problems.

If you've hit it once, you'll recognize it when you hit it again :)

--
H.Merijn Brand Amsterdam Perl Mongers (http://amsterdam.pm.org/)
using & porting perl 5.6.2, 5.8.x, 5.10.x on HP-UX 10.20, 11.00, 11.11,
& 11.23, SuSE 10.1 & 10.2, AIX 5.2, and Cygwin. http://qa.perl.org
http://mirrors.develooper.com/hpux/ http://www.test-smoke.org
http://www.goldmark.org/jeff/stupid-disclaimers/

ntyni at debian

May 17, 2008, 11:37 AM

Post #10 of 11 (113 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

On Sat, May 17, 2008 at 12:02:31AM +0100, Nicholas Clark wrote:

> Changing this doesn't feel right. I suspect that Gisle's reasoning is part of
> my gut feeling on this. Lazy has been the default since 5.002 (1996), and
> prior to that was the only option. I never remember this being a problem.

Thanks to everybody for the comments. I get the point about loading a
library with some unresolved symbols and only using those that resolve,
so the change could break working code.

How about just documenting this a bit more with something like the
attached patch? I still think it's unexpected that 'eval "require Foo"'
isn't enough to trap the error.

Some clarifications:

- I'm not familiar with the reasons for choosing
-Dvendorarch=/usr/lib/perl5, but I'll try to find out.

- the circumstances where this shows up in the Debian context are
somewhat a corner case where 'preinst upgrade' scripts that need XS
modules may get run with a new XS module but the old perl. The
package dependencies are not yet guaranteed to be met at the
'preinst' time, hence the 'eval "require Foo"' construct.

- the only time this has actually showed up so far is in the
Locale::gettext case, and that was solved in the same way as in the 5.6
-> 5.8 transition, namely making the Locale::gettext package pre-depend
on the new perl. However, there may be other similar cases that haven't
been discovered yet.

- the fatal error in this case is "undefined symbol: Perl_Istack_sp_ptr" and
happens in DynaLoader::bootstrap(), when calling the "${module}::bootstrap"
function installed by dl_install_xsub(). This is somewhat different to
the usual "call a function that doesn't resolve" case, but having both
raise an exception in Perl instead of finishing off the process would
certainly be good.

- the difference to the 5.6 -> 5.8 transition is that we have lots more
packages and preinst scripts now, but we don't have a good way to
identify which XS modules are needed by preinst scripts. It may well
be that Locale::gettext is the only one (like it apparently was in the
last transition), but we don't really know that yet.

Hm, one option could be to have the packaging system (dpkg) set
PERL_DL_NONLAZY=1 for all preinst script invocations...

Cheers,
--
Niko Tyni ntyni[at]debian.org

Attachments:

lazy-eval.patch (0.74 KB)

rgarciasuarez at gmail

May 17, 2008, 11:35 PM

Post #11 of 11 (111 views)
Permalink

Re: eval "require Foo" with binary-incompatible XS modules [In reply to]

2008/5/17 Niko Tyni <ntyni[at]debian.org>:
> How about just documenting this a bit more with something like the
> attached patch? I still think it's unexpected that 'eval "require Foo"'
> isn't enough to trap the error.

Yes. Thanks, applied as change #33848.

> Some clarifications:
>
> - I'm not familiar with the reasons for choosing
> -Dvendorarch=/usr/lib/perl5, but I'll try to find out.
>
> - the circumstances where this shows up in the Debian context are
> somewhat a corner case where 'preinst upgrade' scripts that need XS
> modules may get run with a new XS module but the old perl. The
> package dependencies are not yet guaranteed to be met at the
> 'preinst' time, hence the 'eval "require Foo"' construct.

I've heard that dpkg2 was going to support pre-inst and post-uninst
dependencies. (Just like rpm does, which creates approximately the same
number of problems than it solves)

> Hm, one option could be to have the packaging system (dpkg) set
> PERL_DL_NONLAZY=1 for all preinst script invocations...

I'm not 100% sure that's a good idea.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com