Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ModPerl: ModPerl

[mp2] mod_perl 2.0.7 Segfault When Loading Certain Perl Module

 

 

ModPerl modperl RSS feed   Index | Next | Previous | View Threaded


jasonlin at ucar

Aug 13, 2012, 3:01 PM

Post #1 of 6 (969 views)
Permalink
[mp2] mod_perl 2.0.7 Segfault When Loading Certain Perl Module

Hi,

I work for a project that has been using mod_perl for quite a while.
Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl
2.0.7 (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since
then, we've been unable to start Apache server (segmentation faults)
with same httpd.conf which worked previously.

As far as I can tell, the segfault seems to be triggered while trying to
load any Perl module using either PerlModule directive or in <Perl>
block in httpd.conf that directly or indirectly use Carp, or even just
try to load Carp itself. Strace shows that the process would segfault
while loading (or right after loading) Carp.pm. The server can still
successfully start and work when carefully removing such modules in
httpd.conf.

Is anyone seeing this same issue or have any ideas how it can be fixed
other than trying to hack through all the Perl modules to make sure Carp
is not called?


perl -V
Summary of my perl5 (revision 5 version 16 subversion 0) configuration:

Platform:
osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp
tue jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
config_args='-Dprefix=/ops/tools -des
-Accflags=-DPERL_RELOCATABLE_INCPUSH'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing
-pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
optimize='-O2',
cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
/lib64 /usr/lib64 /usr/local/lib64
libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.12'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
-fstack-protector'


Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
Built under linux
Compiled at Aug 1 2012 16:14:39
%ENV:
PERL5LIB="/ops/tools/lib"
@INC:
/ops/tools/lib
/ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
/ops/tools/lib/perl5/site_perl/5.16.0
/ops/tools/lib/perl5/5.16.0/x86_64-linux
/ops/tools/lib/perl5/5.16.0
.

Note that we compiled Perl with relocation flag
(-DPERL_RELOCATABLE_INCPUSH) turned on, yet same issue can also be found
without the flag in our test.

Any help/advice would be appreciated, thank you in advance.


Best,
Jason


fred at redhotpenguin

Aug 14, 2012, 11:17 AM

Post #2 of 6 (929 views)
Permalink
Re: [mp2] mod_perl 2.0.7 Segfault When Loading Certain Perl Module [In reply to]

Can you try the latest version from source?

http://perl.apache.org/download/source.html

It looks like you are using perl 5.16, some recent fixes were
committed there. The subversion tip will likely be the 2.0.8 RC in a
week or so.

On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin <jasonlin [at] ucar> wrote:
> Hi,
>
> I work for a project that has been using mod_perl for quite a while.
> Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl 2.0.7
> (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then, we've
> been unable to start Apache server (segmentation faults) with same
> httpd.conf which worked previously.
>
> As far as I can tell, the segfault seems to be triggered while trying to
> load any Perl module using either PerlModule directive or in <Perl> block in
> httpd.conf that directly or indirectly use Carp, or even just try to load
> Carp itself. Strace shows that the process would segfault while loading (or
> right after loading) Carp.pm. The server can still successfully start and
> work when carefully removing such modules in httpd.conf.
>
> Is anyone seeing this same issue or have any ideas how it can be fixed other
> than trying to hack through all the Perl modules to make sure Carp is not
> called?
>
>
> perl -V
> Summary of my perl5 (revision 5 version 16 subversion 0) configuration:
>
> Platform:
> osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
> uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
> jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
> config_args='-Dprefix=/ops/tools -des
> -Accflags=-DPERL_RELOCATABLE_INCPUSH'
> hint=recommended, useposix=true, d_sigaction=define
> useithreads=undef, usemultiplicity=undef
> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
> use64bitint=define, use64bitall=define, uselongdouble=undef
> usemymalloc=n, bincompat5005=undef
> Compiler:
> cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
> -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
> -D_FILE_OFFSET_BITS=64',
> optimize='-O2',
> cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
> -fstack-protector -I/usr/local/include'
> ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
> gccosandvers=''
> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
> lseeksize=8
> alignbytes=8, prototype=define
> Linker and Libraries:
> ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
> libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
> /lib64 /usr/lib64 /usr/local/lib64
> libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
> perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
> libc=, so=so, useshrplib=false, libperl=libperl.a
> gnulibc_version='2.12'
> Dynamic Linking:
> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
> cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
> -fstack-protector'
>
>
> Characteristics of this binary (from libperl):
> Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
> PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
> PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
> USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
> USE_LOCALE_COLLATE USE_LOCALE_CTYPE
> USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
> Built under linux
> Compiled at Aug 1 2012 16:14:39
> %ENV:
> PERL5LIB="/ops/tools/lib"
> @INC:
> /ops/tools/lib
> /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
> /ops/tools/lib/perl5/site_perl/5.16.0
> /ops/tools/lib/perl5/5.16.0/x86_64-linux
> /ops/tools/lib/perl5/5.16.0
> .
>
> Note that we compiled Perl with relocation flag (-DPERL_RELOCATABLE_INCPUSH)
> turned on, yet same issue can also be found without the flag in our test.
>
> Any help/advice would be appreciated, thank you in advance.
>
>
> Best,
> Jason


jasonlin at ucar

Aug 14, 2012, 2:57 PM

Post #3 of 6 (926 views)
Permalink
Re: [mp2] mod_perl 2.0.7 Segfault When Loading Certain Perl Module [In reply to]

Hello Fred,

Thank you for pointing out the upcoming release. Unfortunately, after
checking out and installing the latest version from SVN, I'm still
seeing the same issue. Apache still segfault while trying to load any
Perl module that directly or indirectly use Carp. According to "SVN info
-r HEAD", the reversion I'm using is 1373130.

One thing I noticed from strace is that the module seems to be loaded
successfully the first time ( according to mod_perl2 doc, Apache 2 will
perform a restart right after initial start. so every module will be
loaded twice... ), and only when it is to be loaded the second time the
segfault occurs. Perhaps there are some differences between the first
and second module load during apache/mod_perl initialization that's
causing the segfault?


Best,
Jason

On 08/14/2012 12:17 PM, Fred Moyer wrote:
> Can you try the latest version from source?
>
> http://perl.apache.org/download/source.html
>
> It looks like you are using perl 5.16, some recent fixes were
> committed there. The subversion tip will likely be the 2.0.8 RC in a
> week or so.
>
> On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin <jasonlin [at] ucar> wrote:
>> Hi,
>>
>> I work for a project that has been using mod_perl for quite a while.
>> Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl 2.0.7
>> (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then, we've
>> been unable to start Apache server (segmentation faults) with same
>> httpd.conf which worked previously.
>>
>> As far as I can tell, the segfault seems to be triggered while trying to
>> load any Perl module using either PerlModule directive or in <Perl> block in
>> httpd.conf that directly or indirectly use Carp, or even just try to load
>> Carp itself. Strace shows that the process would segfault while loading (or
>> right after loading) Carp.pm. The server can still successfully start and
>> work when carefully removing such modules in httpd.conf.
>>
>> Is anyone seeing this same issue or have any ideas how it can be fixed other
>> than trying to hack through all the Perl modules to make sure Carp is not
>> called?
>>
>>
>> perl -V
>> Summary of my perl5 (revision 5 version 16 subversion 0) configuration:
>>
>> Platform:
>> osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
>> uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
>> jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
>> config_args='-Dprefix=/ops/tools -des
>> -Accflags=-DPERL_RELOCATABLE_INCPUSH'
>> hint=recommended, useposix=true, d_sigaction=define
>> useithreads=undef, usemultiplicity=undef
>> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
>> use64bitint=define, use64bitall=define, uselongdouble=undef
>> usemymalloc=n, bincompat5005=undef
>> Compiler:
>> cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
>> -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
>> -D_FILE_OFFSET_BITS=64',
>> optimize='-O2',
>> cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
>> -fstack-protector -I/usr/local/include'
>> ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
>> gccosandvers=''
>> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
>> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
>> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
>> lseeksize=8
>> alignbytes=8, prototype=define
>> Linker and Libraries:
>> ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
>> libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
>> /lib64 /usr/lib64 /usr/local/lib64
>> libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
>> perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
>> libc=, so=so, useshrplib=false, libperl=libperl.a
>> gnulibc_version='2.12'
>> Dynamic Linking:
>> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
>> cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
>> -fstack-protector'
>>
>>
>> Characteristics of this binary (from libperl):
>> Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
>> PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
>> PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
>> USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
>> USE_LOCALE_COLLATE USE_LOCALE_CTYPE
>> USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
>> Built under linux
>> Compiled at Aug 1 2012 16:14:39
>> %ENV:
>> PERL5LIB="/ops/tools/lib"
>> @INC:
>> /ops/tools/lib
>> /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
>> /ops/tools/lib/perl5/site_perl/5.16.0
>> /ops/tools/lib/perl5/5.16.0/x86_64-linux
>> /ops/tools/lib/perl5/5.16.0
>> .
>>
>> Note that we compiled Perl with relocation flag (-DPERL_RELOCATABLE_INCPUSH)
>> turned on, yet same issue can also be found without the flag in our test.
>>
>> Any help/advice would be appreciated, thank you in advance.
>>
>>
>> Best,
>> Jason


dhunt at ucar

Aug 20, 2012, 10:00 AM

Post #4 of 6 (886 views)
Permalink
Re: [mp2] mod_perl 2.0.7 Segfault When Loading Certain Perl Module [In reply to]

Hi Fred, modperl folks:

We've narrowed down this bug a bit more. We are currently running perl
5.16.1 (latest), the latest mod_perl from source and apache version
2.2.22.

If we run a simple httpd.conf with this line:

PerlModule Carp

Then single-process startup will fail on the second perl startup.
(perl starts up twice when apache is started, as explained in
http://perl.apache.org/docs/2.0/user/handlers/server.html).

httpd -X
(segmentation fault)

We can tell that apache starts up perl successfully the first time by
running strace. The segfault can be traced to one line in Carp.pm:

--------------------%<----------------------------------------------------
# Transform an argument to a function into a string.
sub format_arg {

...

# The following handling of "control chars" is direct from
# the original code - it is broken on Unicode though.
# Suggestions?
is_utf8($arg) or $arg =~ s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;
return $arg;
}
--------------------%<-------------------------------------------------------

If we comment out the line which starts 'is_utf8...' then the segfault is
prevented. It strikes me as odd that just *compiling* that line (which is
legal perl) should cause a segfault, but that is what we repeatedly
observe.

Any ideas or thoughts welcome!

Regards,

Doug Hunt

dhunt [at] ucar
Software Engineer
UCAR - COSMIC, Tel. (303) 497-2611

On Tue, 14 Aug 2012, Fred Moyer wrote:

> Can you try the latest version from source?
>
> http://perl.apache.org/download/source.html
>
> It looks like you are using perl 5.16, some recent fixes were
> committed there. The subversion tip will likely be the 2.0.8 RC in a
> week or so.
>
> On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin <jasonlin [at] ucar> wrote:
>> Hi,
>>
>> I work for a project that has been using mod_perl for quite a while.
>> Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl 2.0.7
>> (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then, we've
>> been unable to start Apache server (segmentation faults) with same
>> httpd.conf which worked previously.
>>
>> As far as I can tell, the segfault seems to be triggered while trying to
>> load any Perl module using either PerlModule directive or in <Perl> block in
>> httpd.conf that directly or indirectly use Carp, or even just try to load
>> Carp itself. Strace shows that the process would segfault while loading (or
>> right after loading) Carp.pm. The server can still successfully start and
>> work when carefully removing such modules in httpd.conf.
>>
>> Is anyone seeing this same issue or have any ideas how it can be fixed other
>> than trying to hack through all the Perl modules to make sure Carp is not
>> called?
>>
>>
>> perl -V
>> Summary of my perl5 (revision 5 version 16 subversion 0) configuration:
>>
>> Platform:
>> osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
>> uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
>> jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
>> config_args='-Dprefix=/ops/tools -des
>> -Accflags=-DPERL_RELOCATABLE_INCPUSH'
>> hint=recommended, useposix=true, d_sigaction=define
>> useithreads=undef, usemultiplicity=undef
>> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
>> use64bitint=define, use64bitall=define, uselongdouble=undef
>> usemymalloc=n, bincompat5005=undef
>> Compiler:
>> cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
>> -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
>> -D_FILE_OFFSET_BITS=64',
>> optimize='-O2',
>> cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
>> -fstack-protector -I/usr/local/include'
>> ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
>> gccosandvers=''
>> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
>> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
>> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
>> lseeksize=8
>> alignbytes=8, prototype=define
>> Linker and Libraries:
>> ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
>> libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
>> /lib64 /usr/lib64 /usr/local/lib64
>> libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
>> perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
>> libc=, so=so, useshrplib=false, libperl=libperl.a
>> gnulibc_version='2.12'
>> Dynamic Linking:
>> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
>> cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
>> -fstack-protector'
>>
>>
>> Characteristics of this binary (from libperl):
>> Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
>> PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
>> PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
>> USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
>> USE_LOCALE_COLLATE USE_LOCALE_CTYPE
>> USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
>> Built under linux
>> Compiled at Aug 1 2012 16:14:39
>> %ENV:
>> PERL5LIB="/ops/tools/lib"
>> @INC:
>> /ops/tools/lib
>> /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
>> /ops/tools/lib/perl5/site_perl/5.16.0
>> /ops/tools/lib/perl5/5.16.0/x86_64-linux
>> /ops/tools/lib/perl5/5.16.0
>> .
>>
>> Note that we compiled Perl with relocation flag (-DPERL_RELOCATABLE_INCPUSH)
>> turned on, yet same issue can also be found without the flag in our test.
>>
>> Any help/advice would be appreciated, thank you in advance.
>>
>>
>> Best,
>> Jason
>


dhunt at ucar

Aug 20, 2012, 1:48 PM

Post #5 of 6 (889 views)
Permalink
Re: [mp2] mod_perl 2.0.7 Segfault When Loading Certain Perl Module [In reply to]

Hi once more:

I've installed a mod_perl compiled with MP_DEBUG=1 and obtained a core
file, from which I get this stack trace when loading Carp.pm. It seems
it segfaults when compiling the regular expression from this line
in Carp.pm:

is_utf8($arg) or $arg =~ s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;

Does this help at all?

Regards,

Doug Hunt

-------------------%<--------------------------------------------
(gdb) where
#0 0x0000000000513346 in Perl__invlist_union_maybe_complement_2nd ()
#1 0x000000000051642d in S_regclass ()
#2 0x0000000000518e38 in S_regpiece ()
#3 0x000000000051ab30 in S_regbranch ()
#4 0x000000000051ae2d in S_reg ()
#5 0x0000000000518dae in S_regpiece ()
#6 0x000000000051ab30 in S_regbranch ()
#7 0x000000000051ae2d in S_reg ()
#8 0x000000000051dadc in Perl_re_compile ()
#9 0x00000000004d2d24 in Perl_pmruntime ()
#10 0x0000000000506404 in Perl_yyparse ()
#11 0x00000000005721b0 in S_doeval ()
#12 0x0000000000573ea9 in Perl_pp_require ()
#13 0x0000000000489ee1 in modperl_pp_require ()
#14 0x000000000053fc93 in Perl_runops_standard ()
#15 0x00000000004dd0c3 in Perl_eval_sv ()
#16 0x000000000047c3de in modperl_require_module (pv=0x1b8a498 "Carp",
logfailure=1)
#17 0x0000000000475e6d in modperl_config_apply_PerlModule (s=0x1b68950,
scfg=0x1b6ca00, perl=0x1beeb10, p=0x1b5f138)
#18 0x000000000047322f in modperl_startup (s=0x1b68950, p=0x1b5f138)
#19 0x0000000000472ee2 in modperl_startup (s=0x1b68950, p=0x1b5f138)
#20 0x000000000047415b in modperl_init (base_server=0x1b68950,
p=0x1b5f138)
#21 0x0000000000474484 in modperl_hook_init (pconf=0x1b5f138,
plog=0x1ba5368, ptemp=0x1b69188, s=0x1b68950)
#22 0x0000000000464701 in ap_run_open_logs ()
#23 0x000000000044abed in main ()
----------------------%<-------------------------------------------------



dhunt [at] ucar
Software Engineer
UCAR - COSMIC, Tel. (303) 497-2611

On Mon, 20 Aug 2012, Doug Hunt wrote:

> Hi Fred, modperl folks:
>
> We've narrowed down this bug a bit more. We are currently running perl
> 5.16.1 (latest), the latest mod_perl from source and apache version 2.2.22.
>
> If we run a simple httpd.conf with this line:
>
> PerlModule Carp
>
> Then single-process startup will fail on the second perl startup.
> (perl starts up twice when apache is started, as explained in
> http://perl.apache.org/docs/2.0/user/handlers/server.html).
>
> httpd -X
> (segmentation fault)
>
> We can tell that apache starts up perl successfully the first time by running
> strace. The segfault can be traced to one line in Carp.pm:
>
> --------------------%<----------------------------------------------------
> # Transform an argument to a function into a string.
> sub format_arg {
>
> ...
>
> # The following handling of "control chars" is direct from
> # the original code - it is broken on Unicode though.
> # Suggestions?
> is_utf8($arg) or $arg =~
> s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;
> return $arg;
> }
> --------------------%<-------------------------------------------------------
>
> If we comment out the line which starts 'is_utf8...' then the segfault is
> prevented. It strikes me as odd that just *compiling* that line (which is
> legal perl) should cause a segfault, but that is what we repeatedly observe.
>
> Any ideas or thoughts welcome!
>
> Regards,
>
> Doug Hunt
>
> dhunt [at] ucar
> Software Engineer
> UCAR - COSMIC, Tel. (303) 497-2611
>
> On Tue, 14 Aug 2012, Fred Moyer wrote:
>
>> Can you try the latest version from source?
>>
>> http://perl.apache.org/download/source.html
>>
>> It looks like you are using perl 5.16, some recent fixes were
>> committed there. The subversion tip will likely be the 2.0.8 RC in a
>> week or so.
>>
>> On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin <jasonlin [at] ucar> wrote:
>>> Hi,
>>>
>>> I work for a project that has been using mod_perl for quite a while.
>>> Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl 2.0.7
>>> (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then,
>>> we've
>>> been unable to start Apache server (segmentation faults) with same
>>> httpd.conf which worked previously.
>>>
>>> As far as I can tell, the segfault seems to be triggered while trying to
>>> load any Perl module using either PerlModule directive or in <Perl> block
>>> in
>>> httpd.conf that directly or indirectly use Carp, or even just try to load
>>> Carp itself. Strace shows that the process would segfault while loading
>>> (or
>>> right after loading) Carp.pm. The server can still successfully start and
>>> work when carefully removing such modules in httpd.conf.
>>>
>>> Is anyone seeing this same issue or have any ideas how it can be fixed
>>> other
>>> than trying to hack through all the Perl modules to make sure Carp is not
>>> called?
>>>
>>>
>>> perl -V
>>> Summary of my perl5 (revision 5 version 16 subversion 0) configuration:
>>>
>>> Platform:
>>> osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
>>> uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
>>> jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
>>> config_args='-Dprefix=/ops/tools -des
>>> -Accflags=-DPERL_RELOCATABLE_INCPUSH'
>>> hint=recommended, useposix=true, d_sigaction=define
>>> useithreads=undef, usemultiplicity=undef
>>> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
>>> use64bitint=define, use64bitall=define, uselongdouble=undef
>>> usemymalloc=n, bincompat5005=undef
>>> Compiler:
>>> cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing
>>> -pipe
>>> -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
>>> -D_FILE_OFFSET_BITS=64',
>>> optimize='-O2',
>>> cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
>>> -fstack-protector -I/usr/local/include'
>>> ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
>>> gccosandvers=''
>>> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
>>> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
>>> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
>>> lseeksize=8
>>> alignbytes=8, prototype=define
>>> Linker and Libraries:
>>> ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
>>> libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
>>> /lib64 /usr/lib64 /usr/local/lib64
>>> libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
>>> perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
>>> libc=, so=so, useshrplib=false, libperl=libperl.a
>>> gnulibc_version='2.12'
>>> Dynamic Linking:
>>> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
>>> cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
>>> -fstack-protector'
>>>
>>>
>>> Characteristics of this binary (from libperl):
>>> Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
>>> PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
>>> PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
>>> USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
>>> USE_LOCALE_COLLATE USE_LOCALE_CTYPE
>>> USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
>>> Built under linux
>>> Compiled at Aug 1 2012 16:14:39
>>> %ENV:
>>> PERL5LIB="/ops/tools/lib"
>>> @INC:
>>> /ops/tools/lib
>>> /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
>>> /ops/tools/lib/perl5/site_perl/5.16.0
>>> /ops/tools/lib/perl5/5.16.0/x86_64-linux
>>> /ops/tools/lib/perl5/5.16.0
>>> .
>>>
>>> Note that we compiled Perl with relocation flag
>>> (-DPERL_RELOCATABLE_INCPUSH)
>>> turned on, yet same issue can also be found without the flag in our test.
>>>
>>> Any help/advice would be appreciated, thank you in advance.
>>>
>>>
>>> Best,
>>> Jason
>>
>


dhunt at ucar

Aug 21, 2012, 10:03 AM

Post #6 of 6 (888 views)
Permalink
Re: [mp2] mod_perl 2.0.7 Segfault When Loading Certain Perl Module [In reply to]

Hi again: Tracking this down further. I noticed that when we build this
apache/mod_perl we had turned off the 'make test' step. Attached is a
formal bug report with debugging enabled for both mod_perl and perl. The
test suite does not even start, as the server segfaults before the tests
can be run.

It still seems to be failing near Perl__invlist_union_maybe_complement_2nd
when compiling a regular expression.

Thanks much,

Doug Hunt

dhunt [at] ucar
Software Engineer
UCAR - COSMIC, Tel. (303) 497-2611

On Mon, 20 Aug 2012, Doug Hunt wrote:

> Hi once more:
>
> I've installed a mod_perl compiled with MP_DEBUG=1 and obtained a core file,
> from which I get this stack trace when loading Carp.pm. It seems
> it segfaults when compiling the regular expression from this line in Carp.pm:
>
> is_utf8($arg) or $arg =~
> s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;
>
> Does this help at all?
>
> Regards,
>
> Doug Hunt
>
> -------------------%<--------------------------------------------
> (gdb) where
> #0 0x0000000000513346 in Perl__invlist_union_maybe_complement_2nd ()
> #1 0x000000000051642d in S_regclass ()
> #2 0x0000000000518e38 in S_regpiece ()
> #3 0x000000000051ab30 in S_regbranch ()
> #4 0x000000000051ae2d in S_reg ()
> #5 0x0000000000518dae in S_regpiece ()
> #6 0x000000000051ab30 in S_regbranch ()
> #7 0x000000000051ae2d in S_reg ()
> #8 0x000000000051dadc in Perl_re_compile ()
> #9 0x00000000004d2d24 in Perl_pmruntime ()
> #10 0x0000000000506404 in Perl_yyparse ()
> #11 0x00000000005721b0 in S_doeval ()
> #12 0x0000000000573ea9 in Perl_pp_require ()
> #13 0x0000000000489ee1 in modperl_pp_require ()
> #14 0x000000000053fc93 in Perl_runops_standard ()
> #15 0x00000000004dd0c3 in Perl_eval_sv ()
> #16 0x000000000047c3de in modperl_require_module (pv=0x1b8a498 "Carp",
> logfailure=1)
> #17 0x0000000000475e6d in modperl_config_apply_PerlModule (s=0x1b68950,
> scfg=0x1b6ca00, perl=0x1beeb10, p=0x1b5f138)
> #18 0x000000000047322f in modperl_startup (s=0x1b68950, p=0x1b5f138)
> #19 0x0000000000472ee2 in modperl_startup (s=0x1b68950, p=0x1b5f138)
> #20 0x000000000047415b in modperl_init (base_server=0x1b68950, p=0x1b5f138)
> #21 0x0000000000474484 in modperl_hook_init (pconf=0x1b5f138, plog=0x1ba5368,
> ptemp=0x1b69188, s=0x1b68950)
> #22 0x0000000000464701 in ap_run_open_logs ()
> #23 0x000000000044abed in main ()
> ----------------------%<-------------------------------------------------
>
>
>
> dhunt [at] ucar
> Software Engineer
> UCAR - COSMIC, Tel. (303) 497-2611
>
> On Mon, 20 Aug 2012, Doug Hunt wrote:
>
>> Hi Fred, modperl folks:
>>
>> We've narrowed down this bug a bit more. We are currently running perl
>> 5.16.1 (latest), the latest mod_perl from source and apache version 2.2.22.
>>
>> If we run a simple httpd.conf with this line:
>>
>> PerlModule Carp
>>
>> Then single-process startup will fail on the second perl startup.
>> (perl starts up twice when apache is started, as explained in
>> http://perl.apache.org/docs/2.0/user/handlers/server.html).
>>
>> httpd -X
>> (segmentation fault)
>>
>> We can tell that apache starts up perl successfully the first time by
>> running strace. The segfault can be traced to one line in Carp.pm:
>>
>> --------------------%<----------------------------------------------------
>> # Transform an argument to a function into a string.
>> sub format_arg {
>>
>> ...
>>
>> # The following handling of "control chars" is direct from
>> # the original code - it is broken on Unicode though.
>> # Suggestions?
>> is_utf8($arg) or $arg =~
>> s/([[:cntrl:]]|[[:^ascii:]])/sprintf("\\x{%x}",ord($1))/eg;
>> return $arg;
>> }
>>
>> --------------------%<-------------------------------------------------------
>>
>> If we comment out the line which starts 'is_utf8...' then the segfault is
>> prevented. It strikes me as odd that just *compiling* that line (which is
>> legal perl) should cause a segfault, but that is what we repeatedly
>> observe.
>>
>> Any ideas or thoughts welcome!
>>
>> Regards,
>>
>> Doug Hunt
>>
>> dhunt [at] ucar
>> Software Engineer
>> UCAR - COSMIC, Tel. (303) 497-2611
>>
>> On Tue, 14 Aug 2012, Fred Moyer wrote:
>>
>>> Can you try the latest version from source?
>>>
>>> http://perl.apache.org/download/source.html
>>>
>>> It looks like you are using perl 5.16, some recent fixes were
>>> committed there. The subversion tip will likely be the 2.0.8 RC in a
>>> week or so.
>>>
>>> On Mon, Aug 13, 2012 at 3:01 PM, Jason Lin <jasonlin [at] ucar> wrote:
>>>> Hi,
>>>>
>>>> I work for a project that has been using mod_perl for quite a while.
>>>> Recently, we upgraded our 64-bit web server with Perl 5.16, mod_perl
>>>> 2.0.7
>>>> (it's 2.0.6 with patch), and apache 2.2.22 on CentOS 6.3. Since then,
>>>> we've
>>>> been unable to start Apache server (segmentation faults) with same
>>>> httpd.conf which worked previously.
>>>>
>>>> As far as I can tell, the segfault seems to be triggered while trying to
>>>> load any Perl module using either PerlModule directive or in <Perl> block
>>>> in
>>>> httpd.conf that directly or indirectly use Carp, or even just try to load
>>>> Carp itself. Strace shows that the process would segfault while loading
>>>> (or
>>>> right after loading) Carp.pm. The server can still successfully start and
>>>> work when carefully removing such modules in httpd.conf.
>>>>
>>>> Is anyone seeing this same issue or have any ideas how it can be fixed
>>>> other
>>>> than trying to hack through all the Perl modules to make sure Carp is not
>>>> called?
>>>>
>>>>
>>>> perl -V
>>>> Summary of my perl5 (revision 5 version 16 subversion 0) configuration:
>>>>
>>>> Platform:
>>>> osname=linux, osvers=2.6.32-279.1.1.el6.x86_64, archname=x86_64-linux
>>>> uname='linux fox.cosmic.ucar.edu 2.6.32-279.1.1.el6.x86_64 #1 smp tue
>>>> jul 10 13:47:21 utc 2012 x86_64 x86_64 x86_64 gnulinux '
>>>> config_args='-Dprefix=/ops/tools -des
>>>> -Accflags=-DPERL_RELOCATABLE_INCPUSH'
>>>> hint=recommended, useposix=true, d_sigaction=define
>>>> useithreads=undef, usemultiplicity=undef
>>>> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
>>>> use64bitint=define, use64bitall=define, uselongdouble=undef
>>>> usemymalloc=n, bincompat5005=undef
>>>> Compiler:
>>>> cc='cc', ccflags ='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing
>>>> -pipe
>>>> -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
>>>> -D_FILE_OFFSET_BITS=64',
>>>> optimize='-O2',
>>>> cppflags='-DPERL_RELOCATABLE_INCPUSH -fno-strict-aliasing -pipe
>>>> -fstack-protector -I/usr/local/include'
>>>> ccversion='', gccversion='4.4.6 20120305 (Red Hat 4.4.6-4)',
>>>> gccosandvers=''
>>>> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
>>>> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
>>>> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
>>>> lseeksize=8
>>>> alignbytes=8, prototype=define
>>>> Linker and Libraries:
>>>> ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
>>>> libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/lib
>>>> /lib64 /usr/lib64 /usr/local/lib64
>>>> libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
>>>> perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
>>>> libc=, so=so, useshrplib=false, libperl=libperl.a
>>>> gnulibc_version='2.12'
>>>> Dynamic Linking:
>>>> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
>>>> cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
>>>> -fstack-protector'
>>>>
>>>>
>>>> Characteristics of this binary (from libperl):
>>>> Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
>>>> PERL_MALLOC_WRAP PERL_PRESERVE_IVUV
>>>> PERL_RELOCATABLE_INCPUSH USE_64_BIT_ALL
>>>> USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
>>>> USE_LOCALE_COLLATE USE_LOCALE_CTYPE
>>>> USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
>>>> Built under linux
>>>> Compiled at Aug 1 2012 16:14:39
>>>> %ENV:
>>>> PERL5LIB="/ops/tools/lib"
>>>> @INC:
>>>> /ops/tools/lib
>>>> /ops/tools/lib/perl5/site_perl/5.16.0/x86_64-linux
>>>> /ops/tools/lib/perl5/site_perl/5.16.0
>>>> /ops/tools/lib/perl5/5.16.0/x86_64-linux
>>>> /ops/tools/lib/perl5/5.16.0
>>>> .
>>>>
>>>> Note that we compiled Perl with relocation flag
>>>> (-DPERL_RELOCATABLE_INCPUSH)
>>>> turned on, yet same issue can also be found without the flag in our test.
>>>>
>>>> Any help/advice would be appreciated, thank you in advance.
>>>>
>>>>
>>>> Best,
>>>> Jason
>>>
>>
>
Attachments: bug_report.2012.08.21 (14.9 KB)

ModPerl modperl RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.