make test segfaults with "--enable-shared" on Python 2.3.3

[Berthold Hoellmann]

Hello,

When I use

./configure --with-thread --with-fpectl --with-signal-module \
--with-pymalloc --enable-shared --with-cxx=g++

make test

on 2.3.3 I get

....
test_queue
test_quopri
test_random
test_re
make: *** [test] Speicherzugriffsfehler (Speicherauszug erstellt)

gdb ./python core.29964
GNU gdb 5.3.93
Copyright 2003 Free Software Foundation, Inc.
....
Reading symbols from /home/devel/compile/Python-2.3.3/build/lib.linux-i686-2.3/pyexpat.so...done.
Loaded symbols for /home/devel/compile/Python-2.3.3/build/lib.linux-i686-2.3/pyexpat.so
#0 0x400f126c in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6840)
at Modules/_sre.c:750
750 {
(gdb) bt
#0 0x400f126c in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6840)
at Modules/_sre.c:750
#1 0x400f2084 in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6839)
at Modules/_sre.c:1257
#2 0x400f20ef in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6838)
at Modules/_sre.c:1271
#3 0x400f20ef in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6837)
at Modules/_sre.c:1271
#4 0x400f20ef in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6836)
at Modules/_sre.c:1271
#5 0x400f20ef in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6835)
at Modules/_sre.c:1271
#6 0x400f20ef in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6834)
at Modules/_sre.c:1271
#7 0x400f20ef in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6833)
at Modules/_sre.c:1271
#8 0x400f20ef in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=6832)
at Modules/_sre.c:1271
....
#6838 0x400f20ef in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=2)
at Modules/_sre.c:1271
#6839 0x400f157c in sre_match (state=0xbfffcaf0, pattern=0x41126a0c, level=1)
at Modules/_sre.c:1161
---Type <return> to continue, or q <return> to quit---
#6840 0x400f8a36 in sre_search (state=0xbfffcaf0, pattern=0x411269ee)
at Modules/_sre.c:1411
#6841 0x400f63bf in pattern_search (self=0x411269c0, args=0x40e785ac, kw=0x0)
at Modules/_sre.c:1869
#6842 0x40068e74 in PyCFunction_Call (func=0x410610cc, arg=0x40e785ac, kw=0x0)
at Objects/methodobject.c:93

#6843 0x400a29e6 in call_function (pp_stack=0xbfffcf6c, oparg=6840)
at Python/ceval.c:3439
#6844 0x400a4115 in eval_frame (f=0x84f2ddc) at Python/ceval.c:2116
#6845 0x400a181f in PyEval_EvalCodeEx (co=0x40429c60, globals=0x0, locals=0x0,
args=0x84f2ddc, argcount=2, kws=0x84d3468, kwcount=0, defs=0x40436c18,
defcount=1, closure=0x0) at Python/ceval.c:2663
#6846 0x400540d5 in function_call (func=0x4043ba74, arg=0x40f54c6c, kw=0x41134e84)
at Objects/funcobject.c:504

#6847 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40f54c6c, kw=0x41134e84)
at Objects/abstract.c:1755
#6848 0x400a2cad in ext_do_call (func=0x4043ba74, pp_stack=0xbfffd13c,
flags=1089817708, na=0, nk=0) at Python/ceval.c:3713
#6849 0x400a3fbb in eval_frame (f=0x84fa644) at Python/ceval.c:2151
#6850 0x400a181f in PyEval_EvalCodeEx (co=0x40461e20, globals=0x0, locals=0x0,
args=0x84fa644, argcount=5, kws=0x8423550, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:2663

#6851 0x400a76b5 in fast_function (func=0x41126a0c, pp_stack=0xbfffd2ec, n=5, na=5,
nk=20) at Python/ceval.c:3529
#6852 0x400a2aa7 in call_function (pp_stack=0xbfffd2ec, oparg=6840)
at Python/ceval.c:3458
#6853 0x400a4115 in eval_frame (f=0x84233ec) at Python/ceval.c:2116
#6854 0x400a773e in fast_function (func=0x41126a0c, pp_stack=0xbfffd41c, n=1, na=1,
nk=138556396) at Python/ceval.c:3518
#6855 0x400a2aa7 in call_function (pp_stack=0xbfffd41c, oparg=6840)
---Type <return> to continue, or q <return> to quit---
at Python/ceval.c:3458

#6856 0x400a4115 in eval_frame (f=0x84f328c) at Python/ceval.c:2116
#6857 0x400a181f in PyEval_EvalCodeEx (co=0x40461c20, globals=0x0, locals=0x0,
args=0x84f328c, argcount=2, kws=0x0, kwcount=0, defs=0x404a7a58, defcount=1,
closure=0x0) at Python/ceval.c:2663
#6858 0x4005401b in function_call (func=0x404a695c, arg=0x40f5462c, kw=0x0)
at Objects/funcobject.c:504
#6859 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40f5462c, kw=0x0)
at Objects/abstract.c:1755

#6860 0x400448a4 in instancemethod_call (func=0x1ab8, arg=0x40f5462c, kw=0x0)
at Objects/classobject.c:2433
#6861 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40e78fac, kw=0x0)
at Objects/abstract.c:1755

#6862 0x40082bc8 in slot_tp_call (self=0x40e78fac, args=0x40e78fac, kwds=0x0)
at Objects/typeobject.c:4473
#6863 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40e78fac, kw=0x0)
at Objects/abstract.c:1755
#6864 0x400a7806 in do_call (func=0x4113c36c, pp_stack=0x40e78fac, na=-1, nk=6840)
at Python/ceval.c:3644
#6865 0x400a29ac in call_function (pp_stack=0xbfffd8cc, oparg=6840)
at Python/ceval.c:3460

#6866 0x400a4115 in eval_frame (f=0x84237fc) at Python/ceval.c:2116
#6867 0x400a181f in PyEval_EvalCodeEx (co=0x404a41e0, globals=0x0, locals=0x0,
args=0x84237fc, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:2663
#6868 0x4005401b in function_call (func=0x404a6d14, arg=0x40f543ac, kw=0x0)
at Objects/funcobject.c:504
#6869 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40f543ac, kw=0x0)
at Objects/abstract.c:1755
#6870 0x400448a4 in instancemethod_call (func=0x1ab8, arg=0x40f543ac, kw=0x0)
---Type <return> to continue, or q <return> to quit---
at Objects/classobject.c:2433

#6871 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40e78f8c, kw=0x0)
at Objects/abstract.c:1755
#6872 0x40082bc8 in slot_tp_call (self=0x40e78f8c, args=0x40e78f8c, kwds=0x0)
at Objects/typeobject.c:4473
#6873 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40e78f8c, kw=0x0)
at Objects/abstract.c:1755
#6874 0x400a7806 in do_call (func=0x4113ca8c, pp_stack=0x40e78f8c, na=-1, nk=6840)
at Python/ceval.c:3644

#6875 0x400a29ac in call_function (pp_stack=0xbfffdd7c, oparg=6840)
at Python/ceval.c:3460
#6876 0x400a4115 in eval_frame (f=0x8364c9c) at Python/ceval.c:2116
#6877 0x400a181f in PyEval_EvalCodeEx (co=0x404a41e0, globals=0x0, locals=0x0,
args=0x8364c9c, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:2663
#6878 0x4005401b in function_call (func=0x404a6d14, arg=0x40f5460c, kw=0x0)
at Objects/funcobject.c:504
#6879 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40f5460c, kw=0x0)
at Objects/abstract.c:1755
#6880 0x400448a4 in instancemethod_call (func=0x1ab8, arg=0x40f5460c, kw=0x0)
at Objects/classobject.c:2433
#6881 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40e7854c, kw=0x0)
at Objects/abstract.c:1755
#6882 0x40082bc8 in slot_tp_call (self=0x40e7854c, args=0x40e7854c, kwds=0x0)
at Objects/typeobject.c:4473
#6883 0x4003a0aa in PyObject_Call (func=0x41126a0c, arg=0x40e7854c, kw=0x0)
at Objects/abstract.c:1755
#6884 0x400a7806 in do_call (func=0x410e6f8c, pp_stack=0x40e7854c, na=-1, nk=6840)
at Python/ceval.c:3644
#6885 0x400a29ac in call_function (pp_stack=0xbfffe22c, oparg=6840)
---Type <return> to continue, or q <return> to quit---
at Python/ceval.c:3460
#6886 0x400a4115 in eval_frame (f=0x829a7a4) at Python/ceval.c:2116
#6887 0x400a773e in fast_function (func=0x41126a0c, pp_stack=0xbfffe35c, n=2, na=2,
nk=136947620) at Python/ceval.c:3518
#6888 0x400a2aa7 in call_function (pp_stack=0xbfffe35c, oparg=6840)
at Python/ceval.c:3458
#6889 0x400a4115 in eval_frame (f=0x41b3a014) at Python/ceval.c:2116
#6890 0x400a181f in PyEval_EvalCodeEx (co=0x40461420, globals=0x0, locals=0x0,
args=0x41b3a014, argcount=2, kws=0x84376fc, kwcount=0, defs=0x4049e938,
defcount=1, closure=0x0) at Python/ceval.c:2663
#6891 0x400a76b5 in fast_function (func=0x41126a0c, pp_stack=0xbfffe50c, n=2, na=2,
nk=8) at Python/ceval.c:3529
#6892 0x400a2aa7 in call_function (pp_stack=0xbfffe50c, oparg=6840)
at Python/ceval.c:3458
#6893 0x400a4115 in eval_frame (f=0x8437594) at Python/ceval.c:2116
#6894 0x400a181f in PyEval_EvalCodeEx (co=0x40461460, globals=0x0, locals=0x0,
args=0x8437594, argcount=1, kws=0x814b440, kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:2663
#6895 0x400a76b5 in fast_function (func=0x41126a0c, pp_stack=0xbfffe6bc, n=1, na=1,
nk=4) at Python/ceval.c:3529
#6896 0x400a2aa7 in call_function (pp_stack=0xbfffe6bc, oparg=6840)
at Python/ceval.c:3458
#6897 0x400a4115 in eval_frame (f=0x814b2ec) at Python/ceval.c:2116
#6898 0x400a773e in fast_function (func=0x41126a0c, pp_stack=0xbfffe7ec, n=0, na=0,
nk=135574252) at Python/ceval.c:3518
#6899 0x400a2aa7 in call_function (pp_stack=0xbfffe7ec, oparg=6840)
at Python/ceval.c:3458
#6900 0x400a4115 in eval_frame (f=0x831ca34) at Python/ceval.c:2116
#6901 0x400a181f in PyEval_EvalCodeEx (co=0x4044e5a0, globals=0x0, locals=0x0,
args=0x831ca34, argcount=5, kws=0x80d6870, kwcount=0, defs=0x40459938,
---Type <return> to continue, or q <return> to quit---
defcount=1, closure=0x0) at Python/ceval.c:2663
#6902 0x400a76b5 in fast_function (func=0x41126a0c, pp_stack=0xbfffe99c, n=5, na=5,
nk=20) at Python/ceval.c:3529
#6903 0x400a2aa7 in call_function (pp_stack=0xbfffe99c, oparg=6840)
at Python/ceval.c:3458
#6904 0x400a4115 in eval_frame (f=0x80d6654) at Python/ceval.c:2116
#6905 0x400a181f in PyEval_EvalCodeEx (co=0x4044e520, globals=0x0, locals=0x0,
args=0x80d6654, argcount=0, kws=0x80baac4, kwcount=0, defs=0x4045e428,
defcount=11, closure=0x0) at Python/ceval.c:2663
#6906 0x400a76b5 in fast_function (func=0x41126a0c, pp_stack=0xbfffeb4c, n=0, na=0,
nk=0) at Python/ceval.c:3529
#6907 0x400a2aa7 in call_function (pp_stack=0xbfffeb4c, oparg=6840)
at Python/ceval.c:3458
#6908 0x400a4115 in eval_frame (f=0x80ba974) at Python/ceval.c:2116
#6909 0x400a181f in PyEval_EvalCodeEx (co=0x4044e8a0, globals=0x0,
locals=0x403fe79c, args=0x80ba974, argcount=0, kws=0x0, kwcount=0, defs=0x0,
defcount=0, closure=0x0) at Python/ceval.c:2663
#6910 0x400a1152 in PyEval_EvalCode (co=0x4044e8a0, globals=0x403fe79c,
locals=0x403fe79c) at Python/ceval.c:537
#6911 0x400dcc2d in run_err_node (n=0x403fe79c,
filename=0xbfffefad "./Lib/test/regrtest.py", globals=0x403fe79c,
locals=0x403fe79c, flags=0xbfffecf8) at Python/pythonrun.c:1265
#6912 0x400db668 in PyRun_SimpleFileExFlags (fp=0x8049a00,
filename=0xbfffefad "./Lib/test/regrtest.py", closeit=1, flags=0xbfffecf8)
at Python/pythonrun.c:862
#6913 0x400daf9e in PyRun_AnyFileExFlags (fp=0x8049a00,
filename=0xbfffefad "./Lib/test/regrtest.py", closeit=1, flags=0xbfffecf8)
at Python/pythonrun.c:659
#6914 0x400e3994 in Py_Main (argc=2, argv=0xbfffed74) at Modules/main.c:415
#6915 0x080486f3 in main (argc=5, argv=0xbfffed74) at Modules/ccpython.cc:10
Current language: auto; currently c
(gdb)

Some kind of "infinite" recursion?

Regards

Berthold

 

[Berthold Hoellmann]

Hello,

When I use

./configure --with-thread --with-fpectl --with-signal-module \
--with-pymalloc --enable-shared --with-cxx=g++

make test

on 2.3.3 I get

...
test_queue
test_quopri
test_random
test_re
make: *** [test] Speicherzugriffsfehler (Speicherauszug erstellt)

Everything works fine if I remove the "--enable-shared" flag from
configure.

Regards

Berthold

 

[Michael Hudson]

Hello,

When I use

./configure --with-thread --with-fpectl --with-signal-module \
--with-pymalloc --enable-shared --with-cxx=g++

make test

What platform, compiler, and versions thereof?

on 2.3.3 I get

...
test_queue
test_quopri
test_random
test_re
make: *** [test] Speicherzugriffsfehler (Speicherauszug erstellt)

gdb ./python core.29964
GNU gdb 5.3.93
Copyright 2003 Free Software Foundation, Inc.
...
Reading symbols from /home/devel/compile/Python-2.3.3/build/lib.linux-i686-2.3/pyexpat.so...done.
Loaded symbols for /home/devel/compile/Python-2.3.3/build/lib.linux-i686-2.3/pyexpat.so
[snippety]

Some kind of "infinite" recursion?

Well, there's a test in tes_sre that uses quite a lot of C stack. I
guess it's possible that something about shared library code uses more
stack. Can you run 'ulimit -s <something larger>' and try again?

Cheers,
mwh

 

[Berthold Hoellmann]

What platform, compiler, and versions thereof?

Sorry, of course:

SuSE Linux 9.0,
gcc (GCC) 3.3.1 (SuSE Linux)

I always forget about my *FLAGS:

CXXFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
-ffast-math -fno-math-errno -funsafe-math-optimizations \
-fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow

CFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
-ffast-math -fno-math-errno -funsafe-math-optimizations \
-fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow

(My Processor is a AMD Athlon(tm) XP 2000+, 512MB RAM)

...
Well, there's a test in tes_sre that uses quite a lot of C stack. I
guess it's possible that something about shared library code uses more
stack. Can you run 'ulimit -s <something larger>' and try again?

Well, stacksize is unlimited (somehow)

limit

cputime unlimited
filesize unlimited
datasize unlimited
stacksize unlimited
coredumpsize unlimited
memoryuse unlimited
vmemoryuse unlimited
descriptors 1024
memorylocked unlimited
maxproc 4095

Regards
Berthold

 

[Andrew MacIntyre]

When I use

./configure --with-thread --with-fpectl --with-signal-module \
--with-pymalloc --enable-shared --with-cxx=g++

make test

on 2.3.3 I get

...
test_queue
test_quopri
test_random
test_re
make: *** [test] Speicherzugriffsfehler (Speicherauszug erstellt)

Everything works fine if I remove the "--enable-shared" flag from
configure.

You don't mention what platform you are seeing this on. The output of a
verbose test run (python Lib/test/regrtest.py -v test_re), preferably with
error messages translated to English, may help diagnose the issue.

I know that there are platforms where the amount of stack space available
to a threaded process is not easily controlled, and recent versions of gcc
are creating much larger stack frames than earlier versions. The sre
module in 2.3.x (and earlier) is recursive and thus sensitive to stack
space availability. A core dump is a likely indicator of this. Read the
Modules/_sre.c source file for more info. sre in 2.4 will be
significantly improved in this regard.

FYI, since 2.3 PyMalloc is a default option so you don't need
--with-pymalloc, and most recent Linux and BSD systems will default to
building with --with-thread and --with-signal-module.

 

[Andrew MacIntyre]

CXXFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
-ffast-math -fno-math-errno -funsafe-math-optimizations \
-fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow

CFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
-ffast-math -fno-math-errno -funsafe-math-optimizations \
-fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow

Build Python with the configure defaults before you start getting creative
with optimisation.

Problems resulting from trying to use more aggressive optimisations than
the default should be taken up with the compiler vendor.

 

[Berthold Höllmann]

Build Python with the configure defaults before you start getting creative
with optimisation.

Problems resulting from trying to use more aggressive optimisations than
the default should be taken up with the compiler vendor.

Hello,

make distclean
env CFLAGS= CXXFLAGS= ./configure --with-thread --with-fpectl \
--with-signal-module --with-pymalloc --enable-shared --with-cxx=g++
env LANG=C CFLAGS= CXXFLAGS= make test

gave:

test_quopri
test_random
test_re
make: *** [test] Segmentation fault

aggain.

Regards

Berthold

 

[Berthold Höllmann]

When I use

./configure --with-thread --with-fpectl --with-signal-module \
--with-pymalloc --enable-shared --with-cxx=g++

make test

on 2.3.3 I get

...
test_queue
test_quopri
test_random
test_re
make: *** [test] Speicherzugriffsfehler (Speicherauszug erstellt)

make: *** [test] Segmentation fault (core dumped)

You don't mention what platform you are seeing this on. The output of a
verbose test run (python Lib/test/regrtest.py -v test_re), preferably with
error messages translated to English, may help diagnose the issue.

env LANG=C LD_LIBRARY_PATH=/home/devel/compile/Python-2.3.3:/usr/local/v/lib:/usr/lib/qt3/lib:/usr/local/pgsql/lib:/usr/teTeX/lib./python Lib/test/regrtest.py -v test_re

(LD_LIBRARY_PATH is taken from the "make test" output) gives:

test_re
test_anyall (test.test_re.ReTests) ... ok
test_basic_re_sub (test.test_re.ReTests) ... ok
test_bigcharset (test.test_re.ReTests) ... ok
test_bug_113254 (test.test_re.ReTests) ... ok
test_bug_114660 (test.test_re.ReTests) ... ok
test_bug_117612 (test.test_re.ReTests) ... ok
test_bug_418626 (test.test_re.ReTests) ... ok
test_bug_448951 (test.test_re.ReTests) ... ok
test_bug_449000 (test.test_re.ReTests) ... ok
test_bug_449964 (test.test_re.ReTests) ... ok
test_bug_462270 (test.test_re.ReTests) ... ok
test_bug_527371 (test.test_re.ReTests) ... ok
test_bug_545855 (test.test_re.ReTests) ... ok
test_bug_612074 (test.test_re.ReTests) ... ok
test_bug_725106 (test.test_re.ReTests) ... ok
test_bug_725149 (test.test_re.ReTests) ... ok
test_bug_764548 (test.test_re.ReTests) ... ok
test_category (test.test_re.ReTests) ... ok
test_constants (test.test_re.ReTests) ... ok
test_expand (test.test_re.ReTests) ... ok
test_finditer (test.test_re.ReTests) ... ok
test_flags (test.test_re.ReTests) ... ok
test_getattr (test.test_re.ReTests) ... ok
test_getlower (test.test_re.ReTests) ... ok
test_groupdict (test.test_re.ReTests) ... ok
test_ignore_case (test.test_re.ReTests) ... ok
test_non_consuming (test.test_re.ReTests) ... ok
test_not_literal (test.test_re.ReTests) ... ok
test_pickling (test.test_re.ReTests) ... ok
test_qualified_re_split (test.test_re.ReTests) ... ok
test_qualified_re_sub (test.test_re.ReTests) ... ok
test_re_escape (test.test_re.ReTests) ... ok
test_re_findall (test.test_re.ReTests) ... ok
test_re_groupref (test.test_re.ReTests) ... ok
test_re_groupref_exists (test.test_re.ReTests) ... ok
test_re_match (test.test_re.ReTests) ... ok
test_re_split (test.test_re.ReTests) ... ok
test_re_subn (test.test_re.ReTests) ... ok
test_repeat_minmax (test.test_re.ReTests) ... ok
test_scanner (test.test_re.ReTests) ... ok
test_search_coverage (test.test_re.ReTests) ... ok
test_search_star_plus (test.test_re.ReTests) ... ok
test_special_escapes (test.test_re.ReTests) ... ok
test_sre_character_literals (test.test_re.ReTests) ... ok
test_stack_overflow (test.test_re.ReTests) ... ok
test_symbolic_refs (test.test_re.ReTests) ... ok

----------------------------------------------------------------------
Ran 46 tests in 0.401s

OK
Running re_tests test suite
1 test OK.
CAUTION: stdout isn't compared in verbose mode:
a test that passes in verbose mode may fail without it.

so it seems that the other modules loaded in the "make test" run
decrease the avaliable stack so that the test does not succeed but
calling the test alone finds enough stack space avaliable?

I know that there are platforms where the amount of stack space available
to a threaded process is not easily controlled, and recent versions of gcc
are creating much larger stack frames than earlier versions. The sre
module in 2.3.x (and earlier) is recursive and thus sensitive to stack
space availability. A core dump is a likely indicator of this. Read the
Modules/_sre.c source file for more info. sre in 2.4 will be
significantly improved in this regard.

FYI, since 2.3 PyMalloc is a default option so you don't need
--with-pymalloc, and most recent Linux and BSD systems will default to
building with --with-thread and --with-signal-module.

I really should know, but I usually keep "good" configure settings for
reuse and normally don't think much about the settings when everything
works.

Regards

Berthold

 

[Andrew MacIntyre]

make distclean
env CFLAGS= CXXFLAGS= ./configure --with-thread --with-fpectl \
--with-signal-module --with-pymalloc --enable-shared --with-cxx=g++
env LANG=C CFLAGS= CXXFLAGS= make test

gave:

test_quopri
test_random
test_re
make: *** [test] Segmentation fault

1. Just to confirm that the configure generated Makefile is what it
should be, what are the OPT and BASECFLAGS values?

2. Do a verbose run of the test_re test with the interpreter you built
above (ie with the default optimisations):
./python Lib/test/regrtest.py -v test_re >test_re.log 2>&1

3. If it coredumps (which I expect it to based on your advice above), use
gdb to get the backtrace.

The test log and the gdb backtrace should help isolate where things are
going astray.

BTW, I'm not sure why you're explicitly using --with-thread,
--with-signal-module and --with-pymalloc as they are all enabled by
default. I'm also unsure as to whether you really need to specify
--with-cxx; it might be worth a try configure'ing without it.

 

[Andrew MacIntyre]

so it seems that the other modules loaded in the "make test" run
decrease the avaliable stack so that the test does not succeed but
calling the test alone finds enough stack space avaliable?

The above message crossed with my other reply.

This looks very much like the stack problem I referred to, which I am
familiar with on FreeBSD. I have been able to build with
--enable-shared on FreeBSD (after some configure tweaking), and see a
similar effect - ie test_re coredumps with --enable-shared and passes
without (and this with gcc 2.95.4). From the backtrace, I see that a
USE_RECURSION_LIMIT of 7200 would have worked, compared to the 7500 that
works for the standard build.

That it passes the individual test but not the full test suite for you
suggests that your build is right on the stacksize limit, and lowering
USE_RECURSION_LIMIT by only a small amount would get you through.

Note that there are some subtleties here - even though your ulimit says
unlimited stackspace, in the presence of threads this may not hold.
It certainly doesn't on FreeBSD 4.x where there the pthreads support has a
hardcoded 1MB stack for the primary thread (which is what is running the
regression test and is distinct from the stack created for each
instantiated thread). The LinuxThreads port on FreeBSD seems not to have
the same restriction. Exactly what is happening on your Linux system I
don't know; you would have to do some research to find out.

Depending on how you wish to proceed, you have a couple of options:
- build normally, then blow away Modules/_sre.o and recompile
Modules/_sre.c with -Os (just temporarily doctor the Makefile).
From what I've seen -Os cuts the stack consumption enough to get
you through the full test suite, and doesn't seriously impact sre's
performance.
- modify the USE_RECURSION_LIMIT macro in Modules/_sre.c to a lower value
and rebuild. Linux would be using the default definition of 10000 at
line 98 (for 2.3.3).
- find a way to change the default stack size for the primary thread.

If you play with the second or third options, a report to the Python bug
tracker on SF with the conclusion you reach would be useful. Note that
such a bug report should include details of OS & compiler, so that
suitable #ifdef'ery can be written, and should be based on Python's
default compiler switches. As I noted previously, sre in Python 2.4 will
not have this issue, but Anthony Baxter has said that he expects to do a
2.3.4 release (April/May O4 as I recall).

 

[Berthold Höllmann]

....
Note that there are some subtleties here - even though your ulimit says
unlimited stackspace, in the presence of threads this may not hold.
It certainly doesn't on FreeBSD 4.x where there the pthreads support has a
hardcoded 1MB stack for the primary thread (which is what is running the

I found a document stating it is 2MB for each thread on Linux using Google

regression test and is distinct from the stack created for each
instantiated thread). The LinuxThreads port on FreeBSD seems not to have
the same restriction. Exactly what is happening on your Linux system I
don't know; you would have to do some research to find out.

Depending on how you wish to proceed, you have a couple of options:
- build normally, then blow away Modules/_sre.o and recompile
Modules/_sre.c with -Os (just temporarily doctor the Makefile).
From what I've seen -Os cuts the stack consumption enough to get
you through the full test suite, and doesn't seriously impact sre's
performance.

This does work for me

- modify the USE_RECURSION_LIMIT macro in Modules/_sre.c to a lower value
and rebuild. Linux would be using the default definition of 10000 at
line 98 (for 2.3.3).

#define USE_RECURSION_LIMIT 6835

does succeed,

#define USE_RECURSION_LIMIT 6840

gives segfault for me. Will submit bugreport.

- find a way to change the default stack size for the primary thread.

To be honest: I don't have an idea where to look at for this.

Regards

Berthold