Test error with Python 2.3.4c1

[Berthold Höllmann]

Hello,

The error desribed in SF error report No. 870120 (segmentation fault
in test_re) still persists for me in 2.3.4c1.

Regards
Berthold

 

[Aahz]

The error desribed in SF error report No. 870120 (segmentation fault
in test_re) still persists for me in 2.3.4c1.

Is there a patch for it? Bugfix releases are mostly low-effort
collections of patches that already exist -- there's not much attempt
made to call for new patches before the release goes out. All that
bugfix release candidates are intended to do is make sure that no new
bugs have been created.

 

[Berthold Höllmann]

Is there a patch for it? Bugfix releases are mostly low-effort
collections of patches that already exist -- there's not much attempt
made to call for new patches before the release goes out. All that
bugfix release candidates are intended to do is make sure that no new
bugs have been created.

The patch working for me is to reduce the default USE_RECURSION_LIMIT

diff Modules/_sre.c~ Modules/_sre.c
98c98
< #define USE_RECURSION_LIMIT 10000
---

#define USE_RECURSION_LIMIT 6835

Regards
Berthold

 

[Aahz]

The patch working for me is to reduce the default USE_RECURSION_LIMIT

diff Modules/_sre.c~ Modules/_sre.c
98c98
< #define USE_RECURSION_LIMIT 10000
---

In that case, you should check against the CVS version of Python; I
believe there've been some changes that affect this kind of thing. If
that fixes your problem, please close the bug report.

 

[Berthold Höllmann]

....>
In that case, you should check against the CVS version of Python; I
believe there've been some changes that affect this kind of thing. If
that fixes your problem, please close the bug report.

OK, I just checked the latest CVS and test_re succeeds. But should't
this fixed for 2.3.4 as well?

Another problem not there in the CVS version is an error in test_sax,
that fails for me on 2.3.4c1 with

echo test_sax > tests
make test TESTOPTS="-l -ftests -v"

case $MAKEFLAGS in \
*-s*) LD_LIBRARY_PATH=/home/devel/compile/Python-2.3.4c1:/usr/local/pgsql/lib:/usr/teTeX/lib CC='gcc -pthread' LDSHARED='gcc -pthread -shared' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python -E ./setup.py -q build;; \
*) LD_LIBRARY_PATH=/home/devel/compile/Python-2.3.4c1:/usr/local/pgsql/lib:/usr/teTeX/lib CC='gcc -pthread' LDSHARED='gcc -pthread -shared' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python -E ./setup.py build;; \
esac
running build
running build_ext
running build_scripts
find ./Lib -name '*.py[co]' -print | xargs rm -f
LD_LIBRARY_PATH=/home/devel/compile/Python-2.3.4c1:/usr/local/pgsql/lib:/usr/teTeX/lib ./python -E -tt ./Lib/test/regrtest.py -l -ftests -v
test_sax
Failed test_attrs_empty
Failed test_attrs_wattr
Failed test_double_quoteattr
Failed test_escape_all
Failed test_escape_basic
Failed test_escape_extra
Failed test_expat_attrs_empty
Failed test_expat_attrs_wattr
Failed test_expat_dtdhandler
Failed test_expat_entityresolver
Failed test_expat_file
Failed test_expat_incomplete
Failed test_expat_incremental
Failed test_expat_incremental_reset
Failed test_expat_inpsource_filename
Failed test_expat_inpsource_location
Failed test_expat_inpsource_stream
Failed test_expat_inpsource_sysid
Failed test_expat_locator_noinfo
Failed test_expat_locator_withinfo
Failed test_expat_nsattrs_empty
Failed test_expat_nsattrs_wattr
Failed test_filter_basic
Failed test_make_parser
Failed test_make_parser2
Failed test_nsattrs_empty
Failed test_nsattrs_wattr
Failed test_quoteattr_basic
test test_sax crashed -- exceptions.TypeError: int argument required
Traceback (most recent call last):
File "./Lib/test/regrtest.py", line 394, in runtest
the_package = __import__(abstest, globals(), locals(), [])
File "/home/devel/compile/Python-2.3.4c1/Lib/test/test_sax.py", line 689, in ?
confirm(value(), name)
File "/home/devel/compile/Python-2.3.4c1/Lib/test/test_sax.py", line 502, in test_sax_parse_exception_str
DummyLocator(None, 1)))
File "/usr/local/lib/python2.3/site-packages/_xmlplus/sax/_exceptions.py", line 94, in __str__
return "%s:%d:%d: %s" % (sysid, self.getLineNumber(),
TypeError: int argument required
1 test failed:
test_sax
make: [test] Error 1 (ignored)
....

self.getLineNumber() is None if I check using pdb, but what
disconcerts me is the

File "/usr/local/lib/python2.3/site-packages/_xmlplus/sax/_exceptions.py", line 94, in __str__

line. Why is sax taken from the installed version instead of the
2.3.4c1 version.

Regards
Berthold

 

[Aahz]

OK, I just checked the latest CVS and test_re succeeds. But should't
this fixed for 2.3.4 as well?

Nope. Bugfix releases (see PEP 6) are required to have essentially zero
behavioral changes, and IIRC the series of fixes that fixed your bug
caused some changes to the way sre operates, so it was a non-starter for
the 2.3.x series.

Anyway, please close your bug with a note about it being fixed in 2.4
CVS. As for the other issue, I don't have enough time/energy to even
understand what the issue is. Given that it's about SAX, I suggest you
start a new thread and hopefully someone else will pick it up.

 

[Richard Townsend]

self.getLineNumber() is None if I check using pdb, but what

disconcerts me is the

File

"/usr/local/lib/python2.3/site-packages/_xmlplus/sax/_exceptions.py", line
94, in __str__

line. Why is sax taken from the installed version instead of the
2.3.4c1 version.

I raised a bug report (864379) back in December or a similar problem with
2.3.3 on HP-UX - except there it caused a core dump.

I too was surprised that it was accessing code in the current installation.

The last response to the bug report suggested an incompatibility with
PyXML-0.8.3, but since then there have been no further responses.

 

[Berthold Höllmann]

Nope. Bugfix releases (see PEP 6) are required to have essentially zero
behavioral changes, and IIRC the series of fixes that fixed your bug
caused some changes to the way sre operates, so it was a non-starter for
the 2.3.x series.

Anyway, please close your bug with a note about it being fixed in 2.4
CVS. As for the other issue, I don't have enough time/energy to even
understand what the issue is. Given that it's about SAX, I suggest you
start a new thread and hopefully someone else will pick it up.

OK, I closed the bug, but still would like to see USE_RECURSION_LIMIT
default reduced to something like 6500 (3000 seems to be OK for
FreeBSD on sparc64, so I guess this can't be a behavioral change).

Regards
Berthold

 

[Andrew MacIntyre]

OK, I closed the bug, but still would like to see USE_RECURSION_LIMIT
default reduced to something like 6500 (3000 seems to be OK for
FreeBSD on sparc64, so I guess this can't be a behavioral change).

Prior to 2.4, sre is recursive. Recent versions of gcc (in particular)
at higher optimisation levels are generating much larger stack frames than
used to be the case. This is even moreso the case with 64-bit platforms.

In the particular case of FreeBSD 4.x, in the presence of threads (the
default build choice, using libc_r) there is a hard coded stack size of
1MB, which leaves this environment especially sensitive to stack frame
size. The alternative threads libraries (libkse/libthr) available on 5.x
are less affected, but I don't know to what extent (& Python's configure
script doesn't know about them that I've heard).

I've seen reference to the problem on Linux, but I don't know why that
platform is (apparently) short on stack.

This is a highly platform/compiler dependant issue, which is why the maze
of #ifdef'ery for USE_RECURSION_LIMIT - something that has only happened
since about Python 2.3.1 as I recall.

The most effective solution with recent gcc versions is to recompile sre.c
with -Os (instead of -O3). It is unfortunately awkward to implement with
autoconf. The non-recursive sre in 2.4 is a complete fix, but is too
extensive/invasive a change for back-porting for a bug-fix release.

 

[Berthold =?iso-8859-15?q?H=F6llmann?=]

Prior to 2.4, sre is recursive. Recent versions of gcc (in particular)
at higher optimisation levels are generating much larger stack frames than
used to be the case. This is even moreso the case with 64-bit platforms.

In the particular case of FreeBSD 4.x, in the presence of threads (the
default build choice, using libc_r) there is a hard coded stack size of
1MB, which leaves this environment especially sensitive to stack frame
size. The alternative threads libraries (libkse/libthr) available on 5.x
are less affected, but I don't know to what extent (& Python's configure
script doesn't know about them that I've heard).

I've seen reference to the problem on Linux, but I don't know why that
platform is (apparently) short on stack.

This is a highly platform/compiler dependant issue, which is why the maze
of #ifdef'ery for USE_RECURSION_LIMIT - something that has only happened
since about Python 2.3.1 as I recall.

The most effective solution with recent gcc versions is to recompile sre.c
with -Os (instead of -O3). It is unfortunately awkward to implement with
autoconf. The non-recursive sre in 2.4 is a complete fix, but is too
extensive/invasive a change for back-porting for a bug-fix release.

But the current value for USE_RECURSION_LIMIT is a problem on (some)
Linux Platforms. Why is reducing the default value a problem, when
setting it to a smaller value on some platforms is not a problem. I
thought everything causing segmentation faults from Python is to be
avoided.

Regards
Berthold