P
Peter Gradwell
Hi
We have an SMTPd we have written in perl [1] which receives a large
amount of traffic and will typically have 50 concurrent threads.
To achieve this, we run a perl program which forks very nicely until
after a while, the userland environment just stops responding (no
ssh, sockets etc.) but we think the kernel keeps going (machine
pings).
- machine has to be power cycled.
We've had this problem on FreeBSD 4.8, 4.9 and 4.10-STABLE and
also on FreeBSD 5.2.1 on a range of architectures.
We have also run it with a number of versions of perl, all > 5.6, but
predominantly we are using 5.8.5 now on the default port install. [2].
We believe the problem is some sort of interaction between the way
perl forks and the FreeBSD thread library, however, because the
machines are completely dead, it is somewhat tricky to work out
exactly what the problem is. Nothing is logged by the OS, except
we see that perl crashes quite a lot with a sig11 error.
My question really is has anyone experienced this kind of problem
before
and does anyone have any clues as to what might be causing the crash.
Our current solutions are (a) re-implement using the perl thread
libraries
rather than for or (b) try it on linux. I'd rather find out what
crashes and
fix that though.
thanks
peter
[1] We could use C I suppose, but we all understand perl, and had a
lot of
legacy mail handling logic implemented in perl already.
[2] Output of perl -V
lon-mail-3# perl -V
Summary of my perl5 (revision 5 version 8 subversion 5) configuration:
Platform:
osname=freebsd, osvers=4.10-stable, archname=i386-freebsd-64int
uname='freebsd lon-mail-3.gradwell.net 4.10-stable freebsd
4.10-stable #0: sun sep 19 18:48:57 bst 2004
(e-mail address removed):usrobjusrsrcsyslonmail3190904 i386 '
config_args='-sde -Dprefix=/usr/local
-Darchlib=/usr/local/lib/perl5/5.8.5/mach
-Dprivlib=/usr/local/lib/perl5/5.8.5
-Dman3dir=/usr/local/lib/perl5/5.8.5/perl/man/man3
-Dman1dir=/usr/local/man/man1
-Dsitearch=/usr/local/lib/perl5/site_perl/5.8.5/mach
-Dsitelib=/usr/local/lib/perl5/site_perl/5.8.5
-Dscriptdir=/usr/local/bin
-Dsiteman3dir=/usr/local/lib/perl5/5.8.5/man/man3
-Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv
-Uinstallusrbinperl -Dcc=cc -Doptimize=-O -pipe -Duseshrplib
-Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.5/BSDPAN" -Ud_dosuid
-Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=undef uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags
='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.5/BSDPAN" -DHAS_FPSETMASK
-DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
-I/usr/local/include',
optimize='-O -pipe ',
cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.5/BSDPAN"
-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
-I/usr/local/include'
ccversion='', gccversion='2.95.4 20020320 [FreeBSD]',
gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags ='-Wl,-E -L/usr/local/lib'
libpth=/usr/lib /usr/local/lib
libs=-lm -lcrypt -lutil -lc
perllibs=-lm -lcrypt -lutil -lc
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='
-Wl,-R/usr/local/lib/perl5/5.8.5/mach/CORE'
cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: USE_64_BIT_INT USE_LARGE_FILES
Built under freebsd
Compiled at Sep 19 2004 23:10:48
@INC:
/usr/local/lib/perl5/site_perl/5.8.5/mach
/usr/local/lib/perl5/site_perl/5.8.5
/usr/local/lib/perl5/site_perl/5.8.0
/usr/local/lib/perl5/site_perl/5.005
/usr/local/lib/perl5/site_perl
/usr/local/lib/perl5/5.8.5/BSDPAN
/usr/local/lib/perl5/5.8.5/mach
/usr/local/lib/perl5/5.8.5
We have an SMTPd we have written in perl [1] which receives a large
amount of traffic and will typically have 50 concurrent threads.
To achieve this, we run a perl program which forks very nicely until
after a while, the userland environment just stops responding (no
ssh, sockets etc.) but we think the kernel keeps going (machine
pings).
- machine has to be power cycled.
We've had this problem on FreeBSD 4.8, 4.9 and 4.10-STABLE and
also on FreeBSD 5.2.1 on a range of architectures.
We have also run it with a number of versions of perl, all > 5.6, but
predominantly we are using 5.8.5 now on the default port install. [2].
We believe the problem is some sort of interaction between the way
perl forks and the FreeBSD thread library, however, because the
machines are completely dead, it is somewhat tricky to work out
exactly what the problem is. Nothing is logged by the OS, except
we see that perl crashes quite a lot with a sig11 error.
My question really is has anyone experienced this kind of problem
before
and does anyone have any clues as to what might be causing the crash.
Our current solutions are (a) re-implement using the perl thread
libraries
rather than for or (b) try it on linux. I'd rather find out what
crashes and
fix that though.
thanks
peter
[1] We could use C I suppose, but we all understand perl, and had a
lot of
legacy mail handling logic implemented in perl already.
[2] Output of perl -V
lon-mail-3# perl -V
Summary of my perl5 (revision 5 version 8 subversion 5) configuration:
Platform:
osname=freebsd, osvers=4.10-stable, archname=i386-freebsd-64int
uname='freebsd lon-mail-3.gradwell.net 4.10-stable freebsd
4.10-stable #0: sun sep 19 18:48:57 bst 2004
(e-mail address removed):usrobjusrsrcsyslonmail3190904 i386 '
config_args='-sde -Dprefix=/usr/local
-Darchlib=/usr/local/lib/perl5/5.8.5/mach
-Dprivlib=/usr/local/lib/perl5/5.8.5
-Dman3dir=/usr/local/lib/perl5/5.8.5/perl/man/man3
-Dman1dir=/usr/local/man/man1
-Dsitearch=/usr/local/lib/perl5/site_perl/5.8.5/mach
-Dsitelib=/usr/local/lib/perl5/site_perl/5.8.5
-Dscriptdir=/usr/local/bin
-Dsiteman3dir=/usr/local/lib/perl5/5.8.5/man/man3
-Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv
-Uinstallusrbinperl -Dcc=cc -Doptimize=-O -pipe -Duseshrplib
-Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.5/BSDPAN" -Ud_dosuid
-Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=undef uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags
='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.5/BSDPAN" -DHAS_FPSETMASK
-DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
-I/usr/local/include',
optimize='-O -pipe ',
cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.5/BSDPAN"
-DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe
-I/usr/local/include'
ccversion='', gccversion='2.95.4 20020320 [FreeBSD]',
gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags ='-Wl,-E -L/usr/local/lib'
libpth=/usr/lib /usr/local/lib
libs=-lm -lcrypt -lutil -lc
perllibs=-lm -lcrypt -lutil -lc
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='
-Wl,-R/usr/local/lib/perl5/5.8.5/mach/CORE'
cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: USE_64_BIT_INT USE_LARGE_FILES
Built under freebsd
Compiled at Sep 19 2004 23:10:48
@INC:
/usr/local/lib/perl5/site_perl/5.8.5/mach
/usr/local/lib/perl5/site_perl/5.8.5
/usr/local/lib/perl5/site_perl/5.8.0
/usr/local/lib/perl5/site_perl/5.005
/usr/local/lib/perl5/site_perl
/usr/local/lib/perl5/5.8.5/BSDPAN
/usr/local/lib/perl5/5.8.5/mach
/usr/local/lib/perl5/5.8.5