Problem with Multi- threaded Server

[janicehwang1325]

hi,

I post the program before, however, based on the response, i simplify
my server code in the following. I still got error when the clients on
the same LAN trying to connect to the server it gives me segmentation
fault error. What is the cause? For your information, the prefork
server runs perfectly well without any errors. However, due to
specification, i need to design my server handling multiple clients
using threads.


--------------------------------------------------------------------------------------------------------------------------------------------
use threads;

#!/usr/bin/perl

use POSIX ":sys_wait_h";
use IO::Socket::SSL;
use IO::Handle;
use Time::Local;
use DBI;

my ($sock, $s, $v_mode);

unless (@ARGV == 1) { die "usage: perl $0 <Port Number>" };
($port) = @ARGV;

# Check to make sure that we were not accidentally run in the wrong
# directory:
unless (-d "certs") {
if (-d "../certs") {
chdir "..";
} else {
die "No CA Certs\n";
}
}

if(!($sock = IO::Socket::SSL->new( Listen => 20,
LocalPort => $port,
Proto => 'tcp',
ReuseAddr => 1,
SSL_verify_mode => 0x01,
SSL_use_cert => 1,
SSL_passwd_cb => sub {return
"haha"},
SSL_key_file =>
'certs/server_key.pem',
SSL_cert_file =>
'certs/server_certs.pem',
)) ) {
warn "unable to create socket: ", &IO::Socket::SSL::errstr, "\n";
exit(0);

}
warn "SSL socket created: $sock.\n";
warn "waiting for next connection......\n";

sub handle_connection{
$s = shift;
$output = shift || $s;
$exit = 0;

#$s->autoflush(1);

my ($peer_cert, $subject_name, $issuer_name, $date, $str);

if( ! $s ) {
warn "error: ", $s->errstr, "\n";
$exit = 1;
break;
}

chomp($_);

while(<$s>){
if( ref($s) eq "IO::Socket::SSL" && $s->connected()) {
$subject_name =
$s->peer_certificate("subject");
$issuer_name =
$s->peer_certificate("issuer");
print "from client > $_";
if(/equal/i){
#print $s "No new coming log...\n";
last;
}
}
else{
$exit = 1;
}
last if ($exit == 1);
}
print "Thread exited lor..\n";
}

while(1){
while(($s = $sock->accept())) {
#$s->autoflush(1);
print "new thread here .. \n";
#$thread = threads->create(\&handle_connection, $s);
#$thread->detach;
async(\&handle_connection, $s)->detach;
}
}

 

[Ted Zlatanov]

I post the program before, however, based on the response, i simplify
my server code in the following. I still got error when the clients on
the same LAN trying to connect to the server it gives me segmentation
fault error. What is the cause? For your information, the prefork
server runs perfectly well without any errors. However, due to
specification, i need to design my server handling multiple clients
using threads.

Hello,

thanks for posting complete code. Now you know:

- the DBI+SSL server works without threads

Next, find out if:

- the threaded server works without SSL
- the threaded server works without DBI
- the threaded server works without SSL and DBI (trivial case, but needed)

I would say at this point it's very likely you're hitting a bug, so
find out where it is... Then you can try to avoid the bug and you can
tell the developers about it.

Ted

 

[janicehwang1325]

Thanks for the debuggin guidelines. Here are some results for my
testing:

1. Threaded server works fine without SSL.
2. DBI doesn't cause a problem for threaded server (either enable or
disable DBI does not cause the problem with the condition WITHOUT SSL)
3. Threaded server fails when use with SSL.
4. Threaded server works without SSL and DBI.

I tried to debug by printing line after each process. I found out that,
when the second client(same host as the first client) trying to connect
to the server, the server is actually spawning a new thread to handle
it. The error occurs is after the server receiving the data from the
second client. Segmentation fault occurs either after receiving first
few lines of the data and after server spawns a new thread.

Nevertheless, I found out that the server actually detach the threads
after spawning the first thread. I have no idea what is actually going
on. Please advice.

 

[xhoster]

Thanks for the debuggin guidelines. Here are some results for my
testing:

1. Threaded server works fine without SSL.
2. DBI doesn't cause a problem for threaded server (either enable or
disable DBI does not cause the problem with the condition WITHOUT SSL)
3. Threaded server fails when use with SSL

Have you read and understood the parts about close and SSL_no_shutdown in
the IO::Socket::SSL docs?

I can't get SSL to install, so I can't test it, but I suspect this is
likely the problem.


Xho

 

[Ted Zlatanov]

Have you read and understood the parts about close and SSL_no_shutdown in
the IO::Socket::SSL docs?

I can't get SSL to install, so I can't test it, but I suspect this is
likely the problem.

I'm glad we've established the problem is just in the SSL layer.

An additional suggestion: after you looked at the docs, if you still
can't figure the problem out, contact the IO::Socket::SSL
maintainers. Until the problem is resolved, you could turn off SSL so
your server can be used, if that's prudent depending on your
particular security situation.

Ted

 

[janicehwang1325]

Ya, I miss that part and I amend my program by putting the close with
SSL_no_shutdown. However, this only help when the client program ends
the connection and the server program would not hang. It does not
actually solve the segmentation fault problem. I did send my problem to
the maintainers of IO::Socket::SSL. While waiting for their reply, will
keep debuggin. Thank you very much for your information. I appreciates
a lot. Anyhow, any progressions on the program will be shared.

 

[janicehwang1325]

I found another funny error in the code. When I test the socket whether
it is detached or not, by $thread->detach() or die "can't detach"; it
prompts that "A thread exited while 2 threads were running." I test it
with the IO::Socket::INET, same error occurs. I wonder this is the
cause of the problem or just another problem as I can run the program
with IO::Socket::INET perfectly without the checking on detach.

 

[xhoster]

Ya, I miss that part and I amend my program by putting the close with
SSL_no_shutdown. However, this only help when the client program ends
the connection and the server program would not hang. It does not
actually solve the segmentation fault problem.

The SSLeay module seems to be fundamentally unsafe for threads. I tried a
few things to hack it with CLONED, etc. but haven't been able to. The mere
existence of a thread is enough to trigger the problem, the thread doesn't
need to do anything with the socket at all. By changing the "close"
parameters, I could make it segfault either ealier (upon the initial close)
or later (half way through the 2nd accept) but couldn't get rid of it
altogether.


__SERVER__
use threads;
use POSIX ":sys_wait_h";
use IO::Socket::SSL qw(debug4);
use IO::Handle;
use strict;

my ($sock);

unless (@ARGV == 1) { die "usage: perl $0 <Port Number>" };
my ($port) = @ARGV;

if(!($sock = IO::Socket::SSL->new( Listen => 20, LocalPort => $port,
Proto => 'tcp', ReuseAddr => 1,
SSL_key_file => 'certs/newkey.pem',
SSL_cert_file => 'certs/newcert.pem',
)) ) {
warn "unable to create socket: ", &IO::Socket::SSL::errstr, "\n";
exit(0);

}

warn "SSL socket created: $sock.\n";
while(1){

while((my $s = $sock->accept())) {
warn "I accepted $s";
print "new thread here .. \n";
my $thread = threads->create(sub {sleep 5});
$thread->join;
warn "joined";
$s->close(SSL_no_shutdown => 1);
}
}


__CLIENT__
#!/usr/bin/perl

use POSIX ":sys_wait_h";
use IO::Socket::SSL;
use IO::Handle;
use strict;
use warnings;

unless (@ARGV == 1) { die "usage: perl $0 <Port Number>" };
my ($port) = @ARGV;
my $sock = IO::Socket::SSL->new("localhost:$port") or die "$! $@ ",
&IO::Socket::SSL::errstr; use Data:umper;
warn "SSL socket created: $sock.\n";
warn Dumper $sock;

print $sock "asldfjalsdjfsadlfj\n";
__END__

 

[Ted Zlatanov]

Ya, I miss that part and I amend my program by putting the close with
SSL_no_shutdown. However, this only help when the client program ends
the connection and the server program would not hang. It does not
actually solve the segmentation fault problem. I did send my problem to
the maintainers of IO::Socket::SSL. While waiting for their reply, will
keep debuggin. Thank you very much for your information. I appreciates
a lot. Anyhow, any progressions on the program will be shared.

One last thing: you could try the `stunnel' program as your SSL
front-end, and run your server behind it. I don't know if it's
available on Windows, or if something like it is available on Windows,
but it's definitely something you should consider if SSL breaks your
otherwise good threaded server.

Ted

 

[janicehwang1325]

For your information, I am running my program in FreeBSD(should have
mention earlier, sorry bout this) and connect to the FreeBSD using SSH.
Therefore, I don't think I can use "stunnel" program for it. One more
interesting to share with you all, I tried to run the program on
another machine using PERL 5.8.6, it works!!! However, using back on a
machine with PERL 5.8.5, it fails. I wonder it is related to the memory
thing? That's why the program having segmentation fault problem? I will
try to get more machines to test it on and share with you all.

P/S: I didn't get any reply from the maintainers of IO::Socket::SSL.

 

[janicehwang1325]

Maybe you can try run the program usign PERL 5.8.6 version. I am still
testing if the segfault problem have something to do with the version
of the PERL.

 

[janicehwang1325]

Here is my testing result today:

- Running on Perl version 5.8.6 on another machine is also having same
segfault problem. I really don't know why another machine can run while
the other can't. any idea?

 

[Ted Zlatanov]

Here is my testing result today:

- Running on Perl version 5.8.6 on another machine is also having same
segfault problem. I really don't know why another machine can run while
the other can't. any idea?

It's a bug, that's all. Bugs are by definition unpredictable in their
behavior and effects, and finding a way to isolate and repeat a bug is
a valuable skill because of their unpredictability. What you've
described is helpful to the IO::Socket::SSL maintainers, but we (on
the comp.lang.perl.misc) are not generally qualified to fix this
problem. Maybe comp.lang.perl.modules would be useful as a venue, but
I would go to the source on this one.

The bottom line is, don't mix threads and SSL until you've talked to
the maintainers of the IO::Socket::SSL module.

Ted

 

[Ted Zlatanov]

For your information, I am running my program in FreeBSD(should have
mention earlier, sorry bout this) and connect to the FreeBSD using SSH.
Therefore, I don't think I can use "stunnel" program for it.

Are you sure `stunnel' is not available on FreeBSD? I find that very
susprising. I don't have a FreeBSD machine available, but as far as I
know FreeBSD is pretty good about general Unix compatibility, and is
one of the major Unix OSs.

You may have to run stunnel as root, IF the port you want to use is
privileged (usually that means it's less than 1024 in a Unix
environment). But otherwise you don't need to be root, so you can run
stunnel as yourself, say on port 5643, and forward to your server that
runs without SSL on port 5642 (the ports can be any unused port number
above 1024). This can be done like this:

stunnel -d 5643 -r 5642 [OPTIONS]

I won't give you any more information here, because it's off-topic for
a Perl newsgroup. You should visit the stunnel home page and read the
documentation if you have more questions, especially about the other
options stunnel takes.

Ted

 

[janicehwang1325]

Thanks for the information. I will try to find the bug out and trying
to search for the information for stunnel program. Thank you very much.
Any result will be shared later.

For your information, I am running my program in FreeBSD(should have
mention earlier, sorry bout this) and connect to the FreeBSD using SSH.
Therefore, I don't think I can use "stunnel" program for it.

Are you sure `stunnel' is not available on FreeBSD? I find that very
susprising. I don't have a FreeBSD machine available, but as far as I
know FreeBSD is pretty good about general Unix compatibility, and is
one of the major Unix OSs.

You may have to run stunnel as root, IF the port you want to use is
privileged (usually that means it's less than 1024 in a Unix
environment). But otherwise you don't need to be root, so you can run
stunnel as yourself, say on port 5643, and forward to your server that
runs without SSL on port 5642 (the ports can be any unused port number
above 1024). This can be done like this:

stunnel -d 5643 -r 5642 [OPTIONS]

I won't give you any more information here, because it's off-topic for
a Perl newsgroup. You should visit the stunnel home page and read the
documentation if you have more questions, especially about the other
options stunnel takes.

Ted

 

[xhoster]

The SSLeay module seems to be fundamentally unsafe for threads. I tried
a few things to hack it with CLONED, etc. but haven't been able to. The
mere existence of a thread is enough to trigger the problem, the thread
doesn't need to do anything with the socket at all. By changing the
"close" parameters, I could make it segfault either ealier (upon the
initial close) or later (half way through the 2nd accept) but couldn't
get rid of it altogether.

Just an addition, the SSLeay module is not fundamentally unsafe for
threads, it is just the way that IO::Socket::SSL interacts with it that is
unsafe.

If I disable IO::Socket::SSL:ESTROY, the seg faults stop but it leaks
like a sieve. If I re-enable that but disable
IO::Socket::SSL::SSL_Context:ESTROY, then it still solves the seg-fault
and still leaks, but a lot less. So you would need to find a way to make
sure that that DESTROY only happens in one of the threads, either the
parent or the child, and not in both. Something like DBI's
InactiveDestroy.

Or just fork instead of threads. Ask your boss "How much are you willing
to spend to accomplish your stupid and arbitrary requirements?"


Xho

 

[janicehwang1325]

Thanks for the information. This is very useful to me. However, I
couldn't change my boss's requirement as I had inform him of the
problems I am facing. Therefore, I still need to do a lot of research
regarding this. Sigh.

 

[Ted Zlatanov]

Or just fork instead of threads. Ask your boss "How much are you willing
to spend to accomplish your stupid and arbitrary requirements?"

I don't think that's a good question.

Here's a list of places where threads are usually better than fork():

- limited memory
- threads need to collaborate, especially with semaphores and timing
- programmers are familiar with threads
- performance
- lots of separate threads are expected

....plus many others I can't think of immediately. The requirement is
difficult for Perl, but may be very sensible for the particular situation.

Ted

 

[xhoster]

I don't think that's a good question.

I don't think it is universally, but in this case it seems like it.

Here's a list of places where threads are usually better than fork():

- limited memory

How does that work out in threads favor over fork? Perl's threads suck at
memory, and on most systems forks use COW, which makes static structures
quite compact.

- threads need to collaborate, especially with semaphores and timing

Sure, but I don't see any indication that that is the case here.

- programmers are familiar with threads

Then probably they are familiar with a language in which threads are
a strength rather than a weakness. Mabye they should use that language

- performance

I'm unconvinced that Perl's threads have anything special to offer in that
regard. Sure, if you have well-implemented code using threads vs. poorly
implemented code using forks you might see a big difference. But that can
go the other way around, too.

- lots of separate threads are expected

I don't see a benefit of Perl's threads over forks in this respect, either.

Xho

 

[Ted Zlatanov]

On 10 Jul 2006, (e-mail address removed) wrote:

I don't think it is universally, but in this case it seems like it.

OK, let me rephrase. The way you stated it, that's a terrible
question Most people wouldn't ask that from anyone in a business
environment, let alone their boss.

Also I wasn't looking to discuss the particulars of Perl threads and
why they may sometimes be better; my list was just suggestions. The
point was that threads are not *always* a bad idea, and when you don't
know the full story (which I'm sure you don't, unless you work with
the OP) it's more helpful to offer gentle suggestions than to be
forceful.

How does that work out in threads favor over fork? Perl's threads
suck at memory, and on most systems forks use COW, which makes
static structures quite compact.

Do some benchmarks if you like. I'm well-aware that fork() is better
in many ways, but I doubt you'll get better memory performance with
it. If you find otherwise, I'm all ears.

Sure, but I don't see any indication that that is the case here.

I am not talking about the threaded web server the OP wanted. These
are places where threads are *usually* better than fork(), and
my point was that you can't *always* say threads are worse or better
than forking.

I'm unconvinced that Perl's threads have anything special to offer in that
regard. Sure, if you have well-implemented code using threads vs. poorly
implemented code using forks you might see a big difference. But that can
go the other way around, too.

It depends on the task, but I'll say that if you look for performance
(especially where lots of data is shared, RO or RW), you should at
least consider threads. With GUI interaction you usually want a
separate UI thread to get good performance.

I don't see a benefit of Perl's threads over forks in this respect, either.

Do some benchmarks if you like. I'm sure it takes fewer forks than
threads to run out of resources, all other things being equal.

Ted