Perl threads

Discussion in 'Perl Misc' started by ivakras1@gmail.com, Jul 30, 2007.

  1. Guest

    Hi all!
    I have a perl threads question. Of cuz i have read perl threads
    tutorial and think its clear to me. But some questions...
    Q1: Is threads->create method only creates thread, but not runs theads
    until im join it (or detach)?
    The folowing code and its output makes me think so:
    =====================
    ....
    for (my $i=0;$i<=10;$i++)
    {
    my $trl=threads->create(\&my_funct,$my_args);
    if (@tc=threads->list){print "\t".@tc." threads running";}
    }
    .....
    if (@tc=threads->list)
    {
    print "\nClosing threads: ";
    while (@tc-1)
    {
    @tc=threads->list;
    print @tc." ";
    threads->list->join;
    }
    }
    else
    {
    exit(0);
    }

    ....
    sub my_funct
    {
    $id=int(rand(1000));
    setlogsock('unix'); #im using syslog to write a log file in /
    var/log/messages
    openlog('PerlSniffer','','debug'); #open a log
    syslog('info', "Thread $id start"); #WRITE THREAD START TIME
    $w=int(rand(5));
    sleep($w); #SLEEP $w SECONDS
    syslog('info', "Thread $id stop after $w s sleep"); #WRITE
    THREAD STOP TIME
    closelog;
    return(1);
    }
    .....
    =======================
    The output of that program:

    0 threads running
    1 threads running
    2 threads running
    3 threads running
    4 threads running
    5 threads running
    6 threads running
    7 threads running
    8 threads running
    9 threads running
    10 threads running
    Closing threads: 10 9 8 7 6 5 4 3 2 1 0

    =======================
    /var/log/messages for that:

    Jul 30 12:20:43 test_comp PerlSniffer: Thread 914 start
    Jul 30 12:20:44 test_comp PerlSniffer: Thread 560 start
    Jul 30 12:20:45 test_comp PerlSniffer: Thread 914 stop after 2 s sleep
    Jul 30 12:20:45 test_comp PerlSniffer: Thread 560 stop after 1 s sleep
    Jul 30 12:20:46 test_comp PerlSniffer: Thread 451 start
    Jul 30 12:20:46 test_comp PerlSniffer: Thread 451 stop after 0 s sleep
    Jul 30 12:20:48 test_comp PerlSniffer: Thread 303 start
    Jul 30 12:20:48 test_comp PerlSniffer: Thread 303 stop after 0 s sleep
    Jul 30 12:20:49 test_comp PerlSniffer: Thread 834 start
    Jul 30 12:20:50 test_comp PerlSniffer: Thread 311 start
    Jul 30 12:20:51 test_comp PerlSniffer: Thread 311 stop after 1 s sleep
    Jul 30 12:20:52 test_comp PerlSniffer: Thread 677 start
    Jul 30 12:20:52 test_comp PerlSniffer: Thread 677 stop after 0 s sleep
    Jul 30 12:20:52 test_comp PerlSniffer: Thread 834 stop after 3 s sleep
    Jul 30 12:20:52 test_comp PerlSniffer: Thread 133 start
    Jul 30 12:20:52 test_comp PerlSniffer: Thread 133 stop after 0 s sleep
    Jul 30 12:20:54 test_comp PerlSniffer: Thread 718 start
    Jul 30 12:20:55 test_comp PerlSniffer: Thread 718 stop after 0 s sleep
    Jul 30 12:20:56 test_comp PerlSniffer: Thread 909 start
    Jul 30 12:20:58 test_comp PerlSniffer: Thread 520 start
    Jul 30 12:21:00 test_comp PerlSniffer: Thread 909 stop after 4 s sleep
    Jul 30 12:21:02 test_comp PerlSniffer: Thread 520 stop after 4 s sleep
    =======================
    Coments: i create 11 threads first, and than join them. Each thread
    waits for 0-5 (rand(5)) seconds, and than exits, writes a start time
    and a stop time in /var/log/messages via syslogd. It looks like thread-
    >create only creates the thread, not runs it. The output shoes it:

    threads->list increments every threads->create, but some of created
    threads would be done during this time, if i wrong!!! When i threads-
    >list->join one by one the threads seems to start, and threads->list

    starts decreasing! Can anybody answer me?

    Q2: threads->list returns a list of unjoind or undetached threads. How
    to return a count of all threads, joined and detached both?? Then the
    main program exits while some child threads are still running
    (detached), perl warns me the count of running parentless threads. How
    can i get that count during main program still alive?!

    Thanks a lot and forgive me my baaad english, im russian actually.
    Praying for the answers! Thanks again!
     
    , Jul 30, 2007
    #1
    1. Advertising

  2. zentara Guest

    On Mon, 30 Jul 2007 08:55:55 -0000, wrote:

    >Hi all!
    >I have a perl threads question. Of cuz i have read perl threads
    >tutorial and think its clear to me. But some questions...
    >Q1: Is threads->create method only creates thread, but not runs theads
    >until im join it (or detach)?


    The thread starts running as soon as you create it.

    There are some simple rules.
    1. A thread, whether joinable or detached, MUST return, or reach the end
    of it's code block, for it to terminate. That means you may need some
    logic with shared variables, to signal a thread to return, if you want
    to avoid the "a thread exited while x running" warning. The warning
    is harmless, unless the threads were doing something that needed a clean
    finish. Joining by itself, WILL NOT terminate a thread, unless it has
    returned first.

    2. Use alot of shared variables to track and control your threads.

    3. You have 2 ways to get return values from a thread. One is through
    shared variables, and the other is through the return array.


    #!/usr/bin/perl
    use threads;

    # join() does three things: it waits for a thread to exit,
    # cleans up after it, and returns any data the thread may
    # have produced.

    $thr1 = threads->new(\&sub1);
    $ReturnData1 = $thr1->join;
    print "Thread1 returned @$ReturnData1\n";

    $thr2 = threads->new(\&sub2);
    $ReturnData2 = $thr2->join;
    print "Thread2 returned @$ReturnData2\n";

    $thr3 = threads->new(\&sub3);
    $ReturnData3 = $thr3->join;
    print "Thread3 returned @$ReturnData3\n";


    sub sub1 {
    print "In thread1.....\n";
    sleep 10;
    @values = ('1a','1b', '1c');
    return \@values;
    }

    sub sub2 {
    print "In thread2.....\n";
    sleep 10;
    @values = ('2a','2b', '2c');
    return \@values;
    }

    sub sub3 {
    print "In thread3.....\n";
    sleep 10;
    @values = ('3a','3b', '3c');
    return \@values;
    }
    __END__

    ############################################
    Some pseudo-code for terminating early:

    #in main thread
    my $die:shared = 0;
    for( 1 .. 10 ){$thr = threads->new(\&thread_code);}
    ....
    ....
    if( $foobar ==42 ){ clean_exit() }

    sub clean_exit{
    $die =1;
    $_->join for threads->list();

    }


    #in thread
    sub thread_code{
    for (1..10){
    if( $die == 1){return}
    print "$_\n";
    sleep 1;
    }

    print "I'm at the end of my code block, and ready to be joined\n";

    }


    zentara


    --
    I'm not really a human, but I play one on earth.
    http://zentara.net/japh.html
     
    zentara, Jul 30, 2007
    #2
    1. Advertising

  3. Guest

    Oh i see now... I can controll thread over shared vars. Ok.
    In your example:
    print "I'm at the end of my code block, and ready to be joined\n";
    Is that mean thread waits to be joined or detached anyway? It never
    dies by itself?

    I need to fire up threads and forget about them, and continuing main
    program, so i use detach. How can i count how much threads are running
    now?
    Tnx!
     
    , Jul 30, 2007
    #3
  4. Guest

    wrote:
    > Hi all!
    > I have a perl threads question. Of cuz i have read perl threads
    > tutorial and think its clear to me. But some questions...
    > Q1: Is threads->create method only creates thread, but not runs theads
    > until im join it (or detach)?
    > The folowing code and its output makes me think so:
    > =====================
    > ...



    Using strict is probably even more important under threads than it
    is without threads. You should use strict.



    > for (my $i=0;$i<=10;$i++)
    > {
    > my $trl=threads->create(\&my_funct,$my_args);
    > if (@tc=threads->list){print "\t".@tc." threads running";}
    > }
    > ....
    > if (@tc=threads->list)
    > {
    > print "\nClosing threads: ";
    > while (@tc-1)
    > {
    > @tc=threads->list;
    > print @tc." ";
    > threads->list->join;
    > }
    > }
    > else
    > {
    > exit(0);
    > }
    >
    > ...
    > sub my_funct
    > {
    > $id=int(rand(1000));
    > setlogsock('unix'); #im using syslog to write a log file in /
    > var/log/messages


    Instead of putting in a comment telling us you are using syslog, you
    should show us the code that actually "use"s syslog. (By which
    you presumably mean Sys::Syslog?) Is Sys::Syslog thread safe?


    > Coments: i create 11 threads first, and than join them. Each thread
    > waits for 0-5 (rand(5)) seconds, and than exits, writes a start time
    > and a stop time in /var/log/messages via syslogd. It looks like thread-
    > >create only creates the thread, not runs it. The output shoes it:

    > threads->list increments every threads->create, but some of created
    > threads would be done during this time,


    The criteria for a thread to show up in list is not that it be "running"

    From the doc:
    threads->list();
    This will return a list of all non joined, non
    detached threads.

    So threads that are "finished" but still haven't been joined or detached
    will be in the list.


    if i wrong!!! When i threads-
    > >list->join one by one the threads seems to start, and threads->list

    > starts decreasing! Can anybody answer me?


    threads->list doesn't do what you think it does.


    >
    > Q2: threads->list returns a list of unjoind or undetached threads.


    Er, OK, so you do know this. Then why the confusion earlier?

    > How
    > to return a count of all threads, joined and detached both?? Then the
    > main program exits while some child threads are still running
    > (detached), perl warns me the count of running parentless threads. How
    > can i get that count during main program still alive?!


    The easy way is not to detach the threads. Then, just before the parent
    exits, run through threads->list and join all of them. Or you could use
    Thread::Running

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Jul 30, 2007
    #4
  5. Guest

    On 30 , 21:08, wrote:
    > Instead of putting in a comment telling us you are using syslog, you
    > should show us the code that actually "use"s syslog. (By which
    > you presumably mean Sys::Syslog?) Is Sys::Syslog thread safe?


    use Sys::Syslog qw( :DEFAULT setlogsock); #thats how im using syslog

    > The easy way is not to detach the threads. Then, just before the parent
    > exits, run through threads->list and join all of them. Or you could use
    > Thread::Running


    I working on network sniffer and packet analizer. The main program
    starts to capture network packets with "use Net::pcap". Each 100
    packets captured - main program dumps packets in a file and starts a
    new thread with arguments, contains a file name. Than main thread
    continue sniffing, and the thread starts to analize the file. The idea
    is to make main program work with no pauses for waiting joined thread
    or something else, it causes packet missing. I tried to fire up
    threads with detach (threads creates fast, 10-20 per second). But when
    the main program is done (by ctrl+c or else) - perl warns me the
    number of threads still running. How can i get this number by myself
    to wait for detached threads dies?
     
    , Jul 31, 2007
    #5
  6. zentara Guest

    On Tue, 31 Jul 2007 04:44:12 -0000, wrote:

    >is to make main program work with no pauses for waiting joined thread
    >or something else, it causes packet missing.

    I would suspect that the startup time for the thread, is where the
    packets are being lost. Especially if the system load is heavy, the
    thread takes time to set itself up. You may need to rethink your
    approach to this.

    > I tried to fire up
    >threads with detach (threads creates fast, 10-20 per second). But when
    >the main program is done (by ctrl+c or else) - perl warns me the
    >number of threads still running. How can i get this number by myself
    >to wait for detached threads dies?


    When you detach a thread, the main thread forgets about it, so there is
    no way to get them without keeping track of them yourself.

    Make a shared array or scalar, to keep track of your threads.
    When you create a thread, push it's identifier ( use a hash ) onto
    the array.
    At the end of the thread code block, just before it exits, have it
    remove it's name from the array.

    Then you can tell how many threads you have running, by the
    scalar count of the array.

    If you want to terminate early, remember, you will need a "$die" shared
    variable, to tell the remaining threads to return immediately, and then
    you can cleanly exit. (This means your thread code block needs to be
    written to check for $die often.

    One word of warning, about spawning alot of detached threads containing
    object code from some module......... you may very well will gain memory
    as the script runs, if the cleanup of refcounts is not good. You may get
    lucky, but it is something to watch out for. It is very dis-heartening
    to spend a bunch of time getting it all to work, only to find it sucks
    up ram like a vacuum cleaner. :)

    Personally, if speed is important, to avoid missing packets, I would
    create a bank of reusable joinable threads, that you can cycle thru.
    You may also want to ask this on http://perlmonks.org, where a fellow
    named BrowserUk may be able to show you how to use thread->queue.
    I'm a "crude crafter" and like to hammer all my stuff together with
    shared variables and event loops, but there are some more sophisticated
    techniques out there.

    Here is an example of a reusable thread-bank. I create 3 threads, and
    when they finish, they get pushed back into an array for the next
    available job. It also shows you how to track your threads with a shared
    hash.


    #!/usr/bin/perl
    use warnings;
    use strict;
    use threads;
    use threads::shared;

    #works on WindowsME ActiveState 5.8.6.811

    my $data = shift || 'date'; #sample code to pass to thread

    my %shash;
    #share(%shash); #will work only for first level keys
    my %hash;
    my %workers;
    my $numworkers = 3;

    foreach my $dthread(1..$numworkers){
    share ($shash{$dthread}{'go'});
    share ($shash{$dthread}{'progress'});
    share ($shash{$dthread}{'timekey'}); #actual instance of the thread
    share ($shash{$dthread}{'frame_open'}); #open or close the frame
    share ($shash{$dthread}{'handle'});
    share ($shash{$dthread}{'data'});
    share ($shash{$dthread}{'pid'});
    share ($shash{$dthread}{'die'});

    $shash{$dthread}{'go'} = 0;
    $shash{$dthread}{'progress'} = 0;
    $shash{$dthread}{'timekey'} = 0;
    $shash{$dthread}{'frame_open'} = 0;
    $shash{$dthread}{'handle'} = 0;
    $shash{$dthread}{'data'} = $data;
    $shash{$dthread}{'pid'} = -1;
    $shash{$dthread}{'die'} = 0;
    $hash{$dthread}{'thread'} = threads->new(\&work,$dthread);
    }


    use Tk;
    use Tk::Dialog;

    my $mw = MainWindow->new(-background => 'gray50');

    my $lframe = $mw->Frame( -background => 'gray50',-borderwidth=>10 )
    ->pack(-side =>'left' ,-fill=>'y');
    my $rframe = $mw->Frame( -background => 'gray50',-borderwidth=>10 )
    ->pack(-side =>'right',-fill =>'both' );

    my %actives = (); #hash to hold reusable numbered widgets used for
    downloads
    my @ready = (); #array to hold markers indicating activity is needed

    #make 3 reusable downloader widget sets-------------------------
    foreach(1..$numworkers){
    push @ready, $_;
    #frames to hold indicator
    $actives{$_}{'frame'} = $rframe->Frame( -background => 'gray50' );

    $actives{$_}{'stopbut'} = $actives{$_}{'frame'}->Button(
    -text => "Stop Worker $_",
    -background => 'lightyellow',
    -command => sub { } )->pack( -side => 'left', -padx => 10 );

    $actives{$_}{'label1'} = $actives{$_}{'frame'} ->Label(
    -width => 3,
    -background => 'black',
    -foreground => 'lightgreen',
    -textvariable => \$shash{$_}{'progress'},
    )->pack( -side => 'left' );

    $actives{$_}{'label2'} = $actives{$_}{'frame'} ->Label(
    -width => 1,
    -text => '%',
    -background => 'black',
    -foreground => 'lightgreen',
    )->pack( -side => 'left' );


    $actives{$_}{'label3'} = $actives{$_}{'frame'} ->Label(
    -text => '',
    -background => 'black',
    -foreground => 'skyblue',
    )->pack( -side => 'left',-padx =>10 );

    }
    #--------------------------------------------------

    my $button = $lframe->Button(
    -text => 'Get a worker',
    -background => 'lightgreen',
    -command => sub { &get_a_worker(time) }
    )->pack( -side => 'top', -anchor => 'n', -fill=>'x', -pady =>
    20 );

    my $text = $rframe->Scrolled("Text",
    -scrollbars => 'ose',
    -background => 'black',
    -foreground => 'lightskyblue',
    )->pack(-side =>'top', -anchor =>'n');

    my $repeat;
    my $startbut;
    my $repeaton = 0;
    $startbut = $lframe->Button(
    -text => 'Start Test Count',
    -background => 'hotpink',
    -command => sub {
    my $count = 0;
    $startbut->configure( -state => 'disabled' );
    $repeat = $mw->repeat(
    100,
    sub {
    $count++;
    $text->insert( 'end', "$count\n" );
    $text->see('end');
    }
    );
    $repeaton = 1;
    })->pack( -side => 'top', -fill=>'x', -pady => 20);

    my $stoptbut = $lframe->Button(
    -text => 'Stop Count',
    -command => sub {
    $repeat->cancel;
    $repeaton = 0;
    $startbut->configure( -state => 'normal' );
    })->pack( -side => 'top',-anchor => 'n', -fill=>'x', -pady => 20 );

    my $exitbut = $lframe->Button(
    -text => 'Exit',
    -command => sub {

    foreach my $dthread(keys %hash){
    $shash{$dthread}{'die'} = 1;
    $hash{$dthread}{'thread'}->join
    }

    if ($repeaton) { $repeat->cancel }
    #foreach ( keys %downloads ) {
    # #$downloads{$_}{'repeater'}->cancel;
    #}
    # $mw->destroy;
    exit;
    })->pack( -side => 'top',-anchor => 'n', -fill=>'x', -pady => 20
    );


    #dialog to get file url---------------------
    my $dialog = $mw->Dialog(
    -background => 'lightyellow',
    -title => 'Get File',
    -buttons => [ "OK", "Cancel" ]
    );

    my $hostl = $dialog->add(
    'Label',
    -text => 'Enter File Url',
    -background => 'lightyellow'
    )->pack();

    my $hostd = $dialog->add(
    'Entry',
    -width => 100,
    -textvariable => '',
    -background => 'white'
    )->pack();

    $dialog->bind( '<Any-Enter>' => sub { $hostd->Tk::focus } );

    my $message = $mw->Dialog(
    -background => 'lightyellow',
    -title => 'ERROR',
    -buttons => [ "OK" ]
    );

    my $messagel = $message->add(
    'Label',
    -text => ' ',
    -background => 'hotpink'
    )->pack();

    $mw->repeat(10, sub{
    if(scalar @ready == $numworkers){return}

    foreach my $set(1..$numworkers){
    $actives{$set}{'label1'}->
    configure(-text =>\$shash{$set}{'progress'});

    if(($shash{$set}{'go'} == 0) and
    ($shash{$set}{'frame_open'} == 1))
    {
    my $timekey = $shash{$set}{'timekey'};
    $workers{ $timekey }{'frame'}->packForget;
    $shash{$set}{'frame_open'} = 0;
    push @ready, $workers{$timekey}{'setnum'};
    if((scalar @ready) == 3)
    { }
    $workers{$timekey} = ();
    delete $workers{$timekey};
    }
    }
    });

    $mw->MainLoop;
    ###################################################################

    sub get_a_worker {

    my $timekey = shift;

    $hostd->configure( -textvariable => \$data);
    if ( $dialog->Show() eq 'Cancel' ) { return }

    #----------------------------------------------
    #get an available frameset
    my $setnum;
    if($setnum = shift @ready){print "setnum->$setnum\n"}
    else{ print "no setnum available\n"; return}

    $workers{$timekey}{'setnum'} = $setnum;
    $shash{$setnum}{'timekey'} = $timekey;

    $workers{$timekey}{'frame'} = $actives{$setnum}{'frame'};
    $workers{$timekey}{'frame'}->pack(-side =>'bottom', -fill => 'both' );

    $workers{$timekey}{'stopbut'} = $actives{$setnum}{'stopbut'};
    $workers{$timekey}{'stopbut'}->configure(
    -command => sub {
    $workers{$timekey}{'frame'}->packForget;
    $shash{ $workers{$timekey}{'setnum'} }{'go'} = 0;
    $shash{ $workers{$timekey}{'setnum'} }{'frame_open'} = 0;
    push @ready, $workers{$timekey}{'setnum'};
    if((scalar @ready) == $numworkers)
    { }
    $workers{$timekey} = ();
    delete $workers{$timekey};
    });

    $workers{$timekey}{'label1'} = $actives{$setnum}{'label1'};
    $workers{$timekey}{'label1'}->configure(
    -textvariable => \$shash{$setnum}{'progress'},
    );

    $workers{$timekey}{'label2'} = $actives{$setnum}{'label2'};


    $workers{$timekey}{'label3'} = $actives{$setnum}{'label3'};
    $workers{$timekey}{'label3'}->configure(-text => $timekey);



    $shash{$setnum}{'go'} = 1;
    $shash{$setnum}{'frame_open'} = 1;
    #--------end of get_file sub--------------------------
    }

    ##################################################################
    sub work{
    my $dthread = shift;
    $|++;
    while(1){
    if($shash{$dthread}{'die'} == 1){ return };

    if ( $shash{$dthread}{'go'} == 1 ){

    eval( system( $shash{$dthread}{'data'} ) );

    foreach my $num (1..100){
    $shash{$dthread}{'progress'} = $num;
    print "\t" x $dthread,"$dthread->$num\n";
    select(undef,undef,undef, .5);
    if($shash{$dthread}{'go'} == 0){last}
    if($shash{$dthread}{'die'} == 1){ return };
    }

    $shash{$dthread}{'go'} = 0; #turn off self before returning
    }else
    { select(undef,undef,undef,.1) }
    }
    }
    #####################################################################

    __END__



    --
    I'm not really a human, but I play one on earth.
    http://zentara.net/japh.html
     
    zentara, Jul 31, 2007
    #6
  7. Guest

    OMG...
    I would rethink all... I need a help on my idea in part of scenario..
    perlmonks may help me.
    Tnx a lot all for help here! If anybody can help with ideas-please
    email me!
     
    , Aug 1, 2007
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. dpackwood
    Replies:
    3
    Views:
    1,842
  2. yoda
    Replies:
    2
    Views:
    472
    =?utf-8?Q?Bj=C3=B6rn_Lindstr=C3=B6m?=
    Aug 1, 2005
  3. threads without threads

    , Aug 27, 2004, in forum: C Programming
    Replies:
    4
    Views:
    439
    William Ahern
    Aug 27, 2004
  4. Pedro Pinto

    Java Threads - Get running threads

    Pedro Pinto, Apr 8, 2008, in forum: Java
    Replies:
    2
    Views:
    1,508
    Arne Vajhøj
    Apr 9, 2008
  5. Une bévue
    Replies:
    0
    Views:
    178
    Une bévue
    Jun 14, 2006
Loading...

Share This Page