Perl Multithreading and use of Perl Modules

Discussion in 'Perl Misc' started by Nate, Jul 6, 2005.

  1. Nate

    Nate Guest

    Hello all,
    I'm trying to create a multithreaded program in Perl 5.8 on Linux.
    I am trying to package up my code into a .pm file, but I would still
    like to be able to do some shared memory between running threads.

    I seem to be hitting a road block where I may want a subroutine that's
    in the module to have access to a shared memory variable.
    Because the module starts up in a seperate memory space I can't get
    access to the shared memory variables that I'm declaring in the main
    program.

    In some cases I'm trying to simply read a shared variable, but in other
    cases I would like to update data in a shared hash.


    Does anyone have any suggestions as to how I could best accomplish
    this?

    Thanks,
    Nate
    Nate, Jul 6, 2005
    #1
    1. Advertising

  2. "Nate" <> wrote in
    news::

    > I'm trying to create a multithreaded program in Perl 5.8 on Linux.
    > I am trying to package up my code into a .pm file, but I would still
    > like to be able to do some shared memory between running threads.
    >
    > I seem to be hitting a road block where I may want a subroutine that's
    > in the module to have access to a shared memory variable.
    > Because the module starts up in a seperate memory space I can't get
    > access to the shared memory variables that I'm declaring in the main
    > program.
    >
    > In some cases I'm trying to simply read a shared variable, but in
    > other cases I would like to update data in a shared hash.


    You have not given enough information. It would be immensely helpful if
    you could give a minimal example which does not do what you want, and
    explain exactly what you would like it to do instead, you will have a
    better chance of getting useful responses.

    As it stands, trying to help you involves setting up a lot of
    scaffolding based on guesswork which I doubt many people will be doing.

    Sinan

    --
    A. Sinan Unur <>
    (reverse each component and remove .invalid for email address)

    comp.lang.perl.misc guidelines on the WWW:
    http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
    A. Sinan Unur, Jul 6, 2005
    #2
    1. Advertising

  3. Nate

    Nate Guest

    > "Nate" <> wrote in
    > news::
    >
    > > I'm trying to create a multithreaded program in Perl 5.8 on Linux.
    > > I am trying to package up my code into a .pm file, but I would still
    > > like to be able to do some shared memory between running threads.


    >
    > You have not given enough information. It would be immensely helpful if
    > you could give a minimal example which does not do what you want, and
    > explain exactly what you would like it to do instead, you will have a
    > better chance of getting useful responses.
    >
    > As it stands, trying to help you involves setting up a lot of
    > scaffolding based on guesswork which I doubt many people will be doing.
    >
    > Sinan


    Sure, more info is always helpful...

    The sample below consists of two files:
    mt_test.pl which is the main perl file, and
    testmulti.pm which is the perl module.

    I would like to find the best way to allow the subroutines in
    testmulti.pm to access some shared variables made possible through
    multithreading. If someone out there believes this isbad programming
    practice, then please suggest the best way to get the info I'm looking
    for in the script below.

    The idea in this example is to run a counter in multiple threads. As
    the counters run, I would like to have a shared variable incremented as
    each counter thread executes it's iteration of the counter.

    The code below is broken, in that I'm not sure how to get the
    subroutines in testmulti.pm to access the shared variable from
    mt_test.pl.
    The tracker() subroutine needs to be able to check the shared variables
    in order to report status. The join method only allows data to be
    returned when the thread exits. However, I would like the tracker
    process to be able to collect live data from the counter threads while
    they are running. In reality, these threads will be running for minutes
    or hours, and I would like to have the tracker process report data as
    close to realtime as possible.

    I have gotten this to run when all the subroutines are in the
    mt_test.pl, but I would like to run the subroutines in a module for
    repeated use among other perl programs.

    When it runs in a single perl script it looks like this:
    # ./mt_test.pl
    Counter1 Counter4 Counter5 TOTAL
    1 1
    2 1 3
    3 1 4
    4 2 6
    5 2 1 8


    However, in the "modularized" state I only get this:
    # ./mt_test.pl

    Counter1 Counter4 Counter5 TOTAL
    All Threads have closed, shutting down tracker
    EXIT Values: 1: 0 2: 0 3: 0 Trk:
    Attempt to free unreferenced scalar: SV 0x8c272bc, Perl interpreter:
    0x8b75cf0 during global destruction.
    Attempt to free unreferenced scalar: SV 0x8ba4f00, Perl interpreter:
    0x8b13ed0 during global destruction.
    Attempt to free unreferenced scalar: SV 0x8b431ac, Perl interpreter:
    0x8ab1b58 during global destruction.
    Attempt to free unreferenced scalar: SV 0x8adffa8, Perl interpreter:
    0x89b3050 during global destruction.



    <mt_test.pl>
    #!/usr/bin/perl

    # Threading requires use of the "threads" module
    use threads;
    # In order to use shared variables, you need to include the
    "threads::shared" module
    use threads::shared;
    use testmulti qw(tracker counter);

    ## COUNTER EXAMPLE ##
    # In this example we're going to start 3 threads of incrementing
    counters
    # each counter runs at a different time interval.
    # The purpose of this example is to show how to use shared memory for
    # constantly running threads

    # Establish some shared variables for the counters
    my %stats : shared; #This hash is used to track the stats of each
    counter.
    my $total : shared; #This string variable holds the total of all
    counters and is writable via variable locking

    my $end_test=9; #Here we define how many test intervals to run

    # Let's run 3 counters of different sleep intervals
    my @tests : shared = (1,2,5);

    # Start the threads
    # Here we are creating threads that will call the counter() subroutine.

    # the second argument is the list of variables to pass to the
    subroutine
    # Syntax: $handle = threads->create(subroutine,list of args);
    my $counter1 = threads->create("counter", "$tests[0],$end_test");
    my $counter4 = threads->create("counter", $tests[1]);
    my $counter5 = threads->create("counter", $tests[2]);


    # Run status tracker in seperate thread
    # We will use this thread to collect the data from the shared variables
    # and print the results.
    # "tracker" is the name of the subroutine to call
    # "1" is the number of seconds to sleep in between checks

    my $tracker1 = threads->create("tracker", "1,@tests");

    # The join() method allows you to collect return data from the thread
    when it closes
    my $return1 = $counter1->join();
    my $return4 = $counter4->join();
    my $return5 = $counter5->join();
    my $returnt = $tracker1->join();
    print "EXIT Values: 1: $return1 2: $return4 3: $return5 Trk:
    $returnt\n";

    </mt_test.pl>





    <testmulti.pm>

    # Multithread test package
    package testmulti;
    use strict;
    use base 'Exporter';
    use vars qw(@ISA @EXPORT);
    @ISA = ('Exporter');
    @EXPORT = qw(&tracker &counter);

    #use threads::shared;

    #my %stats : shared; #This hash is used to track the stats of each
    counter.
    #my $total : shared; #This string variable holds the total of all
    counters. We use this to show locking


    # Subroutine that tracks the results of the running counters
    sub tracker {
    my ($interval, @tests) = @_;

    # Take the count of the number of running processes and increase by
    1 since it's 0 based
    my $num_tests = $#tests;
    $num_tests++;

    #print column headers
    print "Counter1 Counter4 Counter5 TOTAL\n" ;
    # Start the while loop for this subroutine
    while () {
    # Pull the list of running threads and increment counter of
    running threads
    my (@running_threads) = threads->list();
    my $num_threads = $#running_threads;
    $num_threads++;
    # Detect a thread is missing and alert
    print "A Thread has shut down\n" if $num_threads < $num_tests;

    # For the purposes of a clean shutdown, we want the tracker to
    return to the main thread when it's done.
    # It becomes done, when all of the other threads have shut down.
    if ($num_threads == 1){
    print "All Threads have closed, shutting down tracker\n";
    return;
    }

    # Print counter result data, if there's anything to print
    print
    "\t$stats{$tests[0]}\t$stats{$tests[1]}\t$stats{$tests[2]}\t$total\n"
    if $stats{$tests[0]};
    sleep $interval;

    }

    }


    #Simple subroutine for counting something
    sub counter{
    my ($interval, $end_test) = @_;
    # This is an "internal counter" that will be used to generate a return
    value
    # I only use it here to simulate having some value to return.
    my $int_ctr=0;

    # Run the counter until we reach the max number of counter tests to
    run, as defined by a global variable
    while($total < $end_test){
    print "\t\t\t\t\tCounter $interval reports the total as $total\n";
    sleep $interval;
    #lock($stats{$interval});
    $stats{$interval}++;
    # Lock the $total variable for writing. This causes other threads
    to block while waiting for the lock to be released
    # THe lock is released when the code context is exited at the end
    of the BLOCK
    lock($total);
    $total++;
    $int_ctr++;
    }
    # Return a known value so that we know what we're getting in the
    join() method
    return $int_ctr;

    }
    1;

    </testmulti.pm>
    Nate, Jul 7, 2005
    #3
  4. "Nate" <> wrote in news:1120743693.919162.212330
    @g43g2000cwa.googlegroups.com:

    >> "Nate" <> wrote in
    >> news::
    >>
    >> > I'm trying to create a multithreaded program in Perl 5.8 on Linux.


    ....

    >> You have not given enough information.


    ....

    > The sample below consists of two files:
    > mt_test.pl which is the main perl file, and
    > testmulti.pm which is the perl module.


    Thanks for posting that. I have not yet run anything but just a cursory
    look makes me wonder about something. See below.

    > <mt_test.pl>
    > #!/usr/bin/perl


    use strict;
    use warnings;

    missing.

    > # Threading requires use of the "threads" module
    > use threads;


    Long comments like this tend to wrap, and make it difficult to get your
    code to run.

    > my $tracker1 = threads->create("tracker", "1,@tests");


    ....

    > sub tracker {
    > my ($interval, @tests) = @_;
    >

    ....
    > my $num_tests = $#tests;
    > $num_tests++;


    This is weird. You pass tracker only two arguments: The string
    "tracker" and the *string* "1,1, 2, 5". That is, in the sub tracker,
    @tests contains only one element.

    I do not know if this has anything to do with your problem, but it seems
    to be a significant error in logic.

    Please also see

    perldoc -q always.

    Sinan
    --
    A. Sinan Unur <>
    (reverse each component and remove .invalid for email address)

    comp.lang.perl.misc guidelines on the WWW:
    http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
    A. Sinan Unur, Jul 7, 2005
    #4
  5. "Nate" <> wrote in news:1120743693.919162.212330
    @g43g2000cwa.googlegroups.com:

    > # Subroutine that tracks the results of the running counters
    > sub tracker {
    > my ($interval, @tests) = @_;

    ....
    > my $num_tests = $#tests;
    > $num_tests++;


    In addition to my comments elsethread, you seem to be unaware that

    my $num_tests = @tests;

    would give the number of elements in @tests. In addition, unless you are
    changing the number of elements in @tests, you do not need the variable
    $num_tests. You can just use @tests in scalar context.

    Sinan
    --
    A. Sinan Unur <>
    (reverse each component and remove .invalid for email address)

    comp.lang.perl.misc guidelines on the WWW:
    http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
    A. Sinan Unur, Jul 7, 2005
    #5
  6. Nate

    Nate Guest

    A. Sinan Unur wrote:
    > Thanks for posting that. I have not yet run anything but just a cursory
    > look makes me wonder about something. See below.
    >
    > > <mt_test.pl>
    > > #!/usr/bin/perl

    >
    > use strict;
    > use warnings;
    >
    > missing.


    Thanks.

    >
    > > # Threading requires use of the "threads" module
    > > use threads;

    >
    > Long comments like this tend to wrap, and make it difficult to get your
    > code to run.
    >
    > > my $tracker1 = threads->create("tracker", "1,@tests");


    I'll remember that.

    > > sub tracker {
    > > my ($interval, @tests) = @_;
    > >

    > ...
    > > my $num_tests = $#tests;
    > > $num_tests++;

    >
    > This is weird. You pass tracker only two arguments: The string
    > "tracker" and the *string* "1,1, 2, 5". That is, in the sub tracker,
    > @tests contains only one element.
    >
    > I do not know if this has anything to do with your problem, but it seems
    > to be a significant error in logic.


    That was an error on my part. In other versions of the code, I passed
    the list elements one by one. Thank you for pointing that error out. It
    was affecting things.


    >
    > Please also see
    >
    > perldoc -q always.
    >
    > Sinan


    More thanks. In looking at that I was able to figure out a way to do
    what I needed.
    It's been a little while since I've done any serious Perl programming,
    and the use of references slipped my mind. I was able to rework the
    code using references and have since been able to get the code to do
    what I want.

    Also, thank you for the reminder regarding the use of the scalar
    context of the array to get the array count. I rememberted that $@array
    would yield the number of elements -1, so I had been incremented them
    to get the value!

    I have fixed the code and it's working now. I'm going to clean it up a
    little and post it here in a clean post.

    Thanks for your help. Sorry for the newbie mistakes, I really have been
    working in Perl for a little while, not that it shows. :)

    -Nate
    Nate, Jul 7, 2005
    #6
  7. "Nate" <> wrote in
    news::

    > Thanks for your help. Sorry for the newbie mistakes, I really have
    > been working in Perl for a little while, not that it shows. :)


    You are welcome. Don't worry about the mistakes. I do not point those out
    to put anyone down, but to make sure the simple errors are fixed before
    moving on to more complicated issues.

    Please do post the new version with these errors fixed (and somewhat fewer
    comments, it is easier to read code than comments in most cases), so
    everyone can take a look to see if there are any other issues remaining.
    For example, I would have passed a reference to %stats to the threads,
    restricted locks to a smaller scope than the sub etc, but would not have
    done more work before the obvious errors were fixed.

    Sinan

    --
    A. Sinan Unur <>
    (reverse each component and remove .invalid for email address)

    comp.lang.perl.misc guidelines on the WWW:
    http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
    A. Sinan Unur, Jul 7, 2005
    #7
  8. Nate

    Nate Guest

    Please see code Below. Thanks to Sinan's comments I was able to get
    this to work as desired.

    >>restricted locks to a smaller scope than the sub etc,

    Sinan, what did you mean by this comment?


    <mt_test.pl>
    #!/usr/bin/perl
    use strict;
    use warnings;

    use threads;
    use threads::shared;
    use testmulti qw(tracker counter);

    my %stats : shared;
    my $total : shared = 0;
    my $end_test=9;

    # run 3 counters of different sleep intervals
    my @tests = (1,4,5);
    my @args1 = ("$tests[0]",$end_test,\%stats,\$total);
    my @args4 = ("$tests[1]",$end_test,\%stats,\$total);
    my @args5 = ("$tests[2]",$end_test,\%stats,\$total);


    # Start the threads
    my $counter1 = threads->create('counter', @args1);
    my $counter4 = threads->create('counter', @args4);
    my $counter5 = threads->create('counter', @args5);


    # Run status tracker in seperate thread
    my $tracker1 =
    threads->create('tracker',(1,scalar(@tests),\%stats,\$total));

    # collect return data from the thread when it closes
    my $return1 = $counter1->join();
    my $return4 = $counter4->join();
    my $return5 = $counter5->join();
    my $returnt = $tracker1->join();
    print "EXIT Values: 1: $return1 2: $return4 3: $return5 Trk:
    $returnt\n";
    </mt_test.pl>
    # Multithread test package
    package testmulti;
    use strict;
    use base 'Exporter';
    use vars qw(@ISA @EXPORT);
    @ISA = ('Exporter');
    @EXPORT = qw(&tracker &counter);

    # track the results of the running counters
    sub tracker {
    my ($interval,$num_tests,$ref_stats,$ref_total) = @_;
    #print "tracker $interval $ref_stats $ref_total\n";

    #print column headers
    print "Counter1 Counter4 Counter5 TOTAL\n" ;
    # Start the while loop for this subroutine
    while () {
    # Get list of running threads and increment counter
    my (@running_threads) = threads->list();
    # Detect a thread is missing and alert
    print "A Thread has shut down\n" if scalar(@running_threads) <
    $num_tests;

    #Tracker should return to main thread when done.
    # We're done when all other threads shut down.
    if (scalar(@running_threads) == 1){
    print "All Threads have closed, shutting down tracker\n";
    return 1;
    }

    # Print counter result data
    #print 'Keys:' . keys(%$ref_stats) . "\n";
    print
    "\t$ref_stats->{'1'}\t$ref_stats->{'4'}\t$ref_stats->{'5'}\t$$ref_total\n";
    sleep $interval;

    }

    }


    #Simple subroutine for counting something
    sub counter{
    my ($interval, $end_test, $ref_stats, $ref_total) = @_;
    #print "recieved: $interval, $end_test, $ref_stats, $ref_total\n";

    # internal counter that will generate the return value
    my $int_ctr=0;

    # Run the counter until we reach the max number of counter tests
    while($$ref_total < $end_test){
    #print "\t\t\t\t\tCounter $interval reports the total as
    ${$ref_total}\n";
    sleep $interval;
    #lock($stats{$interval});
    @$ref_stats{$interval}++;
    # Lock the $total variable for writing.
    lock(${$ref_total});
    ${$ref_total}++;
    $int_ctr++;
    }
    # test return value to join method
    return $int_ctr;

    }
    1;

    <testmulti.pm>

    </testmulti.pm>
    Nate, Jul 7, 2005
    #8
  9. Nate

    Nate Guest

    Huh, I forgot to paste the testmulti.pm

    <testmulti.pm>
    # Multithread test package
    package testmulti;
    use strict;
    use base 'Exporter';
    use vars qw(@ISA @EXPORT);
    @ISA = ('Exporter');
    @EXPORT = qw(&tracker &counter);

    # track the results of the running counters
    sub tracker {
    my ($interval,$num_tests,$ref_stats,$ref_total) = @_;
    #print "tracker $interval $ref_stats $ref_total\n";

    #print column headers
    print "Counter1 Counter4 Counter5 TOTAL\n" ;
    # Start the while loop for this subroutine
    while () {
    # Get list of running threads and increment counter
    my (@running_threads) = threads->list();
    # Detect a thread is missing and alert
    print "A Thread has shut down\n" if scalar(@running_threads) <
    $num_tests;

    #Tracker should return to main thread when done.
    # We're done when all other threads shut down.
    if (scalar(@running_threads) == 1){
    print "All Threads have closed, shutting down tracker\n";
    return 1;
    }

    # Print counter result data
    #print 'Keys:' . keys(%$ref_stats) . "\n";
    print
    "\t$ref_stats->{'1'}\t$ref_stats->{'4'}\t$ref_stats->{'5'}\t$$ref_total\n";
    sleep $interval;

    }

    }


    #Simple subroutine for counting something
    sub counter{
    my ($interval, $end_test, $ref_stats, $ref_total) = @_;
    #print "recieved: $interval, $end_test, $ref_stats, $ref_total\n";

    # internal counter that will generate the return value
    my $int_ctr=0;

    # Run the counter until we reach the max number of counter tests
    while($$ref_total < $end_test){
    #print "\t\t\t\t\tCounter $interval reports the total as
    ${$ref_total}\n";
    sleep $interval;
    #lock($stats{$interval});
    @$ref_stats{$interval}++;
    # Lock the $total variable for writing.
    lock(${$ref_total});
    ${$ref_total}++;
    $int_ctr++;
    }
    # test return value to join method
    return $int_ctr;

    }
    1;

    </testmulti.pm>
    Nate, Jul 7, 2005
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?U2hhcGlybw==?=

    HttpModule multithreading and request and response corelation

    =?Utf-8?B?U2hhcGlybw==?=, Dec 7, 2004, in forum: ASP .Net
    Replies:
    7
    Views:
    779
    Scott Allen
    Dec 8, 2004
  2. Remy Cool
    Replies:
    1
    Views:
    431
    Remy Cool
    Aug 27, 2003
  3. ImpalerCore
    Replies:
    0
    Views:
    835
    ImpalerCore
    Mar 10, 2011
  4. Merrilee Larson

    Do I *have* to use 'OOP' to use modules?

    Merrilee Larson, Nov 17, 2006, in forum: Perl Misc
    Replies:
    41
    Views:
    368
    Arved Sandstrom
    Nov 25, 2006
  5. Replies:
    16
    Views:
    497
Loading...

Share This Page