Perl Fork options

Discussion in 'Perl Misc' started by BryanBBFan@ca.rr.com, Aug 25, 2008.

  1. Guest

    I have an application which uses PBS to allocates 32 nodes on a Linux
    Cluster. A Perl script needs to spawn off 100+ completely independent
    processes from this job. The Perl use fork to do this. The script
    came from a shared memory machine where ther was plenty of memory each
    node could access. The system it needs to run on is not a shared
    memory system, where each node has its own memory. The problem is
    that although 32 nodes have been allocated for this job, each "fork"
    spawns off a process on the same node, and therefore runs out of
    memory.

    My question is: Is there any option to the "fork" command to tell the
    child process to go to particular nodes?

    Thanks
     
    , Aug 25, 2008
    #1
    1. Advertising

  2. Guest

    wrote:
    > I have an application which uses PBS to allocates 32 nodes on a Linux
    > Cluster.


    Phosphate Buffered Saline?

    > A Perl script needs to spawn off 100+ completely independent
    > processes from this job. The Perl use fork to do this. The script
    > came from a shared memory machine where ther was plenty of memory each
    > node could access. The system it needs to run on is not a shared
    > memory system, where each node has its own memory. The problem is
    > that although 32 nodes have been allocated for this job, each "fork"
    > spawns off a process on the same node, and therefore runs out of
    > memory.


    And even if it didn't, wouldn't they all be using the same node's CPU
    and not spreading out? Or does "PBS" take care of that (but without
    taking care of the memory)?

    >
    > My question is: Is there any option to the "fork" command to tell the
    > child process to go to particular nodes?


    Perl's fork is basically just a wrapper around the OS's fork, at least
    if your OS has a fork.

    Do you know how you would accomplish what you want in C or something other
    than Perl? If you do and you tell us how you would do it, we might be able
    to help you translate that into Perl.

    Otherwise, the first thing that comes to my mind is to try system rather
    than fork. The second thing that comes to my mind is that when I use
    a grid with SunGridEngine I found that the best way was to learn all the
    quirks of SGE and then tell SGE I wanted 100 Perl jobs, not tell Perl I
    wanted 100 Perl jobs. Maybe SGE is so different from "PBS" that this
    answer doesn't port.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Aug 25, 2008
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Cliff  Martin
    Replies:
    1
    Views:
    3,055
    Larry Smith
    Jan 31, 2007
  2. jacksu
    Replies:
    0
    Views:
    525
    jacksu
    Oct 9, 2007
  3. Utkado
    Replies:
    2
    Views:
    2,162
  4. Eric Snow

    os.fork and pty.fork

    Eric Snow, Jan 8, 2009, in forum: Python
    Replies:
    0
    Views:
    579
    Eric Snow
    Jan 8, 2009
  5. Bryan Richardson
    Replies:
    6
    Views:
    301
    Bryan Richardson
    Feb 25, 2008
Loading...

Share This Page