Perl and Sun Grid Engine (SGE)

Discussion in 'Perl Misc' started by Leo, Jun 17, 2004.

  1. Leo

    Leo Guest

    Dear all,

    I'm writing code to run on a cluster machine. Basically I need to run
    a Perl program 200 times and each time generate a line of the output.
    I use a C-shell script to call the perl progam and distribute it
    through Sun Grid Engine(SGE) to the nodes of the cluster (because SGE
    only takes shell script). My have two implementations: (1) write the
    output into different files (2) append the output to a single file.
    Obviously the first approach is not efficient. But for the second one,
    I've only got less than 200 results(120-180 output lines). My
    questions are: (1) is this the problem with the deadlock when writing
    files? if so, how perl handle this issue? (like the one in C, means I
    have to write my own code to deal with the situation?) (2) Or SGE has
    the ability to not allow the file-writing at the same time? I hope
    someone can explain this to me and I would appreciate your help!
    Thanks.

    Leo
    Leo, Jun 17, 2004
    #1
    1. Advertising

  2. Leo

    Ben Morrow Guest

    Quoth (Leo):
    >
    > I'm writing code to run on a cluster machine. Basically I need to run
    > a Perl program 200 times and each time generate a line of the output.
    > I use a C-shell script to call the perl progam and distribute it
    > through Sun Grid Engine(SGE) to the nodes of the cluster (because SGE
    > only takes shell script). My have two implementations: (1) write the
    > output into different files (2) append the output to a single file.
    > Obviously the first approach is not efficient. But for the second one,
    > I've only got less than 200 results(120-180 output lines). My
    > questions are: (1) is this the problem with the deadlock when writing
    > files? if so, how perl handle this issue? (like the one in C, means I
    > have to write my own code to deal with the situation?)


    I'm not sure what you mean by 'deadlock', but the answer is that you
    need to lock the files you are using. As the files are (presumably) on
    nfs, you will need to use fcntl rather than flock locking.

    Ben

    --
    $.=1;*g=sub{print@_};sub r($$\$){my($w,$x,$y)=@_;for(keys%$x){/main/&&next;*p=$
    $x{$_};/(\w)::$/&&(r($w.$1,$x.$_,$y),next);$y eq\$p&&&g("$w$_")}};sub t{for(@_)
    {$f&&($_||&g(" "));$f=1;r"","::",$_;$_&&&g(chr(0012))}};t #
    $J::u::s::t, $a::n::eek:::t::h::e::r, $P::e::r::l, $h::a::c::k::e::r, $.
    Ben Morrow, Jun 18, 2004
    #2
    1. Advertising

  3. Leo

    Peter Scott Guest

    In article <>,
    (Leo) writes:
    >I'm writing code to run on a cluster machine. Basically I need to run
    >a Perl program 200 times and each time generate a line of the output.
    >I use a C-shell script to call the perl progam and distribute it
    >through Sun Grid Engine(SGE) to the nodes of the cluster (because SGE
    >only takes shell script).


    You may want to read this:

    http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/

    One hopes that SGE at least has Bourne as an alternative...

    --
    Peter Scott
    http://www.perldebugged.com/
    *** NEW *** http://www.perlmedic.com/
    Peter Scott, Jun 18, 2004
    #3
  4. Leo

    Guest

    (Leo) wrote:
    > Dear all,
    >
    > I'm writing code to run on a cluster machine. Basically I need to run
    > a Perl program 200 times and each time generate a line of the output.
    > I use a C-shell script to call the perl progam and distribute it
    > through Sun Grid Engine(SGE) to the nodes of the cluster (because SGE
    > only takes shell script). My have two implementations: (1) write the
    > output into different files (2) append the output to a single file.


    Use implementation 1.

    > Obviously the first approach is not efficient.


    What about it is not efficient? If you had used that method, you would
    be done by now, rather than rooting around in here for help. I'd call
    that *more* efficient. :)

    > But for the second one,
    > I've only got less than 200 results(120-180 output lines). My
    > questions are: (1) is this the problem with the deadlock when writing
    > files?


    You haven't given us enough information to tell.

    > if so, how perl handle this issue?


    Show us the few lines of your code that pertain to this issue, and we
    may be able to tell you what the problem is.


    > (like the one in C, means I
    > have to write my own code to deal with the situation?)


    How do you deal with it in C? If you tell us, we may be able to
    help you translate that into Perl. If you don't, we can't.


    > (2) Or SGE has
    > the ability to not allow the file-writing at the same time?


    Assuming you handle the files in Perl, then it is really none of
    SGE's business to allow or disallow. But I do notice when it is
    SGE's business to handle I/O (i.e. the shell's stdout and stderr),
    it does so by directing them into different files.


    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Jun 18, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Joan MacEachern
    Replies:
    0
    Views:
    326
    Joan MacEachern
    Oct 1, 2003
  2. Joan MacEachern
    Replies:
    0
    Views:
    349
    Joan MacEachern
    Oct 22, 2003
  3. Replies:
    0
    Views:
    2,868
  4. J.B. Brown
    Replies:
    1
    Views:
    549
    Neil Hodgson
    Jul 23, 2010
  5. MRAB
    Replies:
    0
    Views:
    421
Loading...

Share This Page