Perl and Sun Grid Engine (SGE)

[Leo]

Dear all,

I'm writing code to run on a cluster machine. Basically I need to run
a Perl program 200 times and each time generate a line of the output.
I use a C-shell script to call the perl progam and distribute it
through Sun Grid Engine(SGE) to the nodes of the cluster (because SGE
only takes shell script). My have two implementations: (1) write the
output into different files (2) append the output to a single file.
Obviously the first approach is not efficient. But for the second one,
I've only got less than 200 results(120-180 output lines). My
questions are: (1) is this the problem with the deadlock when writing
files? if so, how perl handle this issue? (like the one in C, means I
have to write my own code to deal with the situation?) (2) Or SGE has
the ability to not allow the file-writing at the same time? I hope
someone can explain this to me and I would appreciate your help!
Thanks.

Leo

 

[Ben Morrow]

Quoth (e-mail address removed) (Leo):

I'm writing code to run on a cluster machine. Basically I need to run
a Perl program 200 times and each time generate a line of the output.
I use a C-shell script to call the perl progam and distribute it
through Sun Grid Engine(SGE) to the nodes of the cluster (because SGE
only takes shell script). My have two implementations: (1) write the
output into different files (2) append the output to a single file.
Obviously the first approach is not efficient. But for the second one,
I've only got less than 200 results(120-180 output lines). My
questions are: (1) is this the problem with the deadlock when writing
files? if so, how perl handle this issue? (like the one in C, means I
have to write my own code to deal with the situation?)

I'm not sure what you mean by 'deadlock', but the answer is that you
need to lock the files you are using. As the files are (presumably) on
nfs, you will need to use fcntl rather than flock locking.

Ben

 

[Peter Scott]

I'm writing code to run on a cluster machine. Basically I need to run
a Perl program 200 times and each time generate a line of the output.
I use a C-shell script to call the perl progam and distribute it
through Sun Grid Engine(SGE) to the nodes of the cluster (because SGE
only takes shell script).

You may want to read this:

http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/

One hopes that SGE at least has Bourne as an alternative...

 

[ctcgag]

Dear all,

I'm writing code to run on a cluster machine. Basically I need to run
a Perl program 200 times and each time generate a line of the output.
I use a C-shell script to call the perl progam and distribute it
through Sun Grid Engine(SGE) to the nodes of the cluster (because SGE
only takes shell script). My have two implementations: (1) write the
output into different files (2) append the output to a single file.

Use implementation 1.

Obviously the first approach is not efficient.

What about it is not efficient? If you had used that method, you would
be done by now, rather than rooting around in here for help. I'd call
that *more* efficient.

But for the second one,
I've only got less than 200 results(120-180 output lines). My
questions are: (1) is this the problem with the deadlock when writing
files?

You haven't given us enough information to tell.

if so, how perl handle this issue?

Show us the few lines of your code that pertain to this issue, and we
may be able to tell you what the problem is.

(like the one in C, means I
have to write my own code to deal with the situation?)

How do you deal with it in C? If you tell us, we may be able to
help you translate that into Perl. If you don't, we can't.

(2) Or SGE has
the ability to not allow the file-writing at the same time?

Assuming you handle the files in Perl, then it is really none of
SGE's business to allow or disallow. But I do notice when it is
SGE's business to handle I/O (i.e. the shell's stdout and stderr),
it does so by directing them into different files.


Xho