Waiting for condor output files

[Milo Thurston]

I've got a script that submits a whole load of condor jobs, and has to
wait for each result to return in order to read it.
Would something like this do, or is there a better solution?

files.each do |file|
if FileTest::exists?(file)
# process the file
else
redo
end
end

Thanks!

 

[Joel VanderWerf]

I've got a script that submits a whole load of condor jobs, and has to
wait for each result to return in order to read it.
Would something like this do, or is there a better solution?

Maybe this (essentially the same but more compact)?

files.each do |file|

sleep 1 until FileTest::exists?(file)

 

[Martin DeMello]

I've got a script that submits a whole load of condor jobs, and has to
wait for each result to return in order to read it.
Would something like this do, or is there a better solution?

Do they have to be returned in the order they were submitted?

martin

 

[Milo Thurston]

Do they have to be returned in the order they were submitted?

They can come back in any order.
I've also found that condor sometimes does not return them, so I'll
have to include a timeout in the loop. CPU usage is quite high, too.
Milo.

 

[Martin DeMello]

They can come back in any order.
I've also found that condor sometimes does not return them, so I'll
have to include a timeout in the loop. CPU usage is quite high, too.

You might be better off with a thread per loop, then, which fills an
entry into a global array or hash when its particular job finishes. That
way, if you want you can start processing files as they come in, and not
block on waiting for the next in sequence to get done.

martin

 

[Milo Thurston]

You might be better off with a thread per loop, then, which fills an
entry into a global array or hash when its particular job finishes. That
way, if you want you can start processing files as they come in, and not
block on waiting for the next in sequence to get done.

That's not something I've encountered yet.
Are there any on-line examples I could take a look at, to see
how it works?
Thanks.

 

[Martin DeMello]

That's not something I've encountered yet.
Are there any on-line examples I could take a look at, to see
how it works?

Check out http://www.rubycentral.com/book/tut_threads.html - Ruby's
threading mechanism is amazingly lightweight.

martin

 

[Ara.T.Howard]

That's not something I've encountered yet.
Are there any on-line examples I could take a look at, to see
how it works?
Thanks.

this already does it

http://freepan.org/ruby/by-dist/fs-watcher/

check out the testsuite for examples

you would probably use it like:


watcher = FileSystemWatcher.new("condor_outdir", "*")
watcher.sleepTime = 1
watcher.start { |status,file|
if status == FileSystemWatcher::CREATED then
puts "#{ file } was created"
elsif status == FileSystemWatcher::MODIFIED then
puts "#{ file } was modified"
elsif status == FileSystemWatcher:ELETED then
puts "#{ file } was deleted"
end
}

cheers.

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it; and a weed grows, even though we do
| not love it. --Dogen
===============================================================================

 

[Milo Thurston]

this already does it

http://freepan.org/ruby/by-dist/fs-watcher/

Excellent - thanks!
Just what I was looking for.

 

[Milo Thurston]

....your mail seems to be bouncing, but you could try
nsb at ceh ac uk
for your condor questions.

 

[Milo Thurston]

http://freepan.org/ruby/by-dist/fs-watcher/

Hmmm...
It seems not to like being told to look for the creation of empty files:

/usr/lib/ruby/site_ruby/1.8/filesystemwatcher.rb:63:in `addFile': File
'/tmp/switchfinder/NC_000868.862310.T.16.out' either doesnt exist or
isnt readable (FSWatcher::InvalidFileError)

So, I've tried creating the files and getting it to look for
modifications, but it simply reads the empty files and returns an empty
results page to the browser.
Have you any idea why?
Thanks,
Milo.