ideas for a "parameter sweep" program?

D

Diego Virasoro

Hello,
I often need to run my computational experiments with a combination of
parameter values, and I was thinking of automatising it with Ruby. Any
idea on what would be the best way to go about it?

What would be the advantage of trying something like a DSL? I've never
done it before so it looks a little bit daunting and it may be an
overkill, but then this is a personal project so I don't have
deadlines and I'd like to learn a bit in the process.

Or maybe should I cook up something with Rake?

Ideas?

Diego
 
I

Intransition

Hello,
I often need to run my computational experiments with a combination of
parameter values, and I was thinking of automatising it with Ruby. Any
idea on what would be the best way to go about it?

What would be the advantage of trying something like a DSL? I've never
done it before so it looks a little bit daunting and it may be an
overkill, but then this is a personal project so I don't have
deadlines and I'd like to learn a bit in the process.

Or maybe should I cook up something with Rake?

Depends on what you want. If you just want to create an executable
that can accept various options look at the built in OptionParser.

http://ruby-doc.org/stdlib/libdoc/optparse/rdoc/classes/OptionParser.html
 
B

Brian Candler

Diego said:
I often need to run my computational experiments with a combination of
parameter values, and I was thinking of automatising it with Ruby. Any
idea on what would be the best way to go about it?

Is the computational experiment written in Ruby? Then wrap it in a
class, and pass the parameters to your initialize method. This allows
you to re-use the code in different ways. For example, you can create
one runner program which takes the parameters from the command line;
another runner program which repeatedly runs the experiment but sweeping
across a range of values; and so on.

Even if the experiment is not written in ruby, you can still wrap it in
a ruby class and call it as an external program using %x{}, system() or
IO.popen().
Or maybe should I cook up something with Rake?

rake supports passing (string) parameters as environment variables:
rake mytest FOO=xyz BAR=456

so a 'rake runner' might look like this:

require 'experiment'
task :mytest do
result = Experiment.new:)foo => ENV['FOO'], :bar =>
ENV['BAR'].to_f).run
puts result
end
 
R

Robert Klemme

2009/11/23 Diego Virasoro said:
I often need to run my computational experiments with a combination of
parameter values, and I was thinking of automatising it with Ruby. Any
idea on what would be the best way to go about it?

Not sure what you actually mean. Are you looking for something like

p1 = [1,2,10,3]
p2 = [1312,42345,235435,65]
p3 = [123,23,476,43]

p1.each do |param1|
p2.each do |param2|
p3.each do |param3|
system "experiment", param1, param2, param3
end
end

or rather

params = [
[1,11,2],
[23,4,21],
[5,1,4],
]

params.each do |pa|
system "experiment", *pa
end

?

Kind regards

robert
 
D

Diego Virasoro

Is the computational experiment written in Ruby? Then wrap it in a
class, and pass the parameters to your initialize method. This allows
you to re-use the code in different ways.

I would like a generic solution that works with any language for any
program (including Fortran that doesn't allow to pass arguments form
the command line). Basically what it should do is look for some
regexp, swap some values, and the rerun the experiment. Then rinse and
repeat. All the output should be catalogued so that afterwards I can
run some data analysis.

Diego
 
B

Brian Candler

Diego said:
I would like a generic solution that works with any language for any
program (including Fortran that doesn't allow to pass arguments form
the command line).

So how do you pass parameters to your Fortran program - in a config file
perhaps? If so, then just generate that file then run the program. It's
up to you to code what you want.

Does it read a flat file? XML? YAML? Something else?

class FlatFileRunner
def initialize(exe, conffile)
@exe, @conffile = exe, conffile
end
def run(params = {})
File.open(@conffile, "w") do |f|
params.each do |k,v|
f.puts "#{k}=#{v}"
end
end
%x{"#{@exe}"}
end
end

prog = FlatFileRunner.new("./flurble", "./flurble.conf")
res = prog.run("foo"=>123)
puts "Result of experiment: #{res}"
 
D

Diego Virasoro

2009/11/23 Diego Virasoro said:
I often need to run my computational experiments with a combination of
parameter values, and I was thinking of automatising it with Ruby. Any
idea on what would be the best way to go about it?

Not sure what you actually mean.  Are you looking for something like

p1 = [1,2,10,3]
p2 = [1312,42345,235435,65]
p3 = [123,23,476,43]

p1.each do |param1|
  p2.each do |param2|
    p3.each do |param3|
      system "experiment", param1, param2, param3
  end
end

Yep, something like this in its simplest form.
But I'd like to make it a little more generic and powerful, for
example allowing for some action before or after the main call, for
the user to express the name of the experiment, for actions before or
after the whole script, etc...

The question wasn't how to do it, but what would be the best way to do
it, if you see what I mean. I've got a few ideas and I am not sure
what will work because some (like a DSL) I've never done before and
others (like a Yaml file) may not be enough.

Diego
 
D

Diego Virasoro

So how do you pass parameters to your Fortran program - in a config file
perhaps? If so, then just generate that file then run the program. It's
up to you to code what you want.

Yes, it would generally read a file. Don't get me wrong: I know how to
do it. Currently I've got a bash script that does it for one
particular executable by using sed and make. I just wanted to
generalise it and given I hate bash Ruby seemed like the best
alternative.

However I would also like to make it a lot more generic and flexible,
as maybe other people in my group may be interested in using it too.

Which lead to the question: what would be the best way? A rake file, a
yaml input file, some internal DSL?

Diego
 
B

Brian Candler

Diego said:
Which lead to the question: what would be the best way? A rake file, a
yaml input file, some internal DSL?

Do The Simplest Thing Which Can Possibly Work, which I think in this
case means: just start writing code.

If you find yourself writing the same thing over and over again, then
write a shortcut for that particular pattern. That shortcut or set of
shortcuts will become your "DSL".

And don't be afraid to refactor later.
 
R

Robert Klemme

2009/11/24 Diego Virasoro said:
2009/11/23 Diego Virasoro said:
I often need to run my computational experiments with a combination of
parameter values, and I was thinking of automatising it with Ruby. Any
idea on what would be the best way to go about it?

Not sure what you actually mean. =A0Are you looking for something like

p1 =3D [1,2,10,3]
p2 =3D [1312,42345,235435,65]
p3 =3D [123,23,476,43]

p1.each do |param1|
=A0 p2.each do |param2|
=A0 =A0 p3.each do |param3|
=A0 =A0 =A0 system "experiment", param1, param2, param3
=A0 end
end

Yep, something like this in its simplest form.
But I'd like to make it a little more generic and powerful, for
example allowing for some action before or after the main call, for
the user to express the name of the experiment, for actions before or
after the whole script, etc...

Then you should probably start with filling on those dots. If the
required feature set is unclear the proposed solution might be
inappropriate.
The question wasn't how to do it, but what would be the best way to do
it, if you see what I mean. I've got a few ideas and I am not sure
what will work because some (like a DSL) I've never done before and
others (like a Yaml file) may not be enough.

As I said: first it needs to be clear what needs to be done. What
kind of inputs do you need or want (a single configuration file or a
number of files, should it be controlled by some command line
arguments etc.) and what actions do you need. Are the actions before
and after the experiment uniform or do they depend on the command line
args of that particular execution. Things like that can make a
difference for the viability of any solution.

Kind regards

robert


--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/
 
D

Diego Virasoro

As I said: first it needs to be clear what needs to be done.  What
kind of inputs do you need or want (a single configuration file or a
number of files, should it be controlled by some command line
arguments etc.) and what actions do you need.  Are the actions before
and after the experiment uniform or do they depend on the command line
args of that particular execution.  Things like that can make a
difference for the viability of any solution.
You are completely right.

I hoped it was something people had already faced, or maybe even
solved, or maybe had ideas to give (I doubt I'll get all the
requirements right now). I think the real solution is to follow
Brian's advice and just start coding... it will mean a lot of
refactoring, but that's life, I guess. :)

Diego
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,479
Members
44,899
Latest member
RodneyMcAu

Latest Threads

Top