parallel tasks with rake ?

  • Thread starter quixoticsycophant
  • Start date
Q

quixoticsycophant

I was excited to switch my makefile system to rake, but I just noticed
rake lacks a (GNU) make option I can't live without: -j4. It's like
"make & ; make & ; make & ; make &" but with the parallel jobs
properly synchronized. On a quad-core machine this makes a world of
difference, being around four times faster than a simple 'make'.

As I understand it, rake should be able to determine which targets can
be built in parallel just as (GNU) make does. rake's multitask
feature isn't quite right because it shouldn't be up to the user to
determine parallel vs non-parallel tasks. In fact, the user should
probably never decide parallelism by fiat; even if the user happens to
be correct, it creates a maintenance problem whereby additional tasks
will eventually render the parallel assertion false.

Perhaps there is something I'm missing about 'multitask' ?
 
W

Wilson Bilkovich

I was excited to switch my makefile system to rake, but I just noticed
rake lacks a (GNU) make option I can't live without: -j4. It's like
"make & ; make & ; make & ; make &" but with the parallel jobs
properly synchronized. On a quad-core machine this makes a world of
difference, being around four times faster than a simple 'make'.

As I understand it, rake should be able to determine which targets can
be built in parallel just as (GNU) make does. rake's multitask
feature isn't quite right because it shouldn't be up to the user to
determine parallel vs non-parallel tasks. In fact, the user should
probably never decide parallelism by fiat; even if the user happens to
be correct, it creates a maintenance problem whereby additional tasks
will eventually render the parallel assertion false.

Perhaps there is something I'm missing about 'multitask' ?

How does Make figure out which jobs can be run in parallel? In my
experience, it just runs everything in parallel, and things break when
the tasks aren't safely parallelizable. (e.g. doing make -j4 install
on a FreeBSD port)
 
Q

quixoticsycophant

Wilson said:
How does Make figure out which jobs can be run in parallel? In my
experience, it just runs everything in parallel, and things break when
the tasks aren't safely parallelizable. (e.g. doing make -j4 install
on a FreeBSD port)

The Make program figures it out from the dependency graph.
Unfortunately, the 'install' target you mention is a fake target which
depends on nothing. Such uses of 'install' are common but incorrect.
As a workaround, 'install' (and any other wrongly written targets)
should be treated as a special case and should not be run with -j.
 
H

Hugh Sasse

The Make program figures it out from the dependency graph.
Unfortunately, the 'install' target you mention is a fake target which
depends on nothing. Such uses of 'install' are common but incorrect.
As a workaround, 'install' (and any other wrongly written targets)
should be treated as a special case and should not be run with -j.

Would marking it as a .PHONY help? I see little advice in the GNU
make manual about how to get this sort of thing right for parallel
make. Where should one look for such info?

Hugh
 
Q

quixoticsycophant

Hugh said:
Would marking it as a .PHONY help? I see little advice in the GNU
make manual about how to get this sort of thing right for parallel
make. Where should one look for such info?

First of all, the workaround isn't so bad:

$ make -j4 && make install

In order for 'make -j4 install' to work, the install target must
simply depend on everything it needs. Needless to say, most Makefile
writers assume a 'make && make install' invocation and don't write
proper prerequisites for the install target. In other words, they
assume a single-process make.

Marking the target .PHONY just means it will run unconditionally (so
for example Make won't be confused by a file named 'install') and
won't affect the correctness or incorrectness of the dependencies.

To get back to the original point, the dependency graph is what
decides parallelism, not the other way around. Unless I misunderstand
the rake docs, the 'multitask' feature of rake is dangerous and should
be removed (see my original post).

Rake is a very nice tool, but it needs a -j equivalent in order to be
considered as an alternative to make (at least in my case). I put it
out to enterprising rubyists to give rake genuine parallel task
execution.
 
J

James Edward Gray II

Rake is a very nice tool, but it needs a -j equivalent in order to be
considered as an alternative to make (at least in my case). I put it
out to enterprising rubyists to give rake genuine parallel task
execution.

I bet the pieces of this could be written up as a fun Ruby Quiz or
two. Determining what can be done in parallel from the dependancy
graph in particular sounds like a great quiz topic. Then you could
Rakify the solutions and submit a patch. Just a thought...

James Edward Gray II
 
G

gus

To get back to the original point, the dependency graph is what
decides parallelism, not the other way around.

In the current implementation, is the dependency graph explicitly built
by rake? Or does it simply recursively run the dependencies as it goes
along?

Guillaume.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,733
Messages
2,569,440
Members
44,830
Latest member
ZADIva7383

Latest Threads

Top