parallel tasks with rake ?

Discussion in 'Ruby' started by quixoticsycophant@yahoo.com, Nov 1, 2006.

  1. Guest

    I was excited to switch my makefile system to rake, but I just noticed
    rake lacks a (GNU) make option I can't live without: -j4. It's like
    "make & ; make & ; make & ; make &" but with the parallel jobs
    properly synchronized. On a quad-core machine this makes a world of
    difference, being around four times faster than a simple 'make'.

    As I understand it, rake should be able to determine which targets can
    be built in parallel just as (GNU) make does. rake's multitask
    feature isn't quite right because it shouldn't be up to the user to
    determine parallel vs non-parallel tasks. In fact, the user should
    probably never decide parallelism by fiat; even if the user happens to
    be correct, it creates a maintenance problem whereby additional tasks
    will eventually render the parallel assertion false.

    Perhaps there is something I'm missing about 'multitask' ?
    , Nov 1, 2006
    #1
    1. Advertising

  2. On 11/1/06, <> wrote:
    > I was excited to switch my makefile system to rake, but I just noticed
    > rake lacks a (GNU) make option I can't live without: -j4. It's like
    > "make & ; make & ; make & ; make &" but with the parallel jobs
    > properly synchronized. On a quad-core machine this makes a world of
    > difference, being around four times faster than a simple 'make'.
    >
    > As I understand it, rake should be able to determine which targets can
    > be built in parallel just as (GNU) make does. rake's multitask
    > feature isn't quite right because it shouldn't be up to the user to
    > determine parallel vs non-parallel tasks. In fact, the user should
    > probably never decide parallelism by fiat; even if the user happens to
    > be correct, it creates a maintenance problem whereby additional tasks
    > will eventually render the parallel assertion false.
    >
    > Perhaps there is something I'm missing about 'multitask' ?
    >
    >


    How does Make figure out which jobs can be run in parallel? In my
    experience, it just runs everything in parallel, and things break when
    the tasks aren't safely parallelizable. (e.g. doing make -j4 install
    on a FreeBSD port)
    Wilson Bilkovich, Nov 1, 2006
    #2
    1. Advertising

  3. Guest

    Wilson Bilkovich wrote:
    > On 11/1/06, <> wrote:
    > > I was excited to switch my makefile system to rake, but I just
    > > noticed rake lacks a (GNU) make option I can't live without: -j4.
    > > It's like "make & ; make & ; make & ; make &" but with the
    > > parallel jobs properly synchronized. On a quad-core machine this
    > > makes a world of difference, being around four times faster than a
    > > simple 'make'.
    > >
    > > As I understand it, rake should be able to determine which targets
    > > can be built in parallel just as (GNU) make does. rake's
    > > multitask feature isn't quite right because it shouldn't be up to
    > > the user to determine parallel vs non-parallel tasks. In fact,
    > > the user should probably never decide parallelism by fiat; even if
    > > the user happens to be correct, it creates a maintenance problem
    > > whereby additional tasks will eventually render the parallel
    > > assertion false.
    > >
    > > Perhaps there is something I'm missing about 'multitask' ?

    >
    > How does Make figure out which jobs can be run in parallel? In my
    > experience, it just runs everything in parallel, and things break when
    > the tasks aren't safely parallelizable. (e.g. doing make -j4 install
    > on a FreeBSD port)


    The Make program figures it out from the dependency graph.
    Unfortunately, the 'install' target you mention is a fake target which
    depends on nothing. Such uses of 'install' are common but incorrect.
    As a workaround, 'install' (and any other wrongly written targets)
    should be treated as a special case and should not be run with -j.
    , Nov 1, 2006
    #3
  4. Hugh Sasse Guest

    On Thu, 2 Nov 2006, wrote:

    >
    > Wilson Bilkovich wrote:
    > >
    > > How does Make figure out which jobs can be run in parallel? In my
    > > experience, it just runs everything in parallel, and things break when
    > > the tasks aren't safely parallelizable. (e.g. doing make -j4 install
    > > on a FreeBSD port)

    >
    > The Make program figures it out from the dependency graph.
    > Unfortunately, the 'install' target you mention is a fake target which
    > depends on nothing. Such uses of 'install' are common but incorrect.
    > As a workaround, 'install' (and any other wrongly written targets)
    > should be treated as a special case and should not be run with -j.


    Would marking it as a .PHONY help? I see little advice in the GNU
    make manual about how to get this sort of thing right for parallel
    make. Where should one look for such info?

    Hugh
    Hugh Sasse, Nov 1, 2006
    #4
  5. Guest

    Hugh Sasse wrote:
    > Would marking it as a .PHONY help? I see little advice in the GNU
    > make manual about how to get this sort of thing right for parallel
    > make. Where should one look for such info?


    First of all, the workaround isn't so bad:

    $ make -j4 && make install

    In order for 'make -j4 install' to work, the install target must
    simply depend on everything it needs. Needless to say, most Makefile
    writers assume a 'make && make install' invocation and don't write
    proper prerequisites for the install target. In other words, they
    assume a single-process make.

    Marking the target .PHONY just means it will run unconditionally (so
    for example Make won't be confused by a file named 'install') and
    won't affect the correctness or incorrectness of the dependencies.

    To get back to the original point, the dependency graph is what
    decides parallelism, not the other way around. Unless I misunderstand
    the rake docs, the 'multitask' feature of rake is dangerous and should
    be removed (see my original post).

    Rake is a very nice tool, but it needs a -j equivalent in order to be
    considered as an alternative to make (at least in my case). I put it
    out to enterprising rubyists to give rake genuine parallel task
    execution.
    , Nov 1, 2006
    #5
  6. On Nov 1, 2006, at 1:15 PM, wrote:

    > Rake is a very nice tool, but it needs a -j equivalent in order to be
    > considered as an alternative to make (at least in my case). I put it
    > out to enterprising rubyists to give rake genuine parallel task
    > execution.


    I bet the pieces of this could be written up as a fun Ruby Quiz or
    two. Determining what can be done in parallel from the dependancy
    graph in particular sounds like a great quiz topic. Then you could
    Rakify the solutions and submit a patch. Just a thought...

    James Edward Gray II
    James Edward Gray II, Nov 1, 2006
    #6
  7. gus Guest

    wrote:
    > To get back to the original point, the dependency graph is what
    > decides parallelism, not the other way around.


    In the current implementation, is the dependency graph explicitly built
    by rake? Or does it simply recursively run the dependencies as it goes
    along?

    Guillaume.
    gus, Nov 1, 2006
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Francis Hwang
    Replies:
    0
    Views:
    226
    Francis Hwang
    May 26, 2006
  2. Esad Hajdarevic

    global rake tasks

    Esad Hajdarevic, Oct 9, 2006, in forum: Ruby
    Replies:
    1
    Views:
    98
    Kalman Noel
    Oct 9, 2006
  3. Esad Hajdarevic

    rakelibdir (was: Global rake tasks)

    Esad Hajdarevic, Oct 16, 2006, in forum: Ruby
    Replies:
    0
    Views:
    88
    Esad Hajdarevic
    Oct 16, 2006
  4. peppermonkey
    Replies:
    1
    Views:
    226
    Gregory Brown
    Feb 10, 2007
  5. John Feminella
    Replies:
    0
    Views:
    227
    John Feminella
    Apr 25, 2010
Loading...

Share This Page