Problem with rake and dependencies

[Brock Rycenga]

Hello:

I am new to rake, and I am stuck on a problem with incremental building.
My rakefile does what I would like, for the most part. If I touch a .c
file, the rakefile functions properly and only builds the necessary
files. However, if I touch the corresponding .h file, my rakefile
assumes everything is up-to-date. I am pretty certain that I am not
importing/loading my dependency files correctly. Please help.

I am using a dependency generation feature that is part of the DSP
compiler we are using. The output from the preprocessor creates files
with a .pp extension, formatted like so:

C:/dir/obj/lcd.obj: C:\dir\lcd.c
C:/dir/obj/lcd.obj: C:\dir\lcd.h
C:/dir/obj/lcd.obj: C:\dir\lcdpanel.h
C:/dir/obj/lcd.obj: C:/CCStudio_v3.3/C6000/cgtools/include/string.h
C:/dir/obj/lcd.obj: C:/CCStudio_v3.3/C6000/cgtools/include/linkage.h

I decided to copy those files to a file with a .mf extension because I
was hoping that the makefile loader would handle things correctly.

Below is my rakefile. Notice that I am attempting to import the
dependency files within my :build_depend task. Unfortunately, I have no
idea what import does, and therefore I have no idea if my dependencies
are getting extracted in a way that rake understands.

###############################################################################
###############################################################################
###############################################################################
###############################################################################
require 'dsp_build_defs'

## The following require's are actually found in my dsp_build_defs.rb
file
require 'rake/clean'
require 'rake/loaders/makefile'
require 'fileutils

include DspRakeFileConstants

###############################################################################
# Create the list of source files, etc.
###############################################################################
#SRC_FILES = FileList["#{DSP_PROJ_ROOT}/**/*.#{C_EXT}"]
SRC_FILES = FileList["#{DSP_PROJ_ROOT}/**/*.#{C_EXT}"].reject {|f| f =~
/cropped/}
DEPEND_FILES = SRC_FILES.ext("#{MF_EXT}")
OBJ_FILES = SRC_FILES.pathmap("#{OBJ_DIR}/%n.#{OBJ_EXT}")

###############################################################################
# Task definitions
###############################################################################
desc 'Default task to build debug version of dsp software'
task :default => :debug_build do
OBJ_FILES.each do |srcfile|
#puts "source file: #{srcfile}"
end
DEPEND_FILES.each do |depfile|
#puts "source file: #{depfile}"
end
end

desc 'Debug_build task that depends on build_depend and build_obj'
task :debug_build => [:build_depend, :build_obj] do
puts "almost ready to link..."
end

desc 'Release_build task - not currently being called'
task :release_build => RELEASE_BLD

desc 'Build_obj task that depends on the object directory and the object
files'
task :build_obj => [OBJ_DIR, OBJ_FILES].flatten.uniq

desc 'Build_depend task that depends on the dependency files'
task :build_depend => DEPEND_FILES do
DEPEND_FILES.each do |dep_file|
puts "Importing #{dep_file}..."
import "#{dep_file}"
end
end

directory OBJ_DIR

##############################################################################
# Rule definitions
##############################################################################
rule ".#{DEPEND_EXT}" => ".#{C_EXT}" do |t|
puts "Building dependency file for #{t.source}"
sh "#{COMPILER} #{C_OPTS} #{C_DEPEND_OPTS} #{C_OBJ_DIR}
#{C_INC_PATHS} #{C_FLAGS} #{C_TARGET} #{t.source}"
end

rule ".#{OBJ_EXT}" => lambda{|objfile| find_source(objfile)} do |t|
puts "Building object file for #{t.source}"
sh "#{COMPILER} #{C_OPTS} #{C_OBJ_DIR} #{C_INC_PATHS} #{C_FLAGS}
#{C_TARGET} #{t.source}"
end

rule ".#{MF_EXT}" => ".#{DEPEND_EXT}" do |t|
puts "Copying #{t.source} to #{t.name}"
sh "cp #{t.source} #{t.name}"
end

rule ".#{BIN_EXT}" do |t|
puts "Linking..."
end
###############################################################################
###############################################################################
###############################################################################
###############################################################################


Thanks, in advance, for the help!


Best Regards,
Brock

 

[Patrick Doyle]

I am not yet a rake enthusiast -- but just this morning I decided,
"OK, it's probably time for me to start learning about rake". So,
with that disclaimer, I would suggest that #import doesn't do what you
think it does. Looking at the rake documentation
(http://rake.rubyforge.org/), it describes File#import as a procedure
that imports partial rakefiles. Not having played much with rake yet,
I'm somewhat surprised that it doesn't bomb when you import the code
composer dependency file. But, be that as it may, what you probably
want to do is to write a rake task that parses each line of the CCS
file and adds a rake dependency based on what it finds there.

Of course, that may fly in the face of whatever FAQ/tutorial/book
you're using, in which case, I would say follow the other source since
(s)he knows infinitely more about rake than I do .

Good luck!

--wpd

Hello:

I am new to rake, and I am stuck on a problem with incremental building.
My rakefile does what I would like, for the most part. If I touch a .c
file, the rakefile functions properly and only builds the necessary
files. However, if I touch the corresponding .h file, my rakefile
assumes everything is up-to-date. I am pretty certain that I am not
importing/loading my dependency files correctly. Please help.

I am using a dependency generation feature that is part of the DSP
compiler we are using. The output from the preprocessor creates files
with a .pp extension, formatted like so:

C:/dir/obj/lcd.obj: C:\dir\lcd.c
C:/dir/obj/lcd.obj: C:\dir\lcd.h
C:/dir/obj/lcd.obj: C:\dir\lcdpanel.h
C:/dir/obj/lcd.obj: C:/CCStudio_v3.3/C6000/cgtools/include/string.h
C:/dir/obj/lcd.obj: C:/CCStudio_v3.3/C6000/cgtools/include/linkage.h

I decided to copy those files to a file with a .mf extension because I
was hoping that the makefile loader would handle things correctly.

Below is my rakefile. Notice that I am attempting to import the
dependency files within my :build_depend task. Unfortunately, I have no
idea what import does, and therefore I have no idea if my dependencies
are getting extracted in a way that rake understands.

###############################################################################
###############################################################################
###############################################################################
###############################################################################
require 'dsp_build_defs'

## The following require's are actually found in my dsp_build_defs.rb
file
require 'rake/clean'
require 'rake/loaders/makefile'
require 'fileutils

include DspRakeFileConstants

###############################################################################
# Create the list of source files, etc.
###############################################################################
#SRC_FILES = FileList["#{DSP_PROJ_ROOT}/**/*.#{C_EXT}"]
SRC_FILES = FileList["#{DSP_PROJ_ROOT}/**/*.#{C_EXT}"].reject {|f| f =~
/cropped/}
DEPEND_FILES = SRC_FILES.ext("#{MF_EXT}")
OBJ_FILES = SRC_FILES.pathmap("#{OBJ_DIR}/%n.#{OBJ_EXT}")

###############################################################################
# Task definitions
###############################################################################
desc 'Default task to build debug version of dsp software'
task :default => :debug_build do
OBJ_FILES.each do |srcfile|
#puts "source file: #{srcfile}"
end
DEPEND_FILES.each do |depfile|
#puts "source file: #{depfile}"
end
end

desc 'Debug_build task that depends on build_depend and build_obj'
task :debug_build => [:build_depend, :build_obj] do
puts "almost ready to link..."
end

desc 'Release_build task - not currently being called'
task :release_build => RELEASE_BLD

desc 'Build_obj task that depends on the object directory and the object
files'
task :build_obj => [OBJ_DIR, OBJ_FILES].flatten.uniq

desc 'Build_depend task that depends on the dependency files'
task :build_depend => DEPEND_FILES do
DEPEND_FILES.each do |dep_file|
puts "Importing #{dep_file}..."
import "#{dep_file}"
end
end

directory OBJ_DIR

##############################################################################
# Rule definitions
##############################################################################
rule ".#{DEPEND_EXT}" => ".#{C_EXT}" do |t|
puts "Building dependency file for #{t.source}"
sh "#{COMPILER} #{C_OPTS} #{C_DEPEND_OPTS} #{C_OBJ_DIR}
#{C_INC_PATHS} #{C_FLAGS} #{C_TARGET} #{t.source}"
end

rule ".#{OBJ_EXT}" => lambda{|objfile| find_source(objfile)} do |t|
puts "Building object file for #{t.source}"
sh "#{COMPILER} #{C_OPTS} #{C_OBJ_DIR} #{C_INC_PATHS} #{C_FLAGS}
#{C_TARGET} #{t.source}"
end

rule ".#{MF_EXT}" => ".#{DEPEND_EXT}" do |t|
puts "Copying #{t.source} to #{t.name}"
sh "cp #{t.source} #{t.name}"
end

rule ".#{BIN_EXT}" do |t|
puts "Linking..."
end
###############################################################################
###############################################################################
###############################################################################
###############################################################################


Thanks, in advance, for the help!


Best Regards,
Brock

 

[Patrick Doyle]

On 2nd thought, perhaps this thread might help:

http://www.ruby-forum.com/topic/58444

Look for the post from Jim Weirich.

--wpd

Hello:

I am new to rake, and I am stuck on a problem with incremental building.
My rakefile does what I would like, for the most part. If I touch a .c
file, the rakefile functions properly and only builds the necessary
files. However, if I touch the corresponding .h file, my rakefile
assumes everything is up-to-date. I am pretty certain that I am not
importing/loading my dependency files correctly. Please help.

I am using a dependency generation feature that is part of the DSP
compiler we are using. The output from the preprocessor creates files
with a .pp extension, formatted like so:

C:/dir/obj/lcd.obj: C:\dir\lcd.c
C:/dir/obj/lcd.obj: C:\dir\lcd.h
C:/dir/obj/lcd.obj: C:\dir\lcdpanel.h
C:/dir/obj/lcd.obj: C:/CCStudio_v3.3/C6000/cgtools/include/string.h
C:/dir/obj/lcd.obj: C:/CCStudio_v3.3/C6000/cgtools/include/linkage.h

I decided to copy those files to a file with a .mf extension because I
was hoping that the makefile loader would handle things correctly.

Below is my rakefile. Notice that I am attempting to import the
dependency files within my :build_depend task. Unfortunately, I have no
idea what import does, and therefore I have no idea if my dependencies
are getting extracted in a way that rake understands.

###############################################################################
###############################################################################
###############################################################################
###############################################################################
require 'dsp_build_defs'

## The following require's are actually found in my dsp_build_defs.rb
file
require 'rake/clean'
require 'rake/loaders/makefile'
require 'fileutils

include DspRakeFileConstants

###############################################################################
# Create the list of source files, etc.
###############################################################################
#SRC_FILES = FileList["#{DSP_PROJ_ROOT}/**/*.#{C_EXT}"]
SRC_FILES = FileList["#{DSP_PROJ_ROOT}/**/*.#{C_EXT}"].reject {|f| f =~
/cropped/}
DEPEND_FILES = SRC_FILES.ext("#{MF_EXT}")
OBJ_FILES = SRC_FILES.pathmap("#{OBJ_DIR}/%n.#{OBJ_EXT}")

###############################################################################
# Task definitions
###############################################################################
desc 'Default task to build debug version of dsp software'
task :default => :debug_build do
OBJ_FILES.each do |srcfile|
#puts "source file: #{srcfile}"
end
DEPEND_FILES.each do |depfile|
#puts "source file: #{depfile}"
end
end

desc 'Debug_build task that depends on build_depend and build_obj'
task :debug_build => [:build_depend, :build_obj] do
puts "almost ready to link..."
end

desc 'Release_build task - not currently being called'
task :release_build => RELEASE_BLD

desc 'Build_obj task that depends on the object directory and the object
files'
task :build_obj => [OBJ_DIR, OBJ_FILES].flatten.uniq

desc 'Build_depend task that depends on the dependency files'
task :build_depend => DEPEND_FILES do
DEPEND_FILES.each do |dep_file|
puts "Importing #{dep_file}..."
import "#{dep_file}"
end
end

directory OBJ_DIR

##############################################################################
# Rule definitions
##############################################################################
rule ".#{DEPEND_EXT}" => ".#{C_EXT}" do |t|
puts "Building dependency file for #{t.source}"
sh "#{COMPILER} #{C_OPTS} #{C_DEPEND_OPTS} #{C_OBJ_DIR}
#{C_INC_PATHS} #{C_FLAGS} #{C_TARGET} #{t.source}"
end

rule ".#{OBJ_EXT}" => lambda{|objfile| find_source(objfile)} do |t|
puts "Building object file for #{t.source}"
sh "#{COMPILER} #{C_OPTS} #{C_OBJ_DIR} #{C_INC_PATHS} #{C_FLAGS}
#{C_TARGET} #{t.source}"
end

rule ".#{MF_EXT}" => ".#{DEPEND_EXT}" do |t|
puts "Copying #{t.source} to #{t.name}"
sh "cp #{t.source} #{t.name}"
end

rule ".#{BIN_EXT}" do |t|
puts "Linking..."
end
###############################################################################
###############################################################################
###############################################################################
###############################################################################


Thanks, in advance, for the help!


Best Regards,
Brock

 

[Brock Rycenga]

On 2nd thought, perhaps this thread might help:

http://www.ruby-forum.com/topic/58444

Look for the post from Jim Weirich.

--wpd

Hello Jim:

Actually, I am very familiar with that thread. Been studying that all
week. One of the differences is that thread references makedepend -
which output format I am not familiar. I haven't been successful find a
sample of makedepends output. So I am not sure if my preprocessor is
even outputting in a format that is accurately parsed via
rake/loaders/makefile.


Brock

 

[Patrick Doyle]

Hi Brock,
I think I may have just found the source of your problem:

If you look at rake/loaders/makefile.rb (which, on my Windoze system
is somewhere near
ruby/lib/ruby/gems/1.8/gems/rake-0.8.1/lib/rake/loaders/makefile.rb),
you will see something that looks like:

# Process one logical line of makefile data.
def process_line(line)
file_task, args = line.split(':')
return if args.nil?
dependents = args.split
file file_task => dependents
end

This will not handle C:\blah filenames correctly -- it split's them at
the first colon (":").

You could use that file as a source of inspiration to write your own
Rake::MakefileLoader#load method that correctly parses CCS generated
dependency files (perhaps splitting the file task from its dependents
after the first ".obj:", perhaps recognizing/depending on the hope
that/ CCS only places one dependency per line, etc...).

I'm not yet proficient enough in Ruby, and especially regular
expression matching to just rattle this off (but I know resources are
online to fill in all the bits I'm missing, or have gotten just plain
wrong), but you could probably replace the line of code that reads

require 'rake/loaders/makefile'

with something like (freely inspired by rake/loaders/makefile.rb)

module Rake

# Makefile loader to be used with the import file loader.
class MakefileLoader

# Load the makefile dependencies in +fn+.
def load(fn)
buffer = ''
open(fn) do |mf|
mf.each do |line|
# (I'm certain I've got the regular expression thing wrong here)
matches = line.match("(.*\.obj):\w*(.*)$")
file_task = matches[0]
dependency = matches[1]
file file_task => dependents
end
end
end
process_line(buffer) if buffer != ''
end

# Install the handler
Rake.application.add_loader('mf', MakefileLoader.new)
end

Does that make sense? (Once you figure out the right way to do
pattern matching in Ruby)

--wpd

 

[Brock Rycenga]

Hi Brock,
I think I may have just found the source of your problem:

If you look at rake/loaders/makefile.rb (which, on my Windoze system
is somewhere near
ruby/lib/ruby/gems/1.8/gems/rake-0.8.1/lib/rake/loaders/makefile.rb),
you will see something that looks like:

# Process one logical line of makefile data.
def process_line(line)
file_task, args = line.split(':')
return if args.nil?
dependents = args.split
file file_task => dependents
end

This will not handle C:\blah filenames correctly -- it split's them at
the first colon (":").

You could use that file as a source of inspiration to write your own
Rake::MakefileLoader#load method that correctly parses CCS generated
dependency files (perhaps splitting the file task from its dependents
after the first ".obj:", perhaps recognizing/depending on the hope
that/ CCS only places one dependency per line, etc...).

I'm not yet proficient enough in Ruby, and especially regular
expression matching to just rattle this off (but I know resources are
online to fill in all the bits I'm missing, or have gotten just plain
wrong), but you could probably replace the line of code that reads

----- snip! -------

Does that make sense? (Once you figure out the right way to do
pattern matching in Ruby)

--wpd

Patrick:

You da man!

You "inspired" me to write my own CCS-generated dependency file parser.
It was actually quite simple - thanks to your hints above. I am finally
moving ahead again. In case you, or anyone else, is interested, here is
what I modified from my original rakefile (added 2 def's to process the
dependency files and modified the :build_depend task to call).

###############################################################################
desc 'Build_depend task that depends on the dependency files'
task :build_depend => DEPEND_FILES do
puts "Importing file dependencies..."
DEPEND_FILES.each do |dep_file|
process_dpend_file(dep_file)
end
end

def process_dpend_line(line)
objfile, prereq = line.split(': ')
return if prereq.nil?
file objfile => prereq
end

def process_dpend_file(dfile)
open(dfile) do |file_lines|
file_lines.each do |line|
process_dpend_line(line.chomp)
end
end
end
###############################################################################


Thanks again,
Brock