How to edit a file without loading it all into memory

[Jonathan Dobbie]

I need to change the texture paths in maya files. The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

Thanks.

 

[Reid Thompson]

newer versions of sed can replace text in place in files.

there's also perl's

perl -pi -e 's/to change this/to change that/' filename

 

[Robert Klemme]

I need to change the texture paths in maya files. The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

As you said: do it line by line. This is the usual approach to this
problem. You might want to have a look at Ruby's command line options.
Basically you can do something like this for an inplace edit:

ruby -ni.bak -e 'your replacement code'

Kind regards

robert

 

[7stud --]

The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

File.foreach("data.txt") do |line|
#look for something in line
end

 

[Jonathan Dobbie]

File.foreach("data.txt") do |line|
#look for something in line
end

Here is what I originally wrote, and what I just tried. Am I missing
something obvious? The first one duplicates the lines, and the second
one does nothing. (doesn't save the changes)

def change_texture_paths
File.open(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |file|
file.grep( /setAttr ".fileTextureName" -type "string"/ ) do |line|
texture_file_path = line[52..-2]
texture_file_name = texture_file_path.split("\/").last
line = " setAttr \".fileTextureName\" -type \"string\"
\"" + ENV['HOME'] + "/include/" + texture_file_name + "\n"
file.print line
end
end


File.foreach(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |line|
if (line =~ /setAttr ".fileTextureName" -type "string"/ )
texture_file_path = line[52..-2]
texture_file_name = texture_file_path.split("\/").last
line = " setAttr \".fileTextureName\" -type \"string\"
\"" + ENV['HOME'] + "/include/" + texture_file_name + "\n"
file.print line
end
end
end

 

[Kyle Schmitt]

Err.

Well only one of these textures is open at once right?
If this box is actually capable of running maya well, it's sure to
have more than a few hundred megs of free ram (when maya isn't running
of course .

My guess is that you'll get a hefty performance hit for parsing the
files line by line. The box has memory to spare, so why not just use
it?

Of course if there was a way to memap the file under ruby that would
probably be the best...but I dunno how to do that.

--Kyle

 

[Kyle Schmitt]

mem map. not memap..but you get the idea.

 

[7stud --]

Here is what I originally wrote, and what I just tried. Am I missing
something obvious? The first one duplicates the lines, and the second
one does nothing. (doesn't save the changes)

1) Are you under the impression that every method in ruby has the same
parameters?

2) When you see an error message, do you just ignore it?