How to edit a file without loading it all into memory

Discussion in 'Ruby' started by Jonathan Dobbie, Nov 9, 2007.

  1. I need to change the texture paths in maya files. The lines are easy to
    find with grep, and I'm able to create the new line easily enough, but
    how can I do this without holding the entire file into memory? The
    files can be hundreds of megabytes.

    Thanks.
    --
    Posted via http://www.ruby-forum.com/.
     
    Jonathan Dobbie, Nov 9, 2007
    #1
    1. Advertising

  2. newer versions of sed can replace text in place in files.

    there's also perl's

    perl -pi -e 's/to change this/to change that/' filename


    On Sat, 2007-11-10 at 05:51 +0900, Jonathan Dobbie wrote:
    > I need to change the texture paths in maya files. The lines are easy to
    > find with grep, and I'm able to create the new line easily enough, but
    > how can I do this without holding the entire file into memory? The
    > files can be hundreds of megabytes.
    >
    > Thanks.
     
    Reid Thompson, Nov 9, 2007
    #2
    1. Advertising

  3. On 09.11.2007 21:51, Jonathan Dobbie wrote:
    > I need to change the texture paths in maya files. The lines are easy to
    > find with grep, and I'm able to create the new line easily enough, but
    > how can I do this without holding the entire file into memory? The
    > files can be hundreds of megabytes.


    As you said: do it line by line. This is the usual approach to this
    problem. You might want to have a look at Ruby's command line options.
    Basically you can do something like this for an inplace edit:

    ruby -ni.bak -e 'your replacement code'

    Kind regards

    robert
     
    Robert Klemme, Nov 9, 2007
    #3
  4. Jonathan Dobbie

    7stud -- Guest

    Jonathan Dobbie wrote:
    > The lines are easy to
    > find with grep, and I'm able to create the new line easily enough, but
    > how can I do this without holding the entire file into memory? The
    > files can be hundreds of megabytes.
    >


    File.foreach("data.txt") do |line|
    #look for something in line
    end

    --
    Posted via http://www.ruby-forum.com/.
     
    7stud --, Nov 9, 2007
    #4
  5. 7stud -- wrote:
    > Jonathan Dobbie wrote:
    >> The lines are easy to
    >> find with grep, and I'm able to create the new line easily enough, but
    >> how can I do this without holding the entire file into memory? The
    >> files can be hundreds of megabytes.
    >>

    >
    > File.foreach("data.txt") do |line|
    > #look for something in line
    > end


    Here is what I originally wrote, and what I just tried. Am I missing
    something obvious? The first one duplicates the lines, and the second
    one does nothing. (doesn't save the changes)

    def change_texture_paths
    File.open(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |file|
    file.grep( /setAttr ".fileTextureName" -type "string"/ ) do |line|
    texture_file_path = line[52..-2]
    texture_file_name = texture_file_path.split("\/").last
    line = " setAttr \".fileTextureName\" -type \"string\"
    \"" + ENV['HOME'] + "/include/" + texture_file_name + "\n"
    file.print line
    end
    end


    File.foreach(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |line|
    if (line =~ /setAttr ".fileTextureName" -type "string"/ )
    texture_file_path = line[52..-2]
    texture_file_name = texture_file_path.split("\/").last
    line = " setAttr \".fileTextureName\" -type \"string\"
    \"" + ENV['HOME'] + "/include/" + texture_file_name + "\n"
    file.print line
    end
    end
    end
    --
    Posted via http://www.ruby-forum.com/.
     
    Jonathan Dobbie, Nov 9, 2007
    #5
  6. Jonathan Dobbie

    Kyle Schmitt Guest

    Err.

    Well only one of these textures is open at once right?
    If this box is actually capable of running maya well, it's sure to
    have more than a few hundred megs of free ram (when maya isn't running
    of course ;).

    My guess is that you'll get a hefty performance hit for parsing the
    files line by line. The box has memory to spare, so why not just use
    it?

    Of course if there was a way to memap the file under ruby that would
    probably be the best...but I dunno how to do that.

    --Kyle
     
    Kyle Schmitt, Nov 9, 2007
    #6
  7. Jonathan Dobbie

    Kyle Schmitt Guest

    mem map. not memap..but you get the idea.
     
    Kyle Schmitt, Nov 9, 2007
    #7
  8. Jonathan Dobbie

    7stud -- Guest

    Jonathan Dobbie wrote:
    > 7stud -- wrote:
    >> Jonathan Dobbie wrote:
    >>> The lines are easy to
    >>> find with grep, and I'm able to create the new line easily enough, but
    >>> how can I do this without holding the entire file into memory? The
    >>> files can be hundreds of megabytes.
    >>>

    >>
    >> File.foreach("data.txt") do |line|
    >> #look for something in line
    >> end

    >
    > Here is what I originally wrote, and what I just tried. Am I missing
    > something obvious? The first one duplicates the lines, and the second
    > one does nothing. (doesn't save the changes)
    >


    1) Are you under the impression that every method in ruby has the same
    parameters?

    2) When you see an error message, do you just ignore it?

    --
    Posted via http://www.ruby-forum.com/.
     
    7stud --, Nov 10, 2007
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Schultz
    Replies:
    3
    Views:
    595
    =?Utf-8?B?QmlsbCBCb3Jn?=
    Feb 14, 2005
  2. Robert Zurer
    Replies:
    0
    Views:
    480
    Robert Zurer
    Jun 28, 2005
  3. Christian Hiller
    Replies:
    7
    Views:
    1,439
    Andrew
    Oct 9, 2003
  4. Phil Johnson
    Replies:
    3
    Views:
    443
    Phil Johnson
    Apr 25, 2008
  5. Raymond O'Connor
    Replies:
    2
    Views:
    470
    Raymond O'Connor
    Aug 28, 2009
Loading...

Share This Page