Appending to a CSV file

Discussion in 'Ruby' started by Bil Kleb, Jul 12, 2006.

  1. Bil Kleb

    Bil Kleb Guest

    Just got a call from a telecommuting colleague
    who's using Ruby to automate a bunch of CEV[1]
    CFD[2] simulations.

    She says that she can't figure out how to /append/
    to a CSV file using Ruby's standard library.

    Does anyone have the magic recipe, or should
    she be using FasterCSV instead?

    Thanks,
    --
    Bil
    http://fun3d.larc.nasa.gov

    [1] CEV = Crew Exploration Vehicle
    [2] CFD = Computational Fluid Dynamics
     
    Bil Kleb, Jul 12, 2006
    #1
    1. Advertising

  2. Bil Kleb

    ChrisH Guest

    Bil Kleb wrote:
    ....
    > She says that she can't figure out how to /append/
    > to a CSV file using Ruby's standard library.

    ....

    Wouldn't it just be a matter of opening the file in append mode?

    cheers
    Chris
     
    ChrisH, Jul 12, 2006
    #2
    1. Advertising

  3. Bil Kleb

    Chris Hulan Guest

    ChrisH wrote:
    > Bil Kleb wrote:
    > ...
    >> She says that she can't figure out how to /append/
    >> to a CSV file using Ruby's standard library.

    > ...
    >
    > Wouldn't it just be a matter of opening the file in append mode?
    >
    > cheers
    > Chris


    Ok, note to self "read docs before commenting"

    it seems CSV only supports 'r','rb','w','wb' for open.
    FasterCSV delegates to an IO object and supports all IO's modes.
    So it would be the winner

    Cheers

    --
    Posted via http://www.ruby-forum.com/.
     
    Chris Hulan, Jul 12, 2006
    #3
  4. Bil Kleb

    Bil Kleb Guest

    ChrisH wrote:
    > Bil Kleb wrote:
    > ...
    >> She says that she can't figure out how to /append/
    >> to a CSV file using Ruby's standard library.

    > ...
    >
    > Wouldn't it just be a matter of opening the file in append mode?


    It would, but unfortunately CSV doesn't support 'a'
    or 'a+', viz, lib/ruby/1.8/csv.rb,

    83 def CSV.open(path, mode, fs = nil, rs = nil, &block)
    84 if mode == 'r' or mode == 'rb'
    85 open_reader(path, mode, fs, rs, &block)
    86 elsif mode == 'w' or mode == 'wb'
    87 open_writer(path, mode, fs, rs, &block)
    88 else
    89 raise ArgumentError.new("'mode' must be 'r', 'rb', 'w', or 'wb'")
    90 end
    91 end

    Later,
    --
    Bil
    http://fun3d.larc.nasa.gov
     
    Bil Kleb, Jul 12, 2006
    #4
  5. On Jul 12, 2006, at 1:55 PM, Chris Hulan wrote:

    > it seems CSV only supports 'r','rb','w','wb' for open.
    > FasterCSV delegates to an IO object and supports all IO's modes.
    > So it would be the winner


    Interestingly, this design choice in FasterCSV has been questioned in
    the past. I just keep liking it more and more though as issues like
    this arise. :)

    James Edward Gray II
     
    James Edward Gray II, Jul 12, 2006
    #5
  6. Bil Kleb

    Chris Hulan Guest

    James Gray wrote:
    > On Jul 12, 2006, at 1:55 PM, Chris Hulan wrote:
    >
    >> it seems CSV only supports 'r','rb','w','wb' for open.
    >> FasterCSV delegates to an IO object and supports all IO's modes.
    >> So it would be the winner

    >
    > Interestingly, this design choice in FasterCSV has been questioned in
    > the past. I just keep liking it more and more though as issues like
    > this arise. :)
    >
    > James Edward Gray II


    It does seem a bit odd that CSV limits the modes.

    On the other hand, the desired result can be achieved by creating a
    new file, copy existing data, add the new data, delete (or rename to be
    safe)
    the original and rename the new file to the old file name.

    I think supporting the append mode is easier all round...

    cheers
    Chris

    --
    Posted via http://www.ruby-forum.com/.
     
    Chris Hulan, Jul 12, 2006
    #6
  7. Bil Kleb

    Hal Fulton Guest

    Bil Kleb wrote:
    > ChrisH wrote:
    >
    >> Bil Kleb wrote:
    >> ...
    >>
    >>> She says that she can't figure out how to /append/
    >>> to a CSV file using Ruby's standard library.

    >>
    >> ...
    >>
    >> Wouldn't it just be a matter of opening the file in append mode?

    >
    >
    > It would, but unfortunately CSV doesn't support 'a'
    > or 'a+', viz, lib/ruby/1.8/csv.rb,


    Well, darn.

    All I can think of offhand is 1) write the new stuff to a new file
    and then 2) use fileutils to append the new file onto the old one.

    Disclaimers:

    1. I'm not sure fileutils has a good way to do that, but
    if it doesn't, there's always system("cat newfile >>oldfile").

    2. Obviously be sure the old file is closed before trying to
    so anything funny with it.


    Hal
     
    Hal Fulton, Jul 13, 2006
    #7
  8. I'm not very familiar with the CSV lib, but if the heart of the matter
    is appending to a CSV file, you hardly need a full-on CSV lib for
    something like that.

    File.open(my_csv) {|f| f.puts new_record} # think it oughta be as
    simple as that
    File.open(my_csv) {|f| f.puts *new_records} # puts can take a list, too

    Bil Kleb wrote:
    > ChrisH wrote:
    > > Bil Kleb wrote:
    > > ...
    > >> She says that she can't figure out how to /append/
    > >> to a CSV file using Ruby's standard library.

    > > ...
    > >
    > > Wouldn't it just be a matter of opening the file in append mode?

    >
    > It would, but unfortunately CSV doesn't support 'a'
    > or 'a+', viz, lib/ruby/1.8/csv.rb,
    >
    > 83 def CSV.open(path, mode, fs = nil, rs = nil, &block)
    > 84 if mode == 'r' or mode == 'rb'
    > 85 open_reader(path, mode, fs, rs, &block)
    > 86 elsif mode == 'w' or mode == 'wb'
    > 87 open_writer(path, mode, fs, rs, &block)
    > 88 else
    > 89 raise ArgumentError.new("'mode' must be 'r', 'rb', 'w', or 'wb'")
    > 90 end
    > 91 end
    >
    > Later,
    > --
    > Bil
    > http://fun3d.larc.nasa.gov
     
    Seth Thomas Rasmussen, Jul 13, 2006
    #8
  9. Sorry about my last post sort of skipping over the whole matter of the
    write mode.. you're smarter than me, though, I'm sure you'll be fine.
    ;-p


    Bil Kleb wrote:
    > ChrisH wrote:
    > > Bil Kleb wrote:
    > > ...
    > >> She says that she can't figure out how to /append/
    > >> to a CSV file using Ruby's standard library.

    > > ...
    > >
    > > Wouldn't it just be a matter of opening the file in append mode?

    >
    > It would, but unfortunately CSV doesn't support 'a'
    > or 'a+', viz, lib/ruby/1.8/csv.rb,
    >
    > 83 def CSV.open(path, mode, fs = nil, rs = nil, &block)
    > 84 if mode == 'r' or mode == 'rb'
    > 85 open_reader(path, mode, fs, rs, &block)
    > 86 elsif mode == 'w' or mode == 'wb'
    > 87 open_writer(path, mode, fs, rs, &block)
    > 88 else
    > 89 raise ArgumentError.new("'mode' must be 'r', 'rb', 'w', or 'wb'")
    > 90 end
    > 91 end
    >
    > Later,
    > --
    > Bil
    > http://fun3d.larc.nasa.gov
     
    Seth Thomas Rasmussen, Jul 13, 2006
    #9
  10. Bil Kleb

    gregarican Guest

    I haven't used the CSV module before either, but looking into it this
    test script I came up with parses a CSV source file, writing it to a
    new array. Then you can append to that new array and write the results
    back to the original CSV file. It's not pretty and might not be the
    most Rubyish way of getting the job done but it worked for me using
    Ruby 1.8.2 on Win32. Try it out and let me know...

    require 'csv'

    # test.csv consists of three comma delimited fields, something like:
    #
    # 1,"Greg","Admin"
    # 2,"Joe","User"
    # 3,"Jim","Admin"

    new_array = []
    temp_array = []

    reader = CSV.open('test.csv', 'r') do |row|
    (0...row.nitems).each do |item|
    temp_array.push(row[item].data)
    end
    new_array << temp_array
    temp_array = []
    end

    new_items = [4,"Jane","User"]
    new_array << new_items

    CSV.open('test.csv', 'w') do |writer|
    new_array.each do |row|
    writer << row
    end
    end


    Bil Kleb wrote:
    > ChrisH wrote:
    > > Bil Kleb wrote:
    > > ...
    > >> She says that she can't figure out how to /append/
    > >> to a CSV file using Ruby's standard library.

    > > ...
    > >
    > > Wouldn't it just be a matter of opening the file in append mode?

    >
    > It would, but unfortunately CSV doesn't support 'a'
    > or 'a+', viz, lib/ruby/1.8/csv.rb,
    >
    > 83 def CSV.open(path, mode, fs = nil, rs = nil, &block)
    > 84 if mode == 'r' or mode == 'rb'
    > 85 open_reader(path, mode, fs, rs, &block)
    > 86 elsif mode == 'w' or mode == 'wb'
    > 87 open_writer(path, mode, fs, rs, &block)
    > 88 else
    > 89 raise ArgumentError.new("'mode' must be 'r', 'rb', 'w', or 'wb'")
    > 90 end
    > 91 end
    >
    > Later,
    > --
    > Bil
    > http://fun3d.larc.nasa.gov
     
    gregarican, Jul 13, 2006
    #10
  11. Bil Kleb

    ChrisH Guest

    Bil Kleb wrote:
    ....
    > She says that she can't figure out how to /append/
    > to a CSV file using Ruby's standard library.

    ....

    Not sure if you read ruby-forum (http://www.ruby-forum.com/)
    but a response ther indicates that you can do this via the CSV::Writer:

    File.open('csvout.csv', 'a'){ |outfile|
    CSV::Writer.generate(outfile) do |csv|
    csv << ['c1', nil, '', '"', "\r\n", 'c2']
    end
    }


    Cheers
    Chris
     
    ChrisH, Jul 13, 2006
    #11
  12. Bil Kleb

    bbiker Guest

    gregarican wrote:
    > I can't speak for the OP, but I am assuming that they were using the
    > CSV module because there might have been some complications in
    > processing the data source. And perhaps some of the CSV module's
    > methods helped streamline things so they didn't have to reinvent the
    > wheel.
    >
    > No matter if you are using the CSV module or rolling your own way of
    > processing a data source file, you would have to parse through the
    > existing data in some block form and then append to the end of the
    > stream/array/whatever to accomplish what the OP was looking to do.
    >
    > If you post a code snippet of your way to accomplishing this without
    > using the CSV module I would think that the basic concept and workflow
    > of the script would be similar...
    >


    As I said I am a newby to Ruby but not to programming in general.

    Now to me a csv file is simply a flatfile database stored in a text
    file. The only specifications are that fields have a separator
    (normally a comma) and the records have a separator (normally a
    newline). In addition neither of these separators can appear within a
    field.

    So to me, I only need to read a record and write a record. Once a
    record is read, how I slice or dice the record does not require a csv
    module that provides unique functions to process the record. Remember a
    csv record is a string, period. There are no other structural
    requirements.

    Now it is up to me to verify the sanity of the records and believe me
    no module can be written to cover all possiblities since the records
    are free-form and can have any format what so ever.

    Usually, I open the csv file, read in the records into an array, close
    the file and then process the records as appropriately. Should I need
    to write those records back I either re-open the csv file in write mode
    (which clobbers the original file) or open a newfile in write mode.
    Write out each record. If I want to keep the original file as a back
    up, I can rename it and give the new file the old name. If my program
    generates new records that need to be added to an existing file, I open
    the file in append mode and writeout the record(s).

    I normally do not use csv files as a permanent database. They usually
    are "pass-through" files..an external program may write it outputs in a
    csv file which my program processes and/or my program may store its
    outputs in a csv file for another program to use.
     
    bbiker, Jul 16, 2006
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michal Mikolajczyk
    Replies:
    0
    Views:
    659
    Michal Mikolajczyk
    Feb 13, 2004
  2. Skip Montanaro
    Replies:
    0
    Views:
    722
    Skip Montanaro
    Feb 13, 2004
  3. Tintin92
    Replies:
    1
    Views:
    1,716
    Andrew Thompson
    Feb 14, 2007
  4. jliu66
    Replies:
    0
    Views:
    517
    jliu66
    Oct 19, 2007
  5. Li Chen
    Replies:
    18
    Views:
    669
    Azmi Farih
    Mar 23, 2010
Loading...

Share This Page