FasterCSV, Gruff and Large Data Import

Discussion in 'Ruby' started by Dan Webb [dbw], Mar 10, 2009.

  1. Hi all,

    =20

    I've got a CSV file which contains 18k items that I need to import to
    graph. I'm using FasterCSV and Gruff to import and graph the data.=20

    =20

    Obviously I'm going about this wrong as it's overloading the Array
    class. Is there another way around this?

    http://pastie.org/412741

    =20

    =20

    Kind Regards,

    Dan

    =20

    =20
     
    Dan Webb [dbw], Mar 10, 2009
    #1
    1. Advertising

  2. Dan Webb [dbw]

    James Gray Guest

    On Mar 10, 2009, at 9:03 AM, Dan Webb [dbw] wrote:

    > I've got a CSV file which contains 18k items that I need to import to
    > graph. I'm using FasterCSV and Gruff to import and graph the data.
    >
    >
    >
    > Obviously I'm going about this wrong as it's overloading the Array
    > class. Is there another way around this?
    >
    > http://pastie.org/412741


    Well, I think you probably need to trim the data down a bit. Could
    you maybe build up some averages over time periods and feed just those
    averages to Gruff?

    James Edward Gray II
     
    James Gray, Mar 10, 2009
    #2
    1. Advertising

  3. On Mar 10, 2009, at 10:36 AM, James Gray wrote:

    > On Mar 10, 2009, at 9:03 AM, Dan Webb [dbw] wrote:
    >
    >> I've got a CSV file which contains 18k items that I need to import to
    >> graph. I'm using FasterCSV and Gruff to import and graph the data.
    >>
    >>
    >>
    >> Obviously I'm going about this wrong as it's overloading the Array
    >> class. Is there another way around this?
    >>
    >> http://pastie.org/412741

    >
    > Well, I think you probably need to trim the data down a bit. Could =20=


    > you maybe build up some averages over time periods and feed just =20
    > those averages to Gruff?
    >
    > James Edward Gray II


    What kind of graph? How big is the graph? Think about what kind of =20
    resolution is meaningful =96 the default is only 800 pixels wide and you =
    =20
    can't graph sub-pixel values ;-)

    -Rob

    Rob Biedenharn http://agileconsultingllc.com
     
    Rob Biedenharn, Mar 10, 2009
    #3
  4. Not really, well I could but I'm trying to compare it with doing it in a
    similar program such as GNU Plot, plus there are sometimes
    irregularities in the data which I'd like to show if possible.=20

    On a second look the error I'm getting is a gruff related one.=20

    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:943:in
    `larger_than
    _max?': undefined method `>' for ["1\t8877"]:Array (NoMethodError)
    from
    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:453:in
    `data'
    from
    c:/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:15
    70:in `each_with_index'
    from
    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:443:in
    `each'
    from
    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:443:in
    `each_with_index'
    from
    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:443:in
    `data'
    from main.rb:20

    line 20's where the array gets pushed into the graph.=20


    Dan


    -----Original Message-----
    From: James Gray [mailto:]=20
    Sent: 10 March 2009 14:37
    To: ruby-talk ML
    Subject: Re: FasterCSV, Gruff and Large Data Import

    On Mar 10, 2009, at 9:03 AM, Dan Webb [dbw] wrote:

    > I've got a CSV file which contains 18k items that I need to import to
    > graph. I'm using FasterCSV and Gruff to import and graph the data.
    >
    >
    >
    > Obviously I'm going about this wrong as it's overloading the Array
    > class. Is there another way around this?
    >
    > http://pastie.org/412741


    Well, I think you probably need to trim the data down a bit. Could =20
    you maybe build up some averages over time periods and feed just those =20
    averages to Gruff?

    James Edward Gray II
     
    Dan Webb [dbw], Mar 10, 2009
    #4
  5. Dan Webb [dbw]

    James Coglan Guest

    [Note: parts of this message were removed to make it a legal post.]

    > On a second look the error I'm getting is a gruff related one.
    >
    > c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:943:in
    > `larger_than
    > _max?': undefined method `>' for ["1\t8877"]:Array (NoMethodError)
    > from
    > c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:453:in
    > `data'
    > from
    > c:/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:15
    > 70:in `each_with_index'
    > from
    > c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:443:in
    > `each'
    > from
    > c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:443:in
    > `each_with_index'
    > from
    > c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:443:in
    > `data'
    > from main.rb:20
    >
    > line 20's where the array gets pushed into the graph.




    Can you show us your code? Looks like your array of data may itself contain
    arrays -- have you tried flattening the array before giving it to Gruff?
     
    James Coglan, Mar 10, 2009
    #5
  6. Dan Webb [dbw]

    James Gray Guest

    On Mar 10, 2009, at 9:57 AM, Dan Webb [dbw] wrote:

    > On a second look the error I'm getting is a gruff related one.
    >
    > c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:943:in
    > `larger_than
    > _max?': undefined method `>' for ["1\t8877"]:Array (NoMethodError)
    > from


    Gruff expects an Array of values to graph. It looks like you might be
    feeding it an Array of Arrays and even those values look fishy.

    I recommend simplifying the problem. First, try just reading the
    first five lines of your data file and graphing just that.

    When you have that working, you can expand the data and see how things
    go.

    James Edward Gray II
     
    James Gray, Mar 10, 2009
    #6
  7. Hi again,

    As suggested I've changed tact. It now looks like gruff doesn't like the
    array created by FasterCSV, but I don't know why!

    If I create an array manually then feed it into gruff, i.e.=20

    a =3D [1,2,3,4,5]
    g.data("X", a)

    it works absolutely fine. However if I've put the same data into a csv
    file, then read it in using either:=20

    a =3D fastercsv.read("r.csv")

    or=20

    FasterCSV.foreach("r.csv") do |row|=20
    a << row
    end

    If I puts all three they all print out as
    1
    2
    3
    4
    5

    And both errors given by the array when I use fasterCSV are:

    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:943:in
    `larger_than
    _max?': undefined method `>' for ["1", "2", "3", "4", "5"]:Array
    (NoMethodError)

    from
    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:453:in
    `data'
    from
    c:/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.4.0/lib/faster_csv.rb:15
    70:in `each_with_index'
    from
    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:443:in
    `each'
    from
    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:443:in
    `each_with_index'
    from
    c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:443:in
    `data'
    from main.rb:27

    am I missing something spectacular?
    Thanks,
    Dan

    -----Original Message-----
    From: James Gray [mailto:]=20
    Sent: 10 March 2009 15:06
    To: ruby-talk ML
    Subject: Re: FasterCSV, Gruff and Large Data Import

    On Mar 10, 2009, at 9:57 AM, Dan Webb [dbw] wrote:

    > On a second look the error I'm getting is a gruff related one.
    >
    > c:/ruby/lib/ruby/gems/1.8/gems/gruff-0.3.4/lib/gruff/base.rb:943:in
    > `larger_than
    > _max?': undefined method `>' for ["1\t8877"]:Array (NoMethodError)
    > from


    Gruff expects an Array of values to graph. It looks like you might be =20
    feeding it an Array of Arrays and even those values look fishy.

    I recommend simplifying the problem. First, try just reading the =20
    first five lines of your data file and graphing just that.

    When you have that working, you can expand the data and see how things =20
    go.

    James Edward Gray II
     
    Dan Webb [dbw], Mar 11, 2009
    #7
  8. Dan Webb [dbw]

    James Coglan Guest

    [Note: parts of this message were removed to make it a legal post.]

    > a = [1,2,3,4,5]
    > g.data("X", a)
    >
    > it works absolutely fine. However if I've put the same data into a csv
    > file, then read it in using either:
    >
    > a = fastercsv.read("r.csv")
    >
    > or
    >
    > FasterCSV.foreach("r.csv") do |row|
    > a << row
    > end




    Looks like you're feeding the whole CSV table into Gruff all at once. Try
    something like this:

    idx = 1
    FasterCSV.foreach("r.csv") do |row|
    g.data("Series #{idx}", row)
    idx += 1
    end

    --
    James Coglan
    http://github.com/jcoglan
     
    James Coglan, Mar 11, 2009
    #8
  9. However if I feed the fastercsv data into an array then into gruff it
    _should_ work, as feeding a whole array that I've written out by hand
    into gruff works. If you see my logic here, it should work, no?

    Dan


    -----Original Message-----
    From: James Coglan [mailto:]=20
    Sent: 11 March 2009 09:32
    To: ruby-talk ML
    Subject: Re: FasterCSV, Gruff and Large Data Import

    > a =3D [1,2,3,4,5]
    > g.data("X", a)
    >
    > it works absolutely fine. However if I've put the same data into a csv
    > file, then read it in using either:
    >
    > a =3D fastercsv.read("r.csv")
    >
    > or
    >
    > FasterCSV.foreach("r.csv") do |row|
    > a << row
    > end




    Looks like you're feeding the whole CSV table into Gruff all at once.
    Try
    something like this:

    idx =3D 1
    FasterCSV.foreach("r.csv") do |row|
    g.data("Series #{idx}", row)
    idx +=3D 1
    end

    --
    James Coglan
    http://github.com/jcoglan
     
    Dan Webb [dbw], Mar 11, 2009
    #9
  10. Dan Webb [dbw]

    James Coglan Guest

    [Note: parts of this message were removed to make it a legal post.]

    2009/3/11 Dan Webb [dbw] <>

    > However if I feed the fastercsv data into an array then into gruff it
    > _should_ work, as feeding a whole array that I've written out by hand
    > into gruff works. If you see my logic here, it should work, no?



    No, because reading a CSV produces a table, i.e. an array of arrays,
    something like:

    FasterCSV.read("r.csv") #=> [[1,2,3,4], [5,6,7,8], ...]

    Gruff expects a flat array of data for each data series, so you need to loop
    over each row in the table and feed them to Gruff one by one.

    --
    James Coglan
    http://github.com/jcoglan
     
    James Coglan, Mar 11, 2009
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ams Lone

    NameError - Scruffy and Gruff -

    Ams Lone, Aug 12, 2007, in forum: Ruby
    Replies:
    3
    Views:
    141
    Stefan Rusterholz
    Aug 12, 2007
  2. Horacio Sanson

    mrplot and Gruff japanese

    Horacio Sanson, Jan 21, 2008, in forum: Ruby
    Replies:
    0
    Views:
    107
    Horacio Sanson
    Jan 21, 2008
  3. Damaris Fuentes

    Gruff and SVG

    Damaris Fuentes, Feb 9, 2008, in forum: Ruby
    Replies:
    0
    Views:
    121
    Damaris Fuentes
    Feb 9, 2008
  4. Trick Nick

    Gruff and labeling

    Trick Nick, Nov 25, 2008, in forum: Ruby
    Replies:
    1
    Views:
    105
    Trick Nick
    Nov 25, 2008
  5. Nick Gohn
    Replies:
    0
    Views:
    156
    Nick Gohn
    Dec 12, 2009
Loading...

Share This Page