High level csv reader

[George Sakkis]

It occured to me that most times I read a csv file, I'm often doing
from scratch things like assigning labels to columns, mapping fields to
the appropriate type, ignoring some fields, changing their order, etc.
Before I go on and reinvent the wheel, is there a generic high level
wrapper around csv.reader that does all this ?

Thanks,
George

 

[James Stroud]

It occured to me that most times I read a csv file, I'm often doing
from scratch things like assigning labels to columns, mapping fields to
the appropriate type, ignoring some fields, changing their order, etc.
Before I go on and reinvent the wheel, is there a generic high level
wrapper around csv.reader that does all this ?

Thanks,
George

There is a csv in the standard library. Though many of us don't mind
answering questions like this, you can get a lot of answers quicker by
(1) looking to see what's in the standard library and (2) using google.

http://docs.python.org/lib/module-csv.html

James

 

[Fredrik Lundh]

>
There is a csv in the standard library.

I'm not sure the "csv" module qualifies as a high-level wrapper around
itself, though.

</F>

 

[John Machin]

There is a csv in the standard library. Though many of us don't mind
answering questions like this, you can get a lot of answers quicker by
(1) looking to see what's in the standard library and (2) using google.

http://docs.python.org/lib/module-csv.html

James

The OP mentioned "csv.reader". This indicates to me that he *has* read
the csv docs (have you?), and is *already* using the csv module. The
tasks he says he does often are *not* covered by the standard library.
He appears to be asking if there is a higher-level wrapper around the
standard library.

Please consider reading his question carefully.

 

[John Machin]

It occured to me that most times I read a csv file, I'm often doing
from scratch things like assigning labels to columns, mapping fields to
the appropriate type, ignoring some fields, changing their order, etc.
Before I go on and reinvent the wheel, is there a generic high level
wrapper around csv.reader that does all this ?

Hi George,

Firstly one thing to remember: if you are going to reinvent the wheel,
don't forget to also reinvent the axle

AFAIK there is no such animal.

I would need a lot of persuasion that a wrapper or toolbox wasn't just
overhead with little benefit. For example, ignoring some fields and/or
changing the order of others can be done rather simply:

| >>> inrow
| ['a', 'b', 'c', 'd', 'e', 'f']
Suppose we want to ignore the "vowels" and reverse the order of the
others:
| >>> outrow = list(inrow[k] for k in (5, 3, 2, 1))
| >>> outrow
| ['f', 'd', 'c', 'b']

I don't see the value in creating (and documenting!) a one-line
function

def gather_list_items(input_list, item_map):
return list(input_list[k] for k in item_map)

NB something like this is already in the mx kit somewhere IIRC.

Cheers,
John

 

[skip]

George> It occured to me that most times I read a csv file, I'm often
George> doing from scratch things like assigning labels to columns,
George> mapping fields to the appropriate type, ignoring some fields,
George> changing their order, etc. Before I go on and reinvent the
George> wheel, is there a generic high level wrapper around csv.reader
George> that does all this ?

I'm not aware of anything that specifically addresses these ideas. Here are
some thoughts though:

* If you use the DictReader class you can ignore fields you aren't
interested in more easily since you access the fields of interest by
name and with the fieldnames parameter to the constructor can assign
column names to csv data which lacks it (use similar functionality in
DictWriter to create column labels on output). You can also specify
the restkey parameter to the constructor and thus only specify the
fields of interest in the fieldnames parameter. (I think. I've never
actually used that capability, but that's what the documentation
suggests.)

* There was a thread earlier this month with this subject:

paseline(my favorite simple script): does something similar exist?

Check it out for a number of different solutions to formatting the
fields in a line of text. The idea can easily be extended to a list
or dict of values instead, perhaps in a subclass of DictReader.

Skip

 

[George Sakkis]

George> It occured to me that most times I read a csv file, I'm often
George> doing from scratch things like assigning labels to columns,
George> mapping fields to the appropriate type, ignoring some fields,
George> changing their order, etc. Before I go on and reinvent the
George> wheel, is there a generic high level wrapper around csv.reader
George> that does all this ?

I'm not aware of anything that specifically addresses these ideas. Here are
some thoughts though:

* If you use the DictReader class you can ignore fields you aren't
interested in more easily since you access the fields of interest by
name and with the fieldnames parameter to the constructor can assign
column names to csv data which lacks it (use similar functionality in
DictWriter to create column labels on output). You can also specify
the restkey parameter to the constructor and thus only specify the
fields of interest in the fieldnames parameter. (I think. I've never
actually used that capability, but that's what the documentation
suggests.)

* There was a thread earlier this month with this subject:

paseline(my favorite simple script): does something similar exist?

Check it out for a number of different solutions to formatting the
fields in a line of text. The idea can easily be extended to a list
or dict of values instead, perhaps in a subclass of DictReader.

Skip

Indeed, they are both relevant; actually Fredrik's suggestion in that
thread was my starting point. Here's my my current API for the (most
typical) case of fixed-size rows, addressing my most common
requirement, field conversions (no named columns for now):
http://rafb.net/paste/results/4UgvSD50.html.

It looks somewhat involved but it's more along the lines of "making
easy things easy and hard things possible". Comments and suggestions
are most welcome.

George