Request for a change in the csv module.

[Paulo da Silva]

Hi.

I have just seen that csv module, more exactly the Dialect class,
does not have any variable to specify the "floating point" character!
In portuguese this is ','. Not '.'. 3.1415 -> 3,1415.
I think this is also the case of other languages/countries.
If I am correct, i.e. if there is no such possibility, where can
I file a request for a csv change? Excel, for example, automatically
converts '.' to ',' and the separator from ',' to ';'.

Thanks.

 

[attn.steven.kuo]

Hi.

I have just seen that csv module, more exactly the Dialect class,
does not have any variable to specify the "floating point" character!
In portuguese this is ','. Not '.'. 3.1415 -> 3,1415.
I think this is also the case of other languages/countries.
If I am correct, i.e. if there is no such possibility, where can
I file a request for a csv change? Excel, for example, automatically
converts '.' to ',' and the separator from ',' to ';'.

Try using locale.format when writing:

import csv
import locale

locale.setlocale(locale.LC_NUMERIC, 'pt_BR.ISO8859-1')
csv_writer = csv.writer(open("foo.csv","w"), dialect='excel')
rows = (('testing', 1.23), ('testing', 2.34))
formatted_rows = ((string, locale.format('%g', number)) for
(string,number) in rows)
csv_writer.writerows(formatted_rows)


locale.atof and locale.atoi can convert a string back
to a number.

 

[Paulo da Silva]

(e-mail address removed) escreveu:

locale.setlocale(locale.LC_NUMERIC, 'pt_BR.ISO8859-1')
csv_writer = csv.writer(open("foo.csv","w"), dialect='excel')
rows = (('testing', 1.23), ('testing', 2.34))
formatted_rows = ((string, locale.format('%g', number)) for
(string,number) in rows)
csv_writer.writerows(formatted_rows)

That works but it is a pain to use.
May be I'll sublass csv or even I'll write one myself.
It would be much better to be able to specify an additional
variabel to the Dialect class and change csv.

Thanks
Paulo

 

[Diez B. Roggisch]

(e-mail address removed) escreveu:


That works but it is a pain to use.

Why? I think it's straightforward.

May be I'll sublass csv or even I'll write one myself.
It would be much better to be able to specify an additional
variabel to the Dialect class and change csv.

I don't think so. The csv module is about the subtleties that can occur
when parsing textual data with possible escape chars and the like, and
creating that data.

But IHMO its up to the user to feed it with just textual data. Because
automatically converting numbers falls flat on it's face in case of
special requirements like a limit on the number of digits to render and
the like. Better do that in a simple and straightforward preprocessing
state, as shown above.

The same is essentially true for dates as well, btw. How do you want to
deal with them?

Diez

 

[Szabolcs Nagy]

It would be much better to be able to specify an additional
variabel to the Dialect class and change csv.

no it wouldn't
this is a locale specific problem so it should be handled there

 

[Paulo da Silva]

Diez B. Roggisch escreveu:

Paulo da Silva schrieb: ....


Why? I think it's straightforward.

That is not the point. The problem is to have things generalized.
I have some general purpose tables whose fields format I don't know
in advance.
Internally the numeric values are independent of localization.
Having a table, I don't like the idea of preformating things
before sending them to the csv for exportation. A little more
of programming and there will be no need for csv at all
....

The same is essentially true for dates as well, btw. How do you want to
deal with them?

This is a pertinent question I didn't think of. May be csv should handle
data conversions using 'locale'.
It is already automatically converting numbers to text
anyway, but unconditionally using the '.'.
As for the dates the dialect could include a format like YYYY-MM-DD
hh:mm:ss. Dates are not always 'locale' only. In pt_PT we use
YYYY-MM-DD or YY-MM-DD and sometimes DD-MM-YYYY.
One must remember that csv is not part of the language. So, to have
some kind of universality, it must use the specs (in dialect) for
formatting and 'locale' for data conversions.

 

[Diez B. Roggisch]

Diez B. Roggisch escreveu:

That is not the point. The problem is to have things generalized.
I have some general purpose tables whose fields format I don't know
in advance.
Internally the numeric values are independent of localization.
Having a table, I don't like the idea of preformating things
before sending them to the csv for exportation. A little more
of programming and there will be no need for csv at all
...

This is simply _not_ true. The details of a proper CSV format are difficult
to get right, which is precisely why the module is there.

This is a pertinent question I didn't think of. May be csv should handle
data conversions using 'locale'.
It is already automatically converting numbers to text
anyway, but unconditionally using the '.'.
As for the dates the dialect could include a format like YYYY-MM-DD
hh:mm:ss. Dates are not always 'locale' only. In pt_PT we use
YYYY-MM-DD or YY-MM-DD and sometimes DD-MM-YYYY.
One must remember that csv is not part of the language. So, to have
some kind of universality, it must use the specs (in dialect) for
formatting and 'locale' for data conversions.

I'm not too convinced. It's so darn easy to convert the values to what one
really needs for the actual use-case. Even for your generic case - all you
need is type-based formatting dispatch.

But if you add that complexity to the writers of the csv-module, you'll end
up with a nasty configuration mechanism for special casing individual
columns, or otherwise the next user comes and says "well, I need to format
dates differently just for this case, how to do so in the csv-module". As
your own example shows.

The purpose of the module is solely to assist in creating properly formatted
csv content in the sense that it is readable and parsable and results in
strings for column values. Not more, not less (and you seem to
underestimate that task, btw.)

That it does convert non-string-values to strings when writing could be seen
as convenience, or actual bug as it might cause troubles as you perceived
them - but the solution would clearly be to get rid of this functionality,
not enhance it.

Diez

 

[skip]

Paulo> That is not the point. The problem is to have things generalized.

Well, perhaps. One of the goals of the csv module was to do things the way
Excel does things. Ideally, that would include formatting elements with
locale sensitivity. I've been working on a csv module in Python, so I
decided to try the locale.format() calls in its writerow implementation. A
number of test cases broke as a result because apparently locale.format
doesn't do its job quite the same way Excel does. I don't think pushing the
locale-sensitive formatting down into the csv module is going to be a
panacea.

Skip

 

[Paulo da Silva]

Diez B. Roggisch escreveu:

Paulo da Silva wrote: ....


That it does convert non-string-values to strings when writing could be seen
as convenience, or actual bug as it might cause troubles as you perceived
them - but the solution would clearly be to get rid of this functionality,
not enhance it.

It's just a matter of
personal opinion, and IMO, taking in consideration what other python
modules provide, a csv module should make the appropriate data
conversions. It is simple to implement for most (all?) cases and it will
bring a lot of functionality. You just output things.
In csv output you don't need special data formats. Only the data to the
most available precision. Dates are a very special case, as are the
separators, line terminators, etc.

The python way of exporting this data should be something like:
class MyDialect(Dialect):
...
locale="pt_PT@euro"
dateformat="YYYY-MM-DD hh:mm:ss"

...
w=csv.writer(f,dialect=MyDialect)
w.writerows(foo)

Frequent tasks should be done by the modules. Not by the users.
And csv is unusable without output formatting. This is particulary true
if, as you said, the csv didn't provide any conversion at all.
So, why force the programmers to repeat the same tedious procedures
whenever they want to use csv, not to talk about the lack of performance
that implies?

 

[Paulo da Silva]

(e-mail address removed) escreveu:

Paulo> That is not the point. The problem is to have things generalized.

Well, perhaps. One of the goals of the csv module was to do things the way
Excel does things.

It would be nice because many csv users use it to export files to
spreadsheets and Excel is certainly a reference. But there may be a
difference! One thing is to do things as Excel does and another is
to write data in a way Excel understands. The later, IMO, is not so
complicated.

Ideally, that would include formatting elements with

locale sensitivity. I've been working on a csv module in Python, so I
decided to try the locale.format() calls in its writerow implementation. A
number of test cases broke as a result because apparently locale.format
doesn't do its job quite the same way Excel does.

locale.format does not need to write the same way Excel does. It just
should convert in a way Excel understands.
Anyway, a csv user needs to do that if it is not provided by csv.