Editing tabular data [was: PEP8 79 char max]

[Skip Montanaro]

I don't understand. That just moves them to a different file --

doesn't it? You've still got to deal with editing a large table of
data (for example when I want to add instructions to your assembler).

My guess is it would be more foolproof to edit that stuff with a spreadsheet.

Skip

 

[Grant Edwards]

My guess is it would be more foolproof to edit that stuff with a
spreadsheet.

Many years ago, I worked with somebody who used a spreadsheet like
that. I tried it and found it to be way too cumbersome. The overhead
involved of putting tables in to slew of different files and starting
up LibreOffice to edit/view them is huge compared to just editing them
with emacs in a file along with the source code. Maybe my computer is
too old/slow. Maybe it's just due to how bad I am at Excel/LibreOffice...

 

[Skip Montanaro]

My guess is it would be more foolproof to edit that stuff with a

Many years ago, I worked with somebody who used a spreadsheet like
that.

I really love Emacs, however... One of the traders here where I work
(who shall not be named) had a space-delimited data file with hundreds
of rows and 50 or so columns. I could never get him to edit it in any
kind of spreadsheet or put it in a database (expecting him to master
SQL would have been pointless - I would have had to write a GUI tool
for him). He always modified it in Emacs, and would delete columns,
add extra spaces, fragmentary rows, etc. He'd edit this file late at
night, the automated processes the next morning would crap out, and I
would scramble to try and find and fix the problem before the market
opened.

This is clearly a case where choosing the proper tool is important. I
agree that using a spreadsheet to edit a 3x5 CSV file is likely
overkill (might just as well use Notepad or TextEdit), but tabular
data are tabular data, no matter how they might be delimited, and if
there are many of those little data critters, there are better tools
than a text editor (or Python IDE) for maintaining them.

Skip

 

[Wanderer]

My guess is it would be more foolproof to edit that stuff with a spreadsheet.



Skip

Has anyone tried Pyspread?

http://manns.github.io/pyspread/

 

[Skip Montanaro]

Has anyone tried Pyspread?

I have not.

I have a fundamental problem with spreadsheets, the extremely narrow view
of the workspace. There was a piece on NPR the other day about some errors
in some modeling applications. I missed most of it (does someone have a
link? I'm on my phone right now), but the expert commentator was saying
that they are working on standard structures for these sorts of complex
modeling simulations. I think they will need more than that. Something like
pyspread might allow you to mix structured and object oriented programming
with the convenience if a spreadsheet.

Skip

 

[Chris Angelico]

Many years ago, I worked with somebody who used a spreadsheet like
that. I tried it and found it to be way too cumbersome. The overhead
involved of putting tables in to slew of different files and starting
up LibreOffice to edit/view them is huge compared to just editing them
with emacs in a file along with the source code. Maybe my computer is
too old/slow. Maybe it's just due to how bad I am at Excel/LibreOffice...

I'm glad someone else feels that way!

At work, we have a number of CSV files (at my boss's insistence; I
would much rather they be either embedded in the source, or in some
clearer and simpler format) which I like to manipulate in SciTE,
rather than OO/LibreOffice. (I'll not distinguish those two. Far as
I'm concerned, they're one product with two names.) My boss can't
understand why I do this. I can't understand why he objects to having
to edit code files to alter internal data. I have pointed him to [1]
but to no avail.

The one thing I would do, though, is align with tabs rather than
spaces. That gives you an 8:1 (if you keep your tabs at eight, which I
do) improvement in maintainability, because edits that don't cross a
boundary don't require fiddling with the layout.

[1] http://thedailywtf.com/Articles/Soft_Coding.aspx

ChrisA

 

[Neil Cerutti]

Many years ago, I worked with somebody who used a spreadsheet
like that. I tried it and found it to be way too cumbersome.
The overhead involved of putting tables in to slew of
different files and starting up LibreOffice to edit/view them
is huge compared to just editing them with emacs in a file
along with the source code. Maybe my computer is too
old/slow. Maybe it's just due to how bad I am at
Excel/LibreOffice...

I'm glad someone else feels that way!

At work, we have a number of CSV files (at my boss's
insistence; I would much rather they be either embedded in the
source, or in some clearer and simpler format) which I like to
manipulate in SciTE, rather than OO/LibreOffice. (I'll not
distinguish those two. Far as I'm concerned, they're one
product with two names.) My boss can't understand why I do
this. I can't understand why he objects to having to edit code
files to alter internal data. I have pointed him to [1] but to
no avail.

The one thing I would do, though, is align with tabs rather
than spaces. That gives you an 8:1 (if you keep your tabs at
eight, which I do) improvement in maintainability, because
edits that don't cross a boundary don't require fiddling with
the layout.

[1] http://thedailywtf.com/Articles/Soft_Coding.aspx

Thanks for that link. Good food for thought.

Here's an excerpt from one of my more questionable tables:

Attribute, Description, Fund, Amount
AFSO,Air Force Special Ops Command,,
CSEN,English Proficiency Met,,
CSMT,Math Proficiency Met,,
GBFP,MBA Full Program,,
GBMP,MBA Prereq Met,,
GCEC,Continuing Education Civilian Tuition Rate,,
GCEM,Continuing Education Military Tuition Rate,,
GCFP,MCA Prereq Needed,,
GCMP,MCE Prereq Met,,
GCRT,Certificate Student,,
GE25,25% to XCompany,,
GE40,40% to XCompany,,
GEMP,Employee,Fac,100%
GI03,CISSP Scholarship,CISSP,1500
GIHR,Grad In-House Recruiting,,
GRMS,Graduate Military Scholarship,Milit,1200

It lists all the student atributes, a description, what fund that
attribute requires, if any, and what amount. A tiny amount of DSL
is involved, with Faculty Scholarship paying 100% of tuition
instead of a fixed number. Another _ (not shown above), which
means the fund takes an arbitrary amount determined by a person
we have to literally query to discover.

I think I can see the potential problems. Two special codes for
amount is managable, but the more special cases I end up creating
the more of a mess I get. Plus, I haven't really documented the
file.

Most of the information is irrelevant, though I do like receiving
an exception when Admissions tries to sneak in a new attribute
without telling me.

If I instead had a function that handled only the interesting
attributes it might be pretty small. I'll have to think on this.

 

[wxjmfauth]

Le jeudi 1 août 2013 02:50:13 UTC+2, Chris Angelico a écrit :

...

rather than OO/LibreOffice. (I'll not distinguish those two. Far as

I'm concerned, they're one product with two names.)

....

Very interesting aspect in LibreOffice.

As the "center of gravity of the development" has moved
from "US" to "Europe", the product becomes clearly
less (practicaly no more traces) ascii oriented.

jmf

 

[Chris Angelico]

[…] rather than OO/LibreOffice. (I'll not distinguish those two. Far
as I'm concerned, they're one product with two names.)

That's simply false. ...

Claiming they're the same product is ignoring the transfer of
development away from the OpenOffice.org code dump, and to LibreOffice
as the actively-developed product.

To be sure, they're different; but they're part of one family tree.
It's like referring to "Debian/Ubuntu" when you're discussing
something where it makes absolutely zero difference which one you're
talking about. The difference between using LibreOffice and using
OpenOffice is nothing compared to the difference between working with
either of the above and putting a literal in your code.

ChrisA

 

[Grant Edwards]

[?] rather than OO/LibreOffice. (I'll not distinguish those two. Far
as I'm concerned, they're one product with two names.)

That's simply false. ...

Claiming they're the same product is ignoring the transfer of
development away from the OpenOffice.org code dump, and to LibreOffice
as the actively-developed product.

To be sure, they're different; but they're part of one family tree.
It's like referring to "Debian/Ubuntu" when you're discussing
something where it makes absolutely zero difference which one you're
talking about. The difference between using LibreOffice and using
OpenOffice is nothing compared to the difference between working with
either of the above and putting a literal in your code.

In the context in which I mentioned LibreOffice, I don't even consider
there to be a significant difference between Libre/OO and Excel.

 

[Terry Reedy]

[…] rather than OO/LibreOffice. (I'll not distinguish those two. Far
as I'm concerned, they're one product with two names.)

That's simply false. LibreOffice has, since the 2010 fork of the code
base and especially since the exodus of developers to The Document
Foundation [0], gained a great number of improvements [1] and is now the
clear inheritor of active development.

Of relevance to this list, Libre Office upgraded the included Python
interpreter to 3.3. I have no idea whether OO is still using 2.3 or also
updated.

 

[David Robinow]

...
Of relevance to this list, Libre Office upgraded the included Python
interpreter to 3.3. I have no idea whether OO is still using 2.3 or also
updated.

They're up to 2.7 now.

 

[rusi]

I really love Emacs, however... […]

This is clearly a case where choosing the proper tool is important. I
agree that using a spreadsheet to edit a 3x5 CSV file is likely
overkill (might just as well use Notepad or TextEdit), but tabular
data are tabular data, no matter how they might be delimited, and if
there are many of those little data critters, there are better tools
than a text editor (or Python IDE) for maintaining them.

It seems an obvious thing for powerful text editors like Emacs and Vim
to have a third-party mode for editing CSV data with a tabular
interface.


Indeed, such modes exist; one that I found immediately for Emacs is
<URL:http://www.emacswiki.org/emacs/csv-mode.el>. Has anyone got a good
Emacs mode for editing CSV data as a table and saving it back to CSV
data?

Emacs users can have the cake and eat it too; ie use spreadsheet
functionality without having to use a separate spreadsheet file
and software.

The basic idea is to use org-mode which has a table
editor with spreadsheet functionality while continuing to live within a
plain text editor.

It allows to edit a table entirely written in plain text in a visually appealing and clean way, while keeping a (less readable) python data
structure in sync with it.

The example file is below the ---------------

Caveats:
The orig_table string is there only make the source for the table.
The name orig_table is not necessary; a naked triple-string will also work.
The triple string is there to pacify python in the face of non valid syntax..
Ideal would have been comments but python does not have multiline comments

The table between the #BEGIN RECEIVE ORGTBL marks
and the #END RECEIVE ORGTBL marks
is the target or the recipient for the transformed version of the
plain text table.

Experiment as follows:
1. Save the stuff below --------- as something.py
2. Start editing the file in emacs

3. Join the 3 lines into 1 line with single space separators.
+ORGTBL: SEND marks orgtbl-to-generic
:lfmt " \"%s\": [%s,%s,%s,%s,%s],"
:llfmt " \"%s\": [%s,%s,%s,%s,%s]"

[It has to be one line, but if I kept it one line, it
will be randomly be garbled in the mail!]

This line gives the table a name ("marks") so that you can use
several tables in one file, and it specifies how the syntax should
be changed when syncing the python version of the table data.

4. Start orgtbl minor mode with M-x orgtbl-mode
Mode line should show python and orgtbl

5. Delete the contents (keep the 2 # lines intact) of the python table

6. Place cursor within the orig_table and 'send' it as follows

7. Send is done with any one of 'C-c C-c' or 'C-u C-c C-c' or 'C-u C-u C-c C-c'
The first just sends the table as is
The second recomputes the formulas top-down and then sends
The third recomputes until fixpoint (you really should not be making such a
table!!)

8. Play with the table editor by using TAB and S-TAB to
walk through fields and change them, use C-u C-c C-c again to
sync the python version of the table

9. In case the above does now work (if your orgmode is too old)
the orig_table_2 should hopefully work even for older org versions
It furthermore shows the ability to skip columns and to format
column widths to convenience.

----------------------------------
orig_table = """

#+ORGTBL: SEND marks orgtbl-to-generic
:lfmt " \"%s\": [%s,%s,%s,%s,%s],"
:llfmt " \"%s\": [%s,%s,%s,%s,%s]"
| abe | 1 | 2 | 3 | 4 | 10 |
| beth | 9 | 1 | 5 | 9 | 24 |
| catherine | 5 | 6 | 7 | 5 | 23 |
#+TBLFM: $6=$2+$3+$4+$5
"""

stud_db = {
# Dont handedit
# BEGIN RECEIVE ORGTBL marks
"abe": [1,2,3,4,10],
"beth": [9,1,5,9,24],
"catherine": [5,6,7,5,23]
# END RECEIVE ORGTBL marks
}

## In case the above does not work (if org-version too old)

orig_table_2 = """

#+ORGTBL: SEND marks2 orgtbl-to-csv :skip 2
| Name | T1 | T2 | T3 | T4 | Total |
| <6> | | | | | |
| abe | 1 | 2 | 3 | 4 | 10 |
| beth | 9 | 1 | 5 | 9 | 24 |
| catherine | 5 | 6 | 7 | 5 | 23 |
#+TBLFM: $6=$2+$3+$4+$5
"""

stud_db_2 = {
# Dont handedit
# BEGIN RECEIVE ORGTBL marks2
abe,1,2,3,4,10
beth,9,1,5,9,24
catherine,5,6,7,5,23
# END RECEIVE ORGTBL marks2