anyway to create a table-like object?

[Wensui Liu]

Dear all,

is there a way to create a 2-dimension-table-like object such that I
can reference a whole column directly using something like :
mytable.column1 ?

by the way, can python be used for database programming to pull large
volume data (hundred M or even several Gs) out of large database and
do data manipulation and reporting ? if yes, how is the speed and
efficiency ?

thanks.

wensui

 

[James Stroud]

Dear all,

is there a way to create a 2-dimension-table-like object such that I
can reference a whole column directly using something like :
mytable.column1 ?

by the way, can python be used for database programming to pull large
volume data (hundred M or even several Gs) out of large database and
do data manipulation and reporting ? if yes, how is the speed and
efficiency ?

thanks.

wensui

I have a table class that works like this:

atable['some_col'] ==> list of values
atable[some_row_index] ==> list of values
atable[slice_start:slice_end] ==> new table
atable[('some_col', 'other_col', 'another_col')] ==> 3 col. table
atable['some_col', some_row_index] ==> one value
atable['some_col', slice_start:slice_end] ==> list of values
atable[('some_col', 'other_col'), some_row_index] ==> list of 2 vals
atable[('some_col', 'other_col'), slice_start:end]) ==> 2 col. table

As you can see, it has a lot of flexibility, but may not be the ultimate
in terms of speed (and purists may not like the way __getitem__ returns
data based on context). It is for people who care about convenience over
speed and do not adhere to some puritanical ideal about how __getitem__
should work. There is little typechecking (in accord with my own brand
of puritanical philosophy), so be careful if you have ints as column
headers. The datastructure is implemented as a list of lists (by row)
and the "keys" (column headings) are a list. So size limitations are
limited by memory. Theoretically, the __getitem__ (where all the work
and decisions are done) could be factored out and wrapped around dbi for
hard-core database usage. This might entail some serious creativity.

It also has some other features, such as returning matches, etc. It is
not intended, at all, to rival an actual database such as mysql--the
model is more based on how Joe Users use excel files.

If you are interested, I can send you the module. Beyond actual use in
my own and my wife's work, there has been little testing.

James

 

[James Stroud]

Dear all,

is there a way to create a 2-dimension-table-like object such that I
can reference a whole column directly using something like :
mytable.column1 ?

by the way, can python be used for database programming to pull large
volume data (hundred M or even several Gs) out of large database and
do data manipulation and reporting ? if yes, how is the speed and
efficiency ?

thanks.

wensui

I have a table class that works like this:

atable['some_col'] ==> list of values
atable[some_row_index] ==> list of values
atable[slice_start:slice_end] ==> new table
atable[('some_col', 'other_col', 'another_col')] ==> 3 col. table
atable['some_col', some_row_index] ==> one value
atable['some_col', slice_start:slice_end] ==> list of values
atable[('some_col', 'other_col'), some_row_index] ==> list of 2 vals
atable[('some_col', 'other_col'), slice_start:end]) ==> 2 col. table

As you can see, it has a lot of flexibility, but may not be the ultimate
in terms of speed (and purists may not like the way __getitem__ returns
data based on context). It is for people who care about convenience over
speed and do not adhere to some puritanical ideal about how __getitem__
should work. There is little typechecking (in accord with my own brand
of puritanical philosophy), so be careful if you have ints as column
headers. The datastructure is implemented as a list of lists (by row)
and the "keys" (column headings) are a list. So size limitations are
limited by memory. Theoretically, the __getitem__ (where all the work
and decisions are done) could be factored out and wrapped around dbi for
hard-core database usage. This might entail some serious creativity.

It also has some other features, such as returning matches, etc. It is
not intended, at all, to rival an actual database such as mysql--the
model is more based on how Joe Users use excel files.

If you are interested, I can send you the module. Beyond actual use in
my own and my wife's work, there has been little testing.

James

Also, if you want columns as attributes (which may be pretty cool):

def __getattr__(self, att):
if att in self.headers:
return self[att]
else:
return self.__getattribute__(att)

James

 

[Dennis Lee Bieber]

Dear all,

is there a way to create a 2-dimension-table-like object such that I
can reference a whole column directly using something like :
mytable.column1 ?

Well... Python 2.5 includes the SQLite3 adapter (and runtime?) so

by the way, can python be used for database programming to pull large
volume data (hundred M or even several Gs) out of large database and
do data manipulation and reporting ? if yes, how is the speed and
efficiency ?

As for this... I'd say expend the effort on coming up with an SQL
statement that does most of the needed "data manipulation" so that
"reporting" comes down to mostly just fetching result rows and
formatting the output...

Whilst SQL does have aggregates and grouping, if one needs both
line-item and group sums (aka: control breaks), it may be easier to do a
hybrid -- let the DBMS return rows in predetermined group/order, but do
the control breaks in Python.

control = None
sum = 0
for row in cursor:
if not control and control != row["control"]:
write(sum)
sum = 0
control = row["control"]
write(row)
if not control:
write(sum) #last group
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/