Best way to do data source abstraction

[Arthur Pemberton]

What is the best way to do data source abtraction? For example have
different classes with the same interface, but different
implementations.

I was thinking of almost having classA as my main class, and have
classA dynamically "absorb" classFood into to based on the extension
of the input file received by classA. But this doesn't seem possible.

Please advise.

Thank you.

 

[Sybren Stuvel]

Arthur Pemberton enlightened us with:

What is the best way to do data source abtraction?

That depends on your data source. For files, file-like objects are an
abstraction. For databases there is PEP 249.

I was thinking of almost having classA as my main class, and have
classA dynamically "absorb" classFood into to based on the extension
of the input file received by classA. But this doesn't seem
possible.

You don't explain the most important part - "absorb". What does that
mean? And what does it mean to have classA "almost" as your main
class?

Sybren

 

[bruno at modulix]

What is the best way to do data source abtraction? For example have
different classes with the same interface, but different
implementations.

I was thinking of almost having classA as my main class, and have
classA dynamically "absorb" classFood into to based on the extension
of the input file received by classA. But this doesn't seem possible.

Could you explain more accurately what you're trying to do ? FWIW, it
seems that a plain old factory function would do ?

class DatasourceAbstraction(object):
""" base class, factoring common stuff """
# implementation here

class JpegFile(DatasourceAbstraction):
# ....


class PdfFile(DatasourceAbstraction):
# ....

class TxtFile(DatasourceAbstraction):
# ....

# etc
_classes = {
'jpg' : JpegFile,
'txt' : TxtFile,
'pdf' : PdfFile,
# etc..
}

def Datasource(inputfile):
ext = os.path.splitext(inputfile)
return _classes.get(ext, <SomeDefaultClassHere>)(inputfile)

The fact that there's no 'new' keyword in Python, and that classes are
callable objects acting as factories means that it's a no-brainer to use
a plain function (eventually disguised as a Class - the client code just
doesn't care !-) as factory...

Now if I missed the point, please give more explanations...

 

[Larry Bates]

What is the best way to do data source abtraction? For example have
different classes with the same interface, but different
implementations.

I was thinking of almost having classA as my main class, and have
classA dynamically "absorb" classFood into to based on the extension
of the input file received by classA. But this doesn't seem possible.

Please advise.

Thank you.

The best method I've found is to have a class that abstracts
the data and presents the values from the data source via
class attributes. If you also implement it as an iterator
(e.g. give it __iter__ and __next__ methods), you can easily
iterate over the resultset from each data source. With this
method I've abstracted data in CSV files, fixed ASCII files,
SQL tables, Excel Spreadsheets, tab delimited files that are
members of a .ZIP archive, you name it.

Short example (not tested):
class foo:
'''
Class to abstract tab delimited files
'''
def __init__(self, filepath, columnnames):
self.fp=open(filepath, 'r')
self.columnnames=columnnames
return

def __iter__(self):
return self

def next(self):
#
# Try to get the next line from file
#
try: line=self.fp.next()
except StopIteration:
self.fp.close()
raise

#
# Decode the tab delimited line into its parts
#
line=line.rstrip()
if not line: raise StopIteration
values=line.split('\t')
print "values=", values
l=zip(self.columnnames, values)
print l
for column, value in l:
setattr(self, column, value)

return

if __name__ == "__main__":
obj=foo('abc.txt', ['name', 'address1', 'address2', 'city', 'state', 'zip'])

for entry in obj:
print ""
print "Name........", obj.name
print "Address1....", obj.address1
print "Address2....", obj.address2
print "City........", obj.city
print "State.......", obj.state
print "Zip.........", obj.zip


-Larry Bates

 

[Larry Bates]

What is the best way to do data source abtraction? For example have
different classes with the same interface, but different
implementations.

I was thinking of almost having classA as my main class, and have
classA dynamically "absorb" classFood into to based on the extension
of the input file received by classA. But this doesn't seem possible.

Please advise.

Thank you.

The best method I've found is to have a class that abstracts
the data and presents the values from the data source via
class attributes. If you also implement it as an iterator
(e.g. give it __iter__ and __next__ methods), you can easily
iterate over the resultset from each data source. With this
method I've abstracted data in CSV files, fixed ASCII files,
SQL tables, Excel Spreadsheets, tab delimited files that are
members of a .ZIP archive, you name it.

Short example (not tested):
class foo:
'''
Class to abstract tab delimited files
'''
def __init__(self, filepath, columnnames):
self.fp=open(filepath, 'r')
self.columnnames=columnnames
return

def __iter__(self):
return self

def next(self):
#
# Try to get the next line from file
#
try: line=self.fp.next()
except StopIteration:
self.fp.close()
raise

#
# Decode the tab delimited line into its parts
#
line=line.rstrip()
if not line: raise StopIteration
values=line.split('\t')
print "values=", values
l=zip(self.columnnames, values)
print l
for column, value in l:
setattr(self, column, value)

return

if __name__ == "__main__":
obj=foo('abc.txt', ['name', 'address1', 'address2', 'city', 'state', 'zip'])

for entry in obj:
print ""
print "Name........", obj.name
print "Address1....", obj.address1
print "Address2....", obj.address2
print "City........", obj.city
print "State.......", obj.state
print "Zip.........", obj.zip


-Larry Bates