design question, metaclasses?

Discussion in 'Python' started by Darren Dale, Apr 11, 2009.

  1. Darren Dale

    Darren Dale Guest

    I am working on a project that provides a high level interface to hdf5
    files by implementing a thin wrapper around h5py. I would like to
    generalize the project so the same API can be used with other formats,
    like netcdf or ascii files. The format specific code exists in File,
    Group and Dataset classes, which I could reimplement for each format.
    But there are other classes deriving from Group and Dataset which do
    not contain any format-specific code, and I would like to find a way
    to implement the functionality once and apply uniformly across
    supported formats. This is really abstract, but I was thinking of
    something along the lines of:

    format1.Group # implementation of group in format1
    format2.Group # ...
    Base.DerivedGroup # base implementation of DerivedGroup, not directly
    useful
    format1.DerivedGroup = Base.DerivedGroup(format1.Group) # useful
    format2.DerivedGroup = Base.DerivedGroup(format2.Group) # useful

    Could anyone please offer a comment, is this an appropriate use of
    metaclassing, or is there maybe an easier/better alternative?
    Darren Dale, Apr 11, 2009
    #1
    1. Advertising

  2. Darren Dale

    Darren Dale Guest

    On Apr 11, 2:15 pm, Darren Dale <> wrote:
    > I am working on a project that provides a high level interface to hdf5
    > files by implementing a thin wrapper around h5py. I would like to
    > generalize the project so the same API can be used with other formats,
    > like netcdf or ascii files. The format specific code exists in File,
    > Group and Dataset classes, which I could reimplement for each format.
    > But there are other classes deriving from Group and Dataset which do
    > not contain any format-specific code, and I would like to find a way
    > to implement the functionality once and apply uniformly across
    > supported formats. This is really abstract, but I was thinking of
    > something along the lines of:
    >
    > format1.Group # implementation of group in format1
    > format2.Group # ...
    > Base.DerivedGroup # base implementation of DerivedGroup, not directly
    > useful
    > format1.DerivedGroup = Base.DerivedGroup(format1.Group) # useful
    > format2.DerivedGroup = Base.DerivedGroup(format2.Group) # useful
    >
    > Could anyone please offer a comment, is this an appropriate use of
    > metaclassing, or is there maybe an easier/better alternative?


    I don't fully understand metaclasses, but I think I have convinced
    myself that they are not what I was looking for. I think this will do
    what I want it to:

    class Group1(object):

    def origin(self):
    return "Group1"


    class Group2(object):

    def origin(self):
    return "Group2"


    def _SubGroup(superclass):

    class SubGroup(superclass):
    pass

    return SubGroup


    SubGroup = _SubGroup(Group2)
    sub_group = SubGroup()

    print sub_group.origin()
    Darren Dale, Apr 12, 2009
    #2
    1. Advertising

  3. Darren Dale

    Aaron Brady Guest

    On Apr 12, 1:30 pm, Darren Dale <> wrote:
    > On Apr 11, 2:15 pm, Darren Dale <> wrote:
    >
    >

    _
    >
    > > format1.Group # implementation of group in format1
    > > format2.Group # ...
    > > Base.DerivedGroup # base implementation of DerivedGroup, not directly
    > > useful
    > > format1.DerivedGroup = Base.DerivedGroup(format1.Group) # useful
    > > format2.DerivedGroup = Base.DerivedGroup(format2.Group) # useful

    >

    _
    >
    > class Group1(object):
    >
    >     def origin(self):
    >         return "Group1"
    >
    > class Group2(object):
    >
    >     def origin(self):
    >         return "Group2"
    >
    > def _SubGroup(superclass):
    >
    >     class SubGroup(superclass):
    >         pass
    >
    >     return SubGroup
    >
    > SubGroup = _SubGroup(Group2)
    > sub_group = SubGroup()
    >
    > print sub_group.origin()


    You can create new types in one statement:

    SubGroup= type( "SubGroup", ( BaseGroup, ), { } )
    Aaron Brady, Apr 12, 2009
    #3
  4. Darren Dale

    Darren Dale Guest

    On Apr 12, 3:23 pm, Aaron Brady <> wrote:
    > On Apr 12, 1:30 pm, Darren Dale <> wrote:
    >
    >
    >
    > > On Apr 11, 2:15 pm, Darren Dale <> wrote:

    >
    > _
    >
    > > > format1.Group # implementation of group in format1
    > > > format2.Group # ...
    > > > Base.DerivedGroup # base implementation of DerivedGroup, not directly
    > > > useful
    > > > format1.DerivedGroup = Base.DerivedGroup(format1.Group) # useful
    > > > format2.DerivedGroup = Base.DerivedGroup(format2.Group) # useful

    >
    > _
    >
    > > class Group1(object):

    >
    > >     def origin(self):
    > >         return "Group1"

    >
    > > class Group2(object):

    >
    > >     def origin(self):
    > >         return "Group2"

    >
    > > def _SubGroup(superclass):

    >
    > >     class SubGroup(superclass):
    > >         pass

    >
    > >     return SubGroup

    >
    > > SubGroup = _SubGroup(Group2)
    > > sub_group = SubGroup()

    >
    > > print sub_group.origin()

    >
    > You can create new types in one statement:
    >
    > SubGroup= type( "SubGroup", ( BaseGroup, ), { } )


    But how can I implement the *instance* behavior of SubGroup with this
    example? In my original example:

    format1.Group # implementation of group in format1
    format2.Group # implementation of group in format2
    Base.DerivedGroup # base implementation of DerivedGroup, must subclass
    a group
    format1.DerivedGroup = Base.DerivedGroup(format1.Group) # useful
    format2.DerivedGroup = Base.DerivedGroup(format2.Group) # useful

    I'm trying to achieve uniform behavior of my derived groups across
    supported formats. My derived groups are abstracted such that they do
    not need to be reimplemented for each format, I only need to implement
    Group for each format. This is a real mind bender for me, even my
    factory function gets hairy because I have additional classes that
    derive from DerivedGroup. Maybe what I need is the ability to provide
    context at import time, is that possible?
    Darren Dale, Apr 12, 2009
    #4
  5. Darren Dale

    Kay Schluehr Guest

    On 11 Apr., 20:15, Darren Dale <> wrote:

    > I am working on a project that provides a high level interface to hdf5
    > files by implementing a thin wrapper around h5py.
    > I would like to
    > generalize the project so the same API can be used with other formats,
    > like netcdf or ascii files. The format specific code exists in File,
    > Group and Dataset classes, which I could reimplement for each format.
    > But there are other classes deriving from Group and Dataset which do
    > not contain any format-specific code, and I would like to find a way
    > to implement the functionality once and apply uniformly across
    > supported formats.


    Seems like you are doing it wrong. The classical OO approach is to add
    more details / refining classes in subclasses instead of doing it the
    other way round and derive the less specific classes from the more
    specific ones.
    Kay Schluehr, Apr 12, 2009
    #5
  6. Darren Dale

    Darren Dale Guest

    On Apr 12, 4:50 pm, Kay Schluehr <> wrote:
    > On 11 Apr., 20:15, Darren Dale <> wrote:
    >
    > > I am working on a project that provides a high level interface to hdf5
    > > files by implementing a thin wrapper around h5py.
    > > I would like to
    > > generalize the project so the same API can be used with other formats,
    > > like netcdf or ascii files. The format specific code exists in File,
    > > Group and Dataset classes, which I could reimplement for each format.
    > > But there are other classes deriving from Group and Dataset which do
    > > not contain any format-specific code, and I would like to find a way
    > > to implement the functionality once and apply uniformly across
    > > supported formats.

    >
    > Seems like you are doing it wrong. The classical OO approach is to add
    > more details / refining classes in subclasses instead of doing it the
    > other way round and derive the less specific classes from the more
    > specific ones.


    I think I am following the classical OO approach, refining details in
    subclasses. I just want a given subclass implementation describing a
    complex dataset to be able to work on top of multiple hierarchical
    file formats (like NetCDF or HDF5) by deriving from either NetCDF or
    HDF5 base classes that have an identical API. Those base classes
    encapsulate all the format-specific details, the subclasses allow a
    uniform image to be handled differently than a nonuniform image with a
    mask (for example). Maybe I should be delegating rather than
    subclassing.
    Darren Dale, Apr 12, 2009
    #6
  7. Darren Dale

    Aaron Brady Guest

    On Apr 12, 4:53 pm, Darren Dale <> wrote:
    > On Apr 12, 4:50 pm, Kay Schluehr <> wrote:
    >
    >
    >
    > > On 11 Apr., 20:15, Darren Dale <> wrote:

    >
    > > > I am working on a project that provides a high level interface to hdf5
    > > > files by implementing a thin wrapper around h5py.
    > > > I would like to
    > > > generalize the project so the same API can be used with other formats,
    > > > like netcdf or ascii files. The format specific code exists in File,
    > > > Group and Dataset classes, which I could reimplement for each format.
    > > > But there are other classes deriving from Group and Dataset which do
    > > > not contain any format-specific code, and I would like to find a way
    > > > to implement the functionality once and apply uniformly across
    > > > supported formats.

    >
    > > Seems like you are doing it wrong. The classical OO approach is to add
    > > more details / refining classes in subclasses instead of doing it the
    > > other way round and derive the less specific classes from the more
    > > specific ones.

    >
    > I think I am following the classical OO approach, refining details in
    > subclasses. I just want a given subclass implementation describing a
    > complex dataset to be able to work on top of multiple hierarchical
    > file formats (like NetCDF or HDF5) by deriving from either NetCDF or
    > HDF5 base classes that have an identical API. Those base classes
    > encapsulate all the format-specific details, the subclasses allow a
    > uniform image to be handled differently than a nonuniform image with a
    > mask (for example). Maybe I should be delegating rather than
    > subclassing.


    You made me think of the 'mixin' design pattern, also known as
    multiple inheritance. The subclass derives from multiple classes.
    One is the domain-specific description of the data class; the other is
    the file format.

    class NetCDF_DataA_Writer( NetCDF_Writer, DataA_Description ):
    pass

    The two classes can't have conflicting method names, however; or
    you'll have to resolve them by hand, or assign them unique names. In
    this example, the subclass would favor the first super class, and you
    could access the second super class with special names. It kind of
    makes uniform access unreliable.

    class NetCDF_DataA_Writer( NetCDF_Writer, DataA_Description ):
    data_write= DataA_Description.write
    file_write= NetCDF_Writer.write

    The standard library has a couple of examples of this, as I recall,
    such as the UDPServerMixin (sp).
    Aaron Brady, Apr 13, 2009
    #7
  8. >>>>> Darren Dale <> (DD) wrote:

    >DD> On Apr 11, 2:15 pm, Darren Dale <> wrote:
    >>> I am working on a project that provides a high level interface to hdf5
    >>> files by implementing a thin wrapper around h5py. I would like to
    >>> generalize the project so the same API can be used with other formats,
    >>> like netcdf or ascii files. The format specific code exists in File,
    >>> Group and Dataset classes, which I could reimplement for each format.
    >>> But there are other classes deriving from Group and Dataset which do
    >>> not contain any format-specific code, and I would like to find a way
    >>> to implement the functionality once and apply uniformly across
    >>> supported formats. This is really abstract, but I was thinking of
    >>> something along the lines of:
    >>>
    >>> format1.Group # implementation of group in format1
    >>> format2.Group # ...
    >>> Base.DerivedGroup # base implementation of DerivedGroup, not directly
    >>> useful
    >>> format1.DerivedGroup = Base.DerivedGroup(format1.Group) # useful
    >>> format2.DerivedGroup = Base.DerivedGroup(format2.Group) # useful
    >>>
    >>> Could anyone please offer a comment, is this an appropriate use of
    >>> metaclassing, or is there maybe an easier/better alternative?


    >DD> I don't fully understand metaclasses, but I think I have convinced
    >DD> myself that they are not what I was looking for. I think this will do
    >DD> what I want it to:


    >DD> class Group1(object):


    >DD> def origin(self):
    >DD> return "Group1"



    >DD> class Group2(object):


    >DD> def origin(self):
    >DD> return "Group2"



    >DD> def _SubGroup(superclass):


    >DD> class SubGroup(superclass):
    >DD> pass


    >DD> return SubGroup



    >DD> SubGroup = _SubGroup(Group2)


    What is the difference of this with:

    class SubGroup(Group2):
    pass
    ?
    >DD> sub_group = SubGroup()


    >DD> print sub_group.origin()


    From your description I find it very difficult to understand what you
    want. But I have some feeling that multiple inheritance may be something
    you could use.
    --
    Piet van Oostrum <>
    URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
    Private email:
    Piet van Oostrum, Apr 14, 2009
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Stephan Diehl

    question about metaclasses

    Stephan Diehl, Jun 30, 2003, in forum: Python
    Replies:
    2
    Views:
    327
    Stephan Diehl
    Jun 30, 2003
  2. Simon Burton

    metaclasses

    Simon Burton, Jul 28, 2003, in forum: Python
    Replies:
    5
    Views:
    365
    Simon Burton
    Jul 30, 2003
  3. Mike C. Fletcher

    Metaclasses presentation slides available...

    Mike C. Fletcher, Aug 28, 2003, in forum: Python
    Replies:
    3
    Views:
    313
    Alex Martelli
    Aug 29, 2003
  4. =?iso-8859-1?q?Steffen_Gl=FCckselig?=

    Question on metaclasses

    =?iso-8859-1?q?Steffen_Gl=FCckselig?=, Apr 24, 2005, in forum: Python
    Replies:
    9
    Views:
    305
    Reinhold Birkenfeld
    Apr 24, 2005
  5. Laszlo Nagy

    metaclasses (beginner question)

    Laszlo Nagy, Feb 21, 2007, in forum: Python
    Replies:
    4
    Views:
    252
    James Stroud
    Feb 22, 2007
Loading...

Share This Page