ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

Discussion in 'Python' started by Aaron Watters, Dec 10, 2003.

  1. ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    Links:
    Home page with docs and other links:
    http://xsdb.sourceforge.net/
    Sourceforge project with download links
    http://sourceforge.net/projects/xsdb/

    Executive Summary:

    The xsdb package is an open source database implementation implemented
    in Python
    and hosted on SourceForge.

    The xsdb package provides fundamental concurrent database functionality
    with
    concurrency control and recovery. Fundamental characteristics include:

    - Extreme portability and ease of installation and use.
    - A simple semantics of objects with associated descriptions
    compatible with the relational model, object modelling methods,
    and other data organizations such as OLAP.
    - Multiple access paths and indices.
    - Timestamp based concurrency control for safe concurrent database access.
    - Commit/Rollback and recovery support.
    - A variety of underlying storage implementations with configurable
    features and
    performance characteristics.
    - No intrinsic database size limitations.

    The package is intended to provide what you really need from a database
    for most applications,
    without the other stuff (among other goals).

    General Technical Notes:
    The xsdb package is implemented in Python, and the server mode requires
    stackless python.
    An xsdb database (not in server mode) will run using standard C Python
    or Java Python (Jython).

    Please have a look and give it a try. Thanks very much!

    -- Aaron Watters [attempt 2]

    ===
    Even in a perfect world where everyone is equal
    I'd still own the movie rights and be working on the sequel
    -- Elvis Costello "Every day I write the book"
     
    Aaron Watters, Dec 10, 2003
    #1
    1. Advertising

  2. Aaron Watters

    Paul Rubin Guest

    Aaron Watters <> writes:
    > The xsdb package is implemented in Python, and the server mode
    > requires stackless python.


    What is it doing with stackless?
     
    Paul Rubin, Dec 10, 2003
    #2
    1. Advertising

  3. Aaron Watters wrote:

    > General Technical Notes:


    > The xsdb package is implemented in Python, and the server mode requires
    > stackless python.


    Why?

    IMHO this sort of limitations severely reduce any
    project's potential.

    Istvan.
     
    Istvan Albert, Dec 10, 2003
    #3
  4. On Wed, 10 Dec 2003 10:10:44 -0500,
    Istvan Albert <-ZZZ> wrote:
    >> The xsdb package is implemented in Python, and the server mode requires
    >> stackless python.

    > Why?


    Probably because Stackless made it easier to write the server without
    having to wrestle an async socket library such as Medusa or Twisted.

    > IMHO this sort of limitations severely reduce any
    > project's potential.


    Yeah, but it's his code, so he can do whatever he likes.

    Sheesh, Aaron announces a database system that looks really spiffy from the
    examples, like ZODB without the pain of ExtensionClass, and the first two
    responses are griping about one aspect of it. Sometimes folks don't know
    when they're well-off. (Now if only he'd released it before I wrote the
    PyCon proposal tracker using PostgreSQL...)

    --amk
     
    A.M. Kuchling, Dec 10, 2003
    #4
  5. A.M. Kuchling wrote:

    > Sheesh, Aaron announces a database system that looks really spiffy from the
    > examples, like ZODB without the pain of ExtensionClass, and the first two
    > responses are griping about one aspect of it


    But that is not just one aspect of it, it is a matter of the
    most fundamental usability question, will it work on my system?

    The answer is no, I have to install another version of python
    to use it.

    > Probably because Stackless made it easier to write the server without
    > having to wrestle an async socket library such as Medusa or Twisted.


    The question that needs to be answered is whether it would
    be worth to wrestle with those rather than locking out
    the vast majority of the potential users.

    I have a lot of respect and admiration for everyone who
    undertakes a project of this magnitude and complexity,
    on the other hand I think that tying the project to a
    python implementation that most of us have no compelling reason
    to use will severely affect its overall impact. And it would
    be a shame.

    Istvan.
     
    Istvan Albert, Dec 10, 2003
    #5

  6. >> Probably because Stackless made it easier to write the server without
    >> having to wrestle an async socket library such as Medusa or Twisted.


    Istvan> The question that needs to be answered is whether it would be
    Istvan> worth to wrestle with those rather than locking out the vast
    Istvan> majority of the potential users.

    I imagine Aaron provides the code (it is hosted on SF, after all). All you
    need to do is port it to use Twisted or Medusa, then feed the diffs back to
    Aaron. If it results in broader reach for xsdb without making the existing
    code a nightmare to maintain, he'll probably fold it in.

    Skip
     
    Skip Montanaro, Dec 10, 2003
    #6
  7. shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    Istvan Albert <-ZZZ> wrote in message news:<>...
    > Aaron Watters wrote:
    > > General Technical Notes:
    > > The xsdb package is implemented in Python, and the server mode requires
    > > stackless python.

    >
    > Why?
    > IMHO this sort of limitations severely reduce any
    > project's potential.
    > Istvan.


    Because it's the Right Way (tm) to do it :).

    First let me emphasize that only the server layer
    uses stackless at present.

    I'm using stackless because (aside from the fact that
    it was simplest way implement the functionality)
    database concurrency control requires the following:

    If a young transaction tries to read something written by
    an old transaction which has not yet committed it must wait
    until the old transaction decides to commit or abort.

    In order to allow transactions to wait the options are:

    1) Use an event loop and write the application "inside out",
    much like a fortran 4 program attempting to emulate recursion.

    2) Use operating system threads (which have very high overhead
    and sometimes don't really work the same across different
    platforms...)

    3) Use stackless.

    4) punt: automatically abort any transaction which needs
    to wait.

    As a first approach I went for (3) because it was easy. I don't
    plan to do (1) because I treasure my sanity. I intend to
    implement both (2) and (4) as server options before I call
    xsdb a "beta", but I want to also keep the stackless version alive.

    I'm still wishing that real stackless functionality will make it
    into standard Python, but I also don't really understand the
    deep implications.
    -- Aaron Watters

    ===
    I don't know if you've been loving somebody
    I only know it isn't mine. -- Elvis Costello "Alison"
     
    Aaron Watters, Dec 10, 2003
    #7
  8. Aaron Watters

    Paul Rubin Guest

    Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    (Aaron Watters) writes:
    > In order to allow transactions to wait the options are:
    >
    > 1) Use an event loop and write the application "inside out",
    > much like a fortran 4 program attempting to emulate recursion.
    > ...


    Maybe you could find some clever way to do it with Python generators.
     
    Paul Rubin, Dec 10, 2003
    #8
  9. Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely SimpleDatabase goes alpha

    Aaron Watters wrote:

    > ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha


    now, how cool is this. hugunin and watters both reappear after
    many years, on nearly the same day, both with stuff that shows
    that they didn't really give up on Python hacking; they've just
    been working on the perfect design...

    > In order to allow transactions to wait the options are:
    >
    > 1) Use an event loop and write the application "inside out",
    > much like a fortran 4 program attempting to emulate recursion.
    >
    > 2) Use operating system threads (which have very high overhead
    > and sometimes don't really work the same across different
    > platforms...)
    >
    > 3) Use stackless.
    >
    > 4) punt: automatically abort any transaction which needs
    > to wait.


    5) use an event loop and use a generator for the relevant code;
    when you discover that you need to pause, yield to the framework.

    </F>
     
    Fredrik Lundh, Dec 11, 2003
    #9
  10. Aaron Watters

    Paul Rubin Guest

    Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    "Fredrik Lundh" <> writes:
    > 5) use an event loop and use a generator for the relevant code;
    > when you discover that you need to pause, yield to the framework.


    This kind of design really could benefit from Raymond Hettinger's PEP
    of a while back, proposing being able to pass parameters from the
    caller back to the yield statement of a yielded generator.
     
    Paul Rubin, Dec 11, 2003
    #10
  11. Aaron Watters

    Robin Becker Guest

    Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    In article <>, Paul Rubin <http@?.cx>
    writes
    >"Fredrik Lundh" <> writes:
    >> 5) use an event loop and use a generator for the relevant code;
    >> when you discover that you need to pause, yield to the framework.

    >
    >This kind of design really could benefit from Raymond Hettinger's PEP
    >of a while back, proposing being able to pass parameters from the
    >caller back to the yield statement of a yielded generator.

    I was just about to ask if generators allow for a stream like mechanism,
    but obviously if we're not allowed to change the generator state then it
    seems quite hard.
    --
    Robin Becker
     
    Robin Becker, Dec 11, 2003
    #11
  12. Aaron Watters

    Paul Rubin Guest

    Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    Robin Becker <> writes:
    > I was just about to ask if generators allow for a stream like mechanism,
    > but obviously if we're not allowed to change the generator state then it
    > seems quite hard.


    Well, there's always global state.
     
    Paul Rubin, Dec 11, 2003
    #12
  13. Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    regarding the use of stackless in
    http://xsdb.sourceforge.net
    "Fredrik Lundh" <> wrote in message news:<>...
    > Aaron Watters wrote:
    > > 5) use an event loop and use a generator for the relevant code;

    > when you discover that you need to pause, yield to the framework.


    Please tell me I'm missing something, but I don't think
    this will really help. The problem is that I need to "yield"
    or "suspend" or "send something across a channel" from about
    45 places in the code some of which are arbitrarily deep into
    multiple recursions. The generator thing will only allow
    me to go one level deep into a single call -- no? By contrast
    the stackless.channel mechanism is a far more general construct,
    allowing me to "yield" at any point without restructuring the
    code at all. Stackless rules.
    -- Aaron Watters
    ===
    I'm standing in the middle of the desert
    waiting for my ship to come in
    -- Sheryl Crow "Leaving Las Vegas"
     
    Aaron Watters, Dec 11, 2003
    #13
  14. Aaron Watters

    Duncan Booth Guest

    Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    (Aaron Watters) wrote in
    news::

    > regarding the use of stackless in
    > http://xsdb.sourceforge.net
    > "Fredrik Lundh" <> wrote in message
    > news:<>...
    >> Aaron Watters wrote:
    >> > 5) use an event loop and use a generator for the relevant code;

    >> when you discover that you need to pause, yield to the framework.

    >
    > Please tell me I'm missing something, but I don't think
    > this will really help. The problem is that I need to "yield"
    > or "suspend" or "send something across a channel" from about
    > 45 places in the code some of which are arbitrarily deep into
    > multiple recursions. The generator thing will only allow
    > me to go one level deep into a single call -- no? By contrast
    > the stackless.channel mechanism is a far more general construct,
    > allowing me to "yield" at any point without restructuring the
    > code at all. Stackless rules.


    The generator solution may not be appropriate for your task, but it isn't
    entirely accurate to say that you can only go one level deep. You can (sort
    of) yield from arbitrarily deep function nesting, or even from recursive
    functions. The catch though is that you do have to write the code in a
    slightly contorted manner in order to yield from below the first function.

    The rule to follow is simply: any function which wants to yield, or which
    calls a function that wants to yield has to be a generator and has to be
    called from a 'for' loop which itself yields.

    e.g. A generator that walks a tree recursively:

    def inorder(t):
    if t:
    for x in inorder(t.left):
    yield x
    yield t.label
    for x in inorder(t.right):
    yield x


    --
    Duncan Booth
    int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3"
    "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure?
     
    Duncan Booth, Dec 11, 2003
    #14
  15. Aaron Watters

    Jp Calderone Guest

    Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely SimpleDatabase goes alpha

    On Thu, Dec 11, 2003 at 02:03:35PM +0000, Duncan Booth wrote:
    > (Aaron Watters) wrote in
    > news::
    >
    > > regarding the use of stackless in
    > > http://xsdb.sourceforge.net
    > > "Fredrik Lundh" <> wrote in message
    > > news:<>...
    > >> Aaron Watters wrote:
    > >> > 5) use an event loop and use a generator for the relevant code;
    > >> when you discover that you need to pause, yield to the framework.

    > >
    > > Please tell me I'm missing something, but I don't think
    > > this will really help. The problem is that I need to "yield"
    > > or "suspend" or "send something across a channel" from about
    > > 45 places in the code some of which are arbitrarily deep into
    > > multiple recursions. The generator thing will only allow
    > > me to go one level deep into a single call -- no? By contrast
    > > the stackless.channel mechanism is a far more general construct,
    > > allowing me to "yield" at any point without restructuring the
    > > code at all. Stackless rules.

    >
    > The generator solution may not be appropriate for your task, but it isn't
    > entirely accurate to say that you can only go one level deep. You can (sort
    > of) yield from arbitrarily deep function nesting, or even from recursive
    > functions. The catch though is that you do have to write the code in a
    > slightly contorted manner in order to yield from below the first function.
    >
    > The rule to follow is simply: any function which wants to yield, or which
    > calls a function that wants to yield has to be a generator and has to be
    > called from a 'for' loop which itself yields.
    >
    > e.g. A generator that walks a tree recursively:
    >
    > def inorder(t):
    > if t:
    > for x in inorder(t.left):
    > yield x
    > yield t.label
    > for x in inorder(t.right):
    > yield x
    >


    This works, but it is even easier. All you need is top-level code to
    handle it:


    def unroll(f, *a, **kw):
    gstack = [iter(f(*a, **kw))]
    while gstack:
    try:
    e = gstack[-1].next()
    except StopIteration:
    gstack.pop()
    else:
    if isinstance(e, types.GeneratorType):
    gstack.append(e)
    else:
    yield e


    def inorder(t):
    if t:
    yield inorder(t.left)
    yield t.label
    yield inorder(t.right)

    unroll(inorder, t)


    A bit more frameworky code, but it's all isolated in one place, which is
    much nicer than having to spread it all over the place.

    Jp
     
    Jp Calderone, Dec 11, 2003
    #15
  16. Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    Paul Rubin <http://> writes:

    > Robin Becker <> writes:
    > > I was just about to ask if generators allow for a stream like mechanism,
    > > but obviously if we're not allowed to change the generator state then it
    > > seems quite hard.

    >
    > Well, there's always global state.


    Or, if the generator is an instance method, instance state.

    Cheers,
    mwh

    --
    Academic politics is the most vicious and bitter form of politics,
    because the stakes are so low. -- Wallace Sayre
     
    Michael Hudson, Dec 11, 2003
    #16
  17. Aaron Watters

    Duncan Booth Guest

    Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    Jp Calderone <> wrote in
    news::

    > This works, but it is even easier. All you need is top-level code
    > to
    > handle it:
    >
    >
    > def unroll(f, *a, **kw):
    > gstack = [iter(f(*a, **kw))]
    > while gstack:
    > try:
    > e = gstack[-1].next()
    > except StopIteration:
    > gstack.pop()
    > else:
    > if isinstance(e, types.GeneratorType):
    > gstack.append(e)
    > else:
    > yield e
    >
    >
    > def inorder(t):
    > if t:
    > yield inorder(t.left)
    > yield t.label
    > yield inorder(t.right)
    >
    > unroll(inorder, t)
    >
    >
    > A bit more frameworky code, but it's all isolated in one place,
    > which is
    > much nicer than having to spread it all over the place.


    Nice idea, provided you never want to yield a generator. Also should it
    check for a generator, or just for any iterator.

    You can also go for a recursive definition of unroll and use it to unroll
    itself which I think reads a bit more clearly.

    def unroll(iterator):
    for v in iterator:
    if isinstance(v, types.GeneratorType)
    for inner in unroll(v): yield inner
    else:
    yield v

    for node in unroll(inorder(t)):
    ... do whatever ...

    I wonder if this is useful enough to go in itertools?

    --
    Duncan Booth
    int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3"
    "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure?
     
    Duncan Booth, Dec 11, 2003
    #17
  18. Generator state/parameters

    |Robin Becker <> writes:
    |> I was just about to ask if generators allow for a stream like mechanism,
    |> but obviously if we're not allowed to change the generator state then it
    |> seems quite hard.

    Paul Rubin <http://> wrote previously:
    |Well, there's always global state.

    But the state need not be global, just a mutable object yielded by a
    generator. As I thought about this fact, I have come to find Raymond
    Hettinger's proposals for enhancing simple generators less urgent (but I
    probably still vote +1, though now moot).

    >>> def echo():

    ... message = [None]
    ... while message[0] != "EXIT":
    ... yield message
    ...
    >>> for mess in echo():

    ... if mess[0] is not None: print mess[0]
    ... mess[0] = raw_input("Word: ")
    ...
    Word: foo
    foo
    Word: bar
    bar
    Word: EXIT

    This is a toy example, but the point is that we are perfectly able to
    pass data back into a generator without using global state.

    --
    ---[ to our friends at TLAs (spread the word) ]--------------------------
    Echelon North Korea Nazi cracking spy smuggle Columbia fissionable Stego
    White Water strategic Clinton Delta Force militia TEMPEST Libya Mossad
    ---[ Postmodern Enterprises <> ]--------------------------
     
    David Mertz, Ph.D., Dec 11, 2003
    #18
  19. Re: shy stackless Re: ANNOUNCE: xsdb -- the eXtremely Simple Database goes alpha

    Duncan Booth <> wrote in message news:<Xns944E8E75C5995duncanrcpcouk@127.0.0.1>...
    > (Aaron Watters) wrote in
    > news::
    >
    > > regarding the use of stackless in
    > > http://xsdb.sourceforge.net
    > > "Fredrik Lundh" <> wrote in message
    > > news:<>...
    > >> Aaron Watters wrote:
    > >> > 5) use an event loop and use a generator for the relevant code;
    > >> when you discover that you need to pause, yield to the framework.

    > >
    > > Please tell me I'm missing something, but I don't think
    > > this will really help. ...

    >
    > The generator solution may not be appropriate for your task, but it isn't
    > entirely accurate to say that you can only go one level deep....
    >
    > The rule to follow is simply: any function which wants to yield, or which
    > calls a function that wants to yield has to be a generator and has to be
    > called from a 'for' loop which itself yields....


    Yes. I see this would work. But this would then have to be
    pervasive throughout my code -- and even in client code that
    uses the xsdb code directly (but not from a remote client)....
    yuck. No thanks :(. The acceptible options still are threads,
    stackless, or punt.

    -- Aaron Watters
    ===
    How do zen masters walk through walls?
    Doors.
     
    Aaron Watters, Dec 12, 2003
    #19
  20. Re: Generator state/parameters

    (David Mertz, Ph.D.) wrote in message news:<>...
    > But the state need not be global, just a mutable object yielded by a
    > generator. As I thought about this fact, I have come to find Raymond
    > Hettinger's proposals for enhancing simple generators less urgent (but I
    > probably still vote +1, though now moot).
    >
    > >>> def echo():

    > ... message = [None]
    > ... while message[0] != "EXIT":
    > ... yield message
    > ...
    > >>> for mess in echo():

    > ... if mess[0] is not None: print mess[0]
    > ... mess[0] = raw_input("Word: ")
    > ...
    > Word: foo
    > foo
    > Word: bar
    > bar
    > Word: EXIT
    >
    > This is a toy example, but the point is that we are perfectly able to
    > pass data back into a generator without using global state.


    A more verbose but arguably more elegant way would be to wrap the
    generator in a class. Let me repost some code I wrote some time ago.

    """An object-oriented interface to iterators-generators"""

    class Iterator(object):
    """__gen__ is automatically called by __init__, so must have signature
    compatibile with __init__. Subclasses should not need to override __init__:
    you can do it, but you must do it cooperatively or, at least, ensure that
    __gen__ is called correctly and its value assigned to self.iterator.
    """
    def __init__(self,*args,**kw):
    super(Iterator,self).__init__(*args,**kw)
    self.iterator=self.__gen__(*args,**kw)
    def __gen__(self,*args,**kw):
    "Trivial generator, to be overridden in subclasses"
    yield None
    def __iter__(self):
    return self
    def next(self):
    return self.iterator.next()

    class MyIterator(Iterator):
    def __gen__(self):
    self.x=1
    yield self.x # will be changed outside the class
    yield self.x

    iterator=MyIterator()

    print iterator.next()
    iterator.x=5
    print iterator.next()

    Wrapping the generator in the class, I can pass parameters to it (in
    this case x). IOW, here the generator has an explicit "self" rather
    than an implicit "__self__" as in the PEP. I am not sure if I like the
    PEP, wouldn't be easier to have a built-in iterator class?

    Michele
     
    Michele Simionato, Dec 12, 2003
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. loquak

    Extremely simple XML

    loquak, Apr 11, 2004, in forum: Java
    Replies:
    5
    Views:
    436
    Sunwest Technologies
    Apr 12, 2004
  2. Replies:
    3
    Views:
    776
    Andy Dingley
    Dec 22, 2004
  3. Gregory (Grisha) Trubetskoy

    ANNOUNCE: Mod_python 3.1.0 Alpha

    Gregory (Grisha) Trubetskoy, Aug 29, 2003, in forum: Python
    Replies:
    0
    Views:
    299
    Gregory (Grisha) Trubetskoy
    Aug 29, 2003
  4. Lethalman
    Replies:
    0
    Views:
    271
    Lethalman
    Dec 24, 2004
  5. Replies:
    0
    Views:
    326
Loading...

Share This Page