zodb troubles - seeking advice for app design

Discussion in 'Python' started by Diez B. Roggisch, May 6, 2004.

  1. Hi,

    the docs on zodb are spare, so I try my luck in this ng.

    I'm currently developing a application server that exposes its functionality
    using corba/onmiORB. The basic purpose of this app is to take a chunk of
    text, classify it using the crm114 text classificator and return the
    probabilities of the text belonging to a certain class for all defined
    classes.

    Now as the classification is rather expensive (around 400ms on my current
    machine) I want to store the text chunk in a zodb and serve the
    probabilities from there the next time. Another reason for this is that for
    training the crm114, I need a base of items to rely upon. And if an item
    gets classified wrong, a manual override must be possible.


    Everything works fine - in a single threaded app. But under load, the
    omniorb will dispatch the incoming calls on several worker threads. zodb
    requires that for each thread a separate connection has to be used. I'm not
    sure how bad sharing the connection would be, but can imagine that this
    isn't the best idea....

    Now my problem is that the data object instances are separate for each
    connection. So changes made in one thread aren't reflected in the instances
    of other threads.

    I've created some metaclass-magic to create proxy-objects that will access a
    data-item created local to each thread - that works in a test, but means
    opening and closing the db for each incoming call. No idea how that affects
    performanec.

    And in the app server itself I end up having broken persistent instances - a
    problem google also doesn't tell me much about.

    Now I'm somewhat lost on how to actually implement my server - I could share
    the connection and data objects amongst my threads and commit changes on a
    regular basis in a background worker thread that locks the connection for
    the needed time. That makes me lose all transactional benefits. Or I follow
    my already started path, and try to access all data transparently through
    proxies - but the connection-handling-stuff gets tough. Another way might
    be to share one connection for every user over several calls and thus
    threads, but the base problem remains: how to synchronize the object graph.

    Any advices? I also thought about ditching zodb for postgres, but its my
    current believe that the actual problems remain.

    No idea if you people out there can help me - but I have to admit that I'm
    quite lost, so maybe someone can prod me in the right direction or show me
    that my whole idea is flowed and I can go for something totally
    different....

    --
    Regards,

    Diez B. Roggisch
     
    Diez B. Roggisch, May 6, 2004
    #1
    1. Advertising

  2. Diez B. Roggisch

    Lee Harr Guest

    On 2004-05-06, Diez B. Roggisch <> wrote:
    > Hi,
    >
    > the docs on zodb are spare, so I try my luck in this ng.
    >


    Sounds like a good question for the zope list:
     
    Lee Harr, May 6, 2004
    #2
    1. Advertising

  3. In article <>,
    Diez B. Roggisch <> wrote:

    >Everything works fine - in a single threaded app. But under load, the
    >omniorb will dispatch the incoming calls on several worker threads. zodb
    >requires that for each thread a separate connection has to be used. I'm not
    >sure how bad sharing the connection would be, but can imagine that this
    >isn't the best idea....
    >
    >Now my problem is that the data object instances are separate for each
    >connection. So changes made in one thread aren't reflected in the instances
    >of other threads.


    If you want to stick with zodb, one option is to use a POA with the
    main thread model threading policy. That will dispatch all calls with
    the main thread, so your access to zodb will be single threaded. Of
    course, that will kill the performance gain you would have had from
    the threads.

    If you're not set on zodb, another option is to use Berkeley DB. That
    works very nicely in a multi-threaded environment with omniORB.

    Cheers,

    Duncan.

    --
    -- Duncan Grisby --
    -- --
    -- http://www.grisby.org --
     
    Duncan Grisby, May 7, 2004
    #3
  4. "Diez B. Roggisch" <> writes on Thu, 06 May 2004 14:16:10 +0200:
    > the docs on zodb are spare, so I try my luck in this ng.


    You did not look well enough!

    There is the "zodb3.pdf" document and a "ZODB guide" (maybe "ZODB
    tutorial"). At lease they cover your essential question...

    > ...
    > Everything works fine - in a single threaded app. But under load, the
    > omniorb will dispatch the incoming calls on several worker threads. zodb
    > requires that for each thread a separate connection has to be used. I'm not
    > sure how bad sharing the connection would be, but can imagine that this
    > isn't the best idea....


    It is very bad!

    The ZODB does not provide a locking facility.
    It would need to if several threads could change concurrently
    the same data. This would drastically complicate ZODB usage.

    Instead, the ZODB gives each connection its own (partial) copy
    of the ZODB content. Threads work on this copy only.
    The ZODB expects that no two threads access the same connection
    at the same time. If this condition is meet, then different threads never
    modify the same data.

    Of course, different threads can modify different copies of
    the same ZODB object. However, the ZODB will recognize such
    a fact during the (second) commit and raise a ConflictError
    in this case.

    As you see, different threads *MUST NOT* use the same connection.
    Open a new connection for each thread (and close it before
    the thread dies).


    Dieter
     
    Dieter Maurer, May 7, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. RJ
    Replies:
    1
    Views:
    444
    Scott Allen
    Aug 22, 2004
  2. bill
    Replies:
    3
    Views:
    697
  3. iksrazal
    Replies:
    2
    Views:
    360
    enrique
    Apr 27, 2005
  4. Steve W. Jackson

    Seeking class hierarchy advice

    Steve W. Jackson, Apr 26, 2006, in forum: Java
    Replies:
    1
    Views:
    288
    Chris Uppal
    Apr 27, 2006
  5. Dave
    Replies:
    2
    Views:
    381
    David Harmon
    Apr 13, 2004
Loading...

Share This Page