asyncore, asynchat and threads

Discussion in 'Python' started by Joshua Moore-Oliva, Aug 4, 2004.

  1. Ok.. so here is my situation.

    I have an application that I would love to be able to use asynchat with for it's handle_close, found_terminator etc abstraction on.

    However, I must use a separate thread for each connection as the work resulting from the communication can result in blocking for an indefinate period of time.

    What I would love to do for each thread that is created, is to spin off an asynchat subclass and manually run some kind of loop() function that will do the
    polling//select etc for me, in a threadsafe manner, only on the one asynchat class/connection for that thread (e.g. I supply variables to keep track of the data)

    I have come across some sparse documentation that mentioned asyncore.loop() can be supplied a map of some sorts to loop through.. However, I have no idea
    how to create this map to pass to asyncore.loop, nor what thread-safe precautions (if any) I need to make.

    A simple example of a thread-safe way to create an asynchat subclass and get it to run would be great...

    even any pointers, help, ideas on how to make something like this work would be greatly appreciated as well.

    Thanks, Joshua Moore-Oliva
    Joshua Moore-Oliva, Aug 4, 2004
    #1
    1. Advertising

  2. Joshua Moore-Oliva

    Peter Hansen Guest

    Joshua Moore-Oliva wrote:

    > However, I must use a separate thread for each connection as the work
    > resulting from the communication can result in blocking for an

    indefinate
    > period of time.

    ....
    > even any pointers, help, ideas on how to make something like this work
    > would be greatly appreciated as well.


    The usual approach is to use something along the lines of a separate
    thread for the *work* rather than for the connection, or maybe a
    thread pool, and communicate with those threads from the asynchat
    thingies using Queue.Queues...

    I'm not sure we have enough information about what you really need
    to say more. You've given lots of info about what you "would love
    to do" but not enough about what you actually need to do. <wink>

    -Peter
    Peter Hansen, Aug 4, 2004
    #2
    1. Advertising


  3. > The usual approach is to use something along the lines of a separate
    > thread for the *work* rather than for the connection, or maybe a
    > thread pool, and communicate with those threads from the asynchat
    > thingies using Queue.Queues...
    >
    > I'm not sure we have enough information about what you really need
    > to say more. You've given lots of info about what you "would love
    > to do" but not enough about what you actually need to do. <wink>
    >
    > -Peter


    Ok, I will give more info, and also, you touched on a subject that I thought was impossible for my problem domain, but here goes.

    I have a very simple protocol setup as follows.. basically it is a method for retrieving data, however I must route data through a central
    connection to avoid database locking.

    A simplified version of my transaction looks like so..
    '>' means data from the client, '<' means data from the server
    > BATCH_GET

    < 1

    Right after the BATCH_GET I wait on a Queue object for data to send to the remote client.
    Depending on whether there is data available for this specific connection, this call could never return, but other connections
    could also be requesting data, and there could be data available to them.

    I currently do have a separate thread doing all the work, and it puts each threads data into a separate queue.

    The problem that I ran into with using a direct asyncore//asynchat interface is this... (hoping there is a way to work around it)

    consider the very likely possiblity that asynchat retrieves the line BATCH_GET, but there is no data to return.

    I have to not return any data until there is data to return for that specific connection. If there is some way to send data down a asynchat
    socket from another thread (thread-safely), or to somehow wake up an asynchat class, that would be great.

    The problem is that once BATCH_GET is received, unless i handle it there and then, no more events will be generated as the remote client will
    be waiting for a response.

    Hopefully this sufficiently answers your 'what I actually *need* to do' question ;)

    I don't want to flood the email with a lot of irrelevant info, so if you would like more infomation, ask away, but I think this covers the problem sufficiently...

    Joshua Moore-Oliva
    Joshua Moore-Oliva, Aug 4, 2004
    #3
  4. Joshua Moore-Oliva

    Peter Hansen Guest

    Joshua Moore-Oliva wrote:

    > I have to not return any data until there is data to
    > return for that specific connection. If there is some
    > way to send data down a asynchat
    > socket from another thread (thread-safely), or to
    > somehow wake up an asynchat class, that would be great.


    It's been years since I used asyncore (I find Twisted much
    simpler to use) but at least in theory such a thing is
    possible. Twisted has a callFromThread() method that can
    be used to get a callback in another thread to be called
    from the "reactor" (the equivalent of the asyncore core
    loop), which would do what you want. Asyncore might have
    similar facilities.

    I don't _fully_ grasp what you are doing, and I don't work
    well without pictures in such cases (got a whiteboard handy?
    draw a picture and send me a digital photo :), but so
    far it doesn't sound like you have anything approaching
    unusual, compared to many other applications I've seen doing
    the sorts of things yours seems to be doing.

    -Peter
    Peter Hansen, Aug 4, 2004
    #4
  5. Well, I took at look at twisted.. and it seems pretty scary ;)

    What I've done though is made some very minor changes to asyncore and asynchat, and now it appears to be threadsafe//more controllable.

    If all works out well for me I'll submit the patch to python, since it's fully backwards compatbile with the old behaviour.

    Thanks for the help

    Joshua Moore-Oliva

    On August 4, 2004 08:10 am, Peter Hansen wrote:
    > Joshua Moore-Oliva wrote:
    >
    > > I have to not return any data until there is data to
    > > return for that specific connection. If there is some
    > > way to send data down a asynchat
    > > socket from another thread (thread-safely), or to
    > > somehow wake up an asynchat class, that would be great.

    >
    > It's been years since I used asyncore (I find Twisted much
    > simpler to use) but at least in theory such a thing is
    > possible. Twisted has a callFromThread() method that can
    > be used to get a callback in another thread to be called
    > from the "reactor" (the equivalent of the asyncore core
    > loop), which would do what you want. Asyncore might have
    > similar facilities.
    >
    > I don't _fully_ grasp what you are doing, and I don't work
    > well without pictures in such cases (got a whiteboard handy?
    > draw a picture and send me a digital photo :), but so
    > far it doesn't sound like you have anything approaching
    > unusual, compared to many other applications I've seen doing
    > the sorts of things yours seems to be doing.
    >
    > -Peter
    Joshua Moore-Oliva, Aug 4, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michael Welsh

    asyncore asynchat

    Michael Welsh, Jul 15, 2003, in forum: Python
    Replies:
    0
    Views:
    515
    Michael Welsh
    Jul 15, 2003
  2. F.G.Testa

    asyncore/asynchat

    F.G.Testa, Nov 14, 2003, in forum: Python
    Replies:
    4
    Views:
    426
    MetalOne
    Nov 17, 2003
  3. Replies:
    5
    Views:
    593
    Eric Brunel
    Feb 12, 2004
  4. Z. Kotzer
    Replies:
    0
    Views:
    341
    Z. Kotzer
    Mar 15, 2006
  5. David Hirschfield

    Help: asyncore/asynchat and terminator string

    David Hirschfield, Jan 16, 2007, in forum: Python
    Replies:
    0
    Views:
    316
    David Hirschfield
    Jan 16, 2007
Loading...

Share This Page