asyncore, asynchat and threads

  • Thread starter Joshua Moore-Oliva
  • Start date
J

Joshua Moore-Oliva

Ok.. so here is my situation.

I have an application that I would love to be able to use asynchat with for it's handle_close, found_terminator etc abstraction on.

However, I must use a separate thread for each connection as the work resulting from the communication can result in blocking for an indefinate period of time.

What I would love to do for each thread that is created, is to spin off an asynchat subclass and manually run some kind of loop() function that will do the
polling//select etc for me, in a threadsafe manner, only on the one asynchat class/connection for that thread (e.g. I supply variables to keep track of the data)

I have come across some sparse documentation that mentioned asyncore.loop() can be supplied a map of some sorts to loop through.. However, I have no idea
how to create this map to pass to asyncore.loop, nor what thread-safe precautions (if any) I need to make.

A simple example of a thread-safe way to create an asynchat subclass and get it to run would be great...

even any pointers, help, ideas on how to make something like this work would be greatly appreciated as well.

Thanks, Joshua Moore-Oliva
 
P

Peter Hansen

Joshua said:
However, I must use a separate thread for each connection as the work
> resulting from the communication can result in blocking for an indefinate
> period of time. ....
even any pointers, help, ideas on how to make something like this work
> would be greatly appreciated as well.

The usual approach is to use something along the lines of a separate
thread for the *work* rather than for the connection, or maybe a
thread pool, and communicate with those threads from the asynchat
thingies using Queue.Queues...

I'm not sure we have enough information about what you really need
to say more. You've given lots of info about what you "would love
to do" but not enough about what you actually need to do. <wink>

-Peter
 
J

Joshua Moore-Oliva

The usual approach is to use something along the lines of a separate
thread for the *work* rather than for the connection, or maybe a
thread pool, and communicate with those threads from the asynchat
thingies using Queue.Queues...

I'm not sure we have enough information about what you really need
to say more. You've given lots of info about what you "would love
to do" but not enough about what you actually need to do. <wink>

-Peter

Ok, I will give more info, and also, you touched on a subject that I thought was impossible for my problem domain, but here goes.

I have a very simple protocol setup as follows.. basically it is a method for retrieving data, however I must route data through a central
connection to avoid database locking.

A simplified version of my transaction looks like so..
'>' means data from the client, '<' means data from the server
BATCH_GET
< 1

Right after the BATCH_GET I wait on a Queue object for data to send to the remote client.
Depending on whether there is data available for this specific connection, this call could never return, but other connections
could also be requesting data, and there could be data available to them.

I currently do have a separate thread doing all the work, and it puts each threads data into a separate queue.

The problem that I ran into with using a direct asyncore//asynchat interface is this... (hoping there is a way to work around it)

consider the very likely possiblity that asynchat retrieves the line BATCH_GET, but there is no data to return.

I have to not return any data until there is data to return for that specific connection. If there is some way to send data down a asynchat
socket from another thread (thread-safely), or to somehow wake up an asynchat class, that would be great.

The problem is that once BATCH_GET is received, unless i handle it there and then, no more events will be generated as the remote client will
be waiting for a response.

Hopefully this sufficiently answers your 'what I actually *need* to do' question ;)

I don't want to flood the email with a lot of irrelevant info, so if you would like more infomation, ask away, but I think this covers the problem sufficiently...

Joshua Moore-Oliva
 
P

Peter Hansen

Joshua said:
I have to not return any data until there is data to
> return for that specific connection. If there is some
> way to send data down a asynchat
socket from another thread (thread-safely), or to
> somehow wake up an asynchat class, that would be great.

It's been years since I used asyncore (I find Twisted much
simpler to use) but at least in theory such a thing is
possible. Twisted has a callFromThread() method that can
be used to get a callback in another thread to be called
from the "reactor" (the equivalent of the asyncore core
loop), which would do what you want. Asyncore might have
similar facilities.

I don't _fully_ grasp what you are doing, and I don't work
well without pictures in such cases (got a whiteboard handy?
draw a picture and send me a digital photo :), but so
far it doesn't sound like you have anything approaching
unusual, compared to many other applications I've seen doing
the sorts of things yours seems to be doing.

-Peter
 
J

Joshua Moore-Oliva

Well, I took at look at twisted.. and it seems pretty scary ;)

What I've done though is made some very minor changes to asyncore and asynchat, and now it appears to be threadsafe//more controllable.

If all works out well for me I'll submit the patch to python, since it's fully backwards compatbile with the old behaviour.

Thanks for the help

Joshua Moore-Oliva
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top