asynchat and threading

[Rob Snyder]

Greetings -

I've embarked on a small learning project, trying to get a better grasp of
the asyncore and asynchat modules. The project involves building a simple
instant messenger application for use on our LAN - nothing heavy duty, as
99% of my reason for doing this is just to learn.

I got my server portion working well, and I think I understand the basics.
Where I'm getting stuck is on the client side, where I'm trying to involve
both the asyncore / asynchat stuff in a multithreaded app.

Assume for this that I have a window open with an active chat session, and
an open connection to the server. At any moment in time, a message coule be
received on the connection from the server, or the user could start typing
their own message (and ultimately press enter to send it). Or, for that
matter, the user could close the window or resize it, etc.

I've got to be listening for incoming messages while also letting the UI run
to handle events, while checking the event queue for things I care about
(such as the enter key). I may also, in theory, still be in the process of
sending a prior message out.

I noticed early on that once I call asyncore.loop(), I surrender control to
the asyncore library. I don't want to get into a polling situation, so I
like that asyncore has taken over and is effectively waiting on incoming or
outgoing data.

Since I still need to run my UI, though, started a seperate thread for the
asyncore stuff. I figured I would establish a queue that the asyncore could
use to deliver messages to the UI, and I would just have the UI call the
"push" method to get messages sent.

Then, of course, the ugliness ensued. First, I can't have the UI wait on
both the UI's event queue and the queue I established for communication, so
I'd have to go back to polling between the two. Second, after I read
through the asynchat module, I realized that the producer fifo is just a
list, and when I call push, not only is there no inherent synchronization,
I would actually invoking the code to deliver the message in the UI thread
(so now I have two threads running through the same instance of my
async_chat object, which seems dangerous).

There *must* be some way to use this asyncore stuff on the client side, but
I'm not seeing it. I'm about to bail on it for the client, but I thought
I'd put this out there to see if anyone can help me see a different
approach to this stuff.

Thanks for the help!

Rob

 

[Anthony Baxter]

Greetings -

I've embarked on a small learning project, trying to get a better grasp of
the asyncore and asynchat modules. The project involves building a simple
instant messenger application for use on our LAN - nothing heavy duty, as
99% of my reason for doing this is just to learn.

I can only paraphrase what I said in a recent python-dev thread -
absolutely *no-one* should be starting new projects using asyncore and
asynchat. Use twisted, instead - it's cleaner, saner, has an active
development community, and is in all ways a better solution than the
asyncore approach.

asyncore based code causes bleeding from the eyes and brain leakage
through the ears.

 

[Thomas Guettler]

Am Thu, 25 Nov 2004 02:31:17 +1100 schrieb Anthony Baxter:

I can only paraphrase what I said in a recent python-dev thread -
absolutely *no-one* should be starting new projects using asyncore and
asynchat. Use twisted, instead - it's cleaner, saner, has an active
development community, and is in all ways a better solution than the
asyncore approach.

asyncore based code causes bleeding from the eyes and brain leakage
through the ears.

Hi,

then the core of twisted should be in the standard library of python,
and asyncore should become deprecated.

This could save a lot of people a lot of time.

Thomas

 

[Rob Snyder]

Am Thu, 25 Nov 2004 02:31:17 +1100 schrieb Anthony Baxter:


Hi,

then the core of twisted should be in the standard library of python,
and asyncore should become deprecated.

This could save a lot of people a lot of time.

Thomas

Well, Anthony, that explains the blood on my keyboard, then. I am
downloading Twisted now, and will work through the tutorial this afternoon.
At least I feel like my pain was not entirely of my own creation!

As for making it into the standard library - I'm not a fan of putting larger
frameworks, which twisted seems to be - into a standard library. But I
agree with the spirit of your post, Thomas - is there some (reasonably)
comprehensive list of Python development frameworks and non-integrated
libraries? I'm did a lot of google'ing for stuff related to this any never
came across the Twisted Matrix site...

Rob

 

[Michele Simionato]

I can only paraphrase what I said in a recent python-dev thread -
absolutely *no-one* should be starting new projects using asyncore and
asynchat. Use twisted, instead - it's cleaner, saner, has an active
development community, and is in all ways a better solution than the
asyncore approach.

asyncore based code causes bleeding from the eyes and brain leakage
through the ears.

Well, I had just started playing with asynchat, and I would like to
know
more about its shortcomings. Are you saying there are bugs in its
implementation in the standard library? What exactly can go wrong?
Or it is just ill designed and unconvenient to use?

If it is really that bad it should be removed, since I usually assume
that the stuff in the standard library is highly tested and stable.

Michele Simionato

 

[Elbert Lev]

Rob!

Sure it was suggested you may go with twisted. But with minimal
modifications of asyncore you can add a hook.
There are 2 ways to do this:
1. Use small timeout
2. Use a "control socket"

Timeout:

from asyncore import *

def loop(timeout=30.0, use_poll=0, map=None, after_pool = None):
if map is None:
map = socket_map

if use_poll:
if hasattr(select, 'poll'):
poll_fun = poll3
else:
poll_fun = poll2
else:
poll_fun = poll

while map:
poll_fun(timeout, map)
if (after_pool):
after_pool(map)

after_pool - is a function called when you exit poll_fun. You can do
whatever you want in this function.
If you call loop with timeout 0.3 - for all practical reasons your GUI
will react OK. If there are a few sockets (I think 1 in your case),
the overhead will be negligible on any reasonable computer.

The second approach is more effective, but more complex too. You add a
socket (as channel 0) and when something happens (f.e. user presses
enter), you write a byte (or any command you want). Write into this
control socket has to be made atomic to be thread save.

The reason I do not use twisted for client applications is the size of
distribution file. I do not assume python is installed on client PC.
I use distutils with py2exe option. If I use twisted the size is
10mb+. With asyncore - 2-3 mb.

Hope this helps.

 

[Jarek Zgoda]

I can only paraphrase what I said in a recent python-dev thread -
absolutely *no-one* should be starting new projects using asyncore and
asynchat. Use twisted, instead - it's cleaner, saner, has an active
development community, and is in all ways a better solution than the
asyncore approach.

asyncore based code causes bleeding from the eyes and brain leakage
through the ears.

Well, when writing simple (when I write "simple" I mean "really simple")
server or client for some simple ("really simple") protocol, Twisted may
be big overhead. I remember somebody said that if you need multiprotocol
application server then go with Twisted, but if you need client or
server for simple conversational protocol (like HTTP or NNTP),
asyncore/asynchat should be sufficient.

That doesn't mean I downplay Twisted. I really like it.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Well, I had just started playing with asynchat, and I would like to
know
more about its shortcomings. Are you saying there are bugs in its
implementation in the standard library? What exactly can go wrong?
Or it is just ill designed and unconvenient to use?

AFAICT, the main complaint is that it is tied to the TCP transport,
the sockets API, and the select/poll API. IOW, it cannot easily:
- integrate TLS on top of TCP (because TLS might involve no-data
communications, e.g. when TLS negotation happens in the middle
of a TLS session),
- integrate other transports, such as serial ports on Windows,
- integrate other multi-endpoint wait APIs, such as IO completion
ports on Windows NT+, or kqueue on BSD.
It appears that the first limitation is considered the most serious
one, as it doesn't allow to implement networking protocols
independent of the transport protocol (which reportedly is possible
with twisted).

Regards,
Martin

 

[Anthony Baxter]

AFAICT, the main complaint is that it is tied to the TCP transport,
the sockets API, and the select/poll API. IOW, it cannot easily:
- integrate TLS on top of TCP (because TLS might involve no-data
communications, e.g. when TLS negotation happens in the middle
of a TLS session),
- integrate other transports, such as serial ports on Windows,
- integrate other multi-endpoint wait APIs, such as IO completion
ports on Windows NT+, or kqueue on BSD.

- work with other event loops, such as GUI toolkits

There's also the problem that asyncore is more or less orphaned - there's
no-one on the python development team who supports it fully. Bugs in
it tend to get treated poorly.

I wrote the original transport layer for pythondirector with asyncore. Under
any sort of load, it just starts to lose, bigtime. Random errors,
sockets getting
lost, and the like. Switching to twisted made it go much, much faster. Well,
to be fair, when I first ported to twisted, there was a nasty performance
problem - this was relatively easy to track down and fix, and it's been fixed
for a long time now. Trying to fix asyncore was a nightmare.

The other thing to recommend twisted over asyncore is that twisted provides
a fuller set of tools for programming in an async manner, such as Deferreds..
With asyncore, you're kinda left on your own to do all this yourself.

twisted is too large to go into the python core as is - in addition, there's a
mismatch between Python's release cycle speed and twisted's release
cycle speed (although the current delayed-until-who-knows-when Twisted
2.0 might be a harbinger of twisted slowing down to Python's speed <wink>)
It's possible that in the future a small core of twisted could end up in the
stdlib - this topic's come up a couple of times on the twisted list (most
recently, a couple of months ago).

I wonder if the asyncore docs couldn't grow a 'See also' that points at twisted,
though...

 

[Michele Simionato]

<snip a lot of interesting comments>

I have the following problem, and I would like to know if asyncore or
twisted could be of help here, and which one you would suggest.
I have a few users (say 100) who want to send small bunches of text
(say 1K in size) to a remote server via TCP; the server should save
these texts in a small database. The server should be able to catch the
messages from the different users and "synchronize" them (not sure what the
right word is): if two messages arrive at the same time they are written one
after the other to the database. In this way the database has not to
worry about concurrency issues and I could just use shelve; the
messages will be stored as a mapping usernames -> messages. I would
rather NOT use a "real" database, since the load is very
small and I do not want to install and maintain a database server in
the remote machine for something so simple. The order of arrival of the
messages is not important. The message has to be stored when the termination
char arrives, so I was thinking about using asynchat. But maybe twisted has
already a pre-built solution for this problem.

Any hints?


Michele Simionato

 

[Diez B. Roggisch]

twisted could be of help here, and which one you would suggest.

I have a few users (say 100) who want to send small bunches of text
(say 1K in size) to a remote server via TCP; the server should save
these texts in a small database. The server should be able to catch the
messages from the different users and "synchronize" them (not sure what
the right word is): if two messages arrive at the same time they are
written one after the other to the database. In this way the database has

If you use twisted with the select-based reactor (which is default) this
should be no problem at all. As there is only one process running, there
can't be a possible concurrent database access. Apart from that, you could
always explicitely guard your critical section of code with a
threading-lock - then you're free to use the thread-based reactor.

 

[Ng Pheng Siong]

AFAICT, the main complaint is that it is tied to the TCP transport,
the sockets API, and the select/poll API. IOW, it cannot easily:
- integrate TLS on top of TCP (because TLS might involve no-data
communications, e.g. when TLS negotation happens in the middle
of a TLS session),

M2Crypto does SSL/TLS over Medusa/asyncore since 1999. ZServerSSL - SSL for
Zope, which is Medusa-based - has been available since 1999 or 2000.

I don't imagine a pure Python TLS implementation is impossible to layer
over asyncore either.

Cheers.

 

[Eric S. Johansson]

twisted is too large to go into the python core as is - in addition, there's a
mismatch between Python's release cycle speed and twisted's release
cycle speed (although the current delayed-until-who-knows-when Twisted
2.0 might be a harbinger of twisted slowing down to Python's speed <wink>)
It's possible that in the future a small core of twisted could end up in the
stdlib - this topic's come up a couple of times on the twisted list (most
recently, a couple of months ago).

I wonder if the asyncore docs couldn't grow a 'See also' that points at twisted,
though...

this points to an important issue that should be addressed at some point
which is the ability to quickly and easily, on the level of rpm or
apt-get, import external frameworks. I have a body of code which counts
on smtpd.py and pop3d.py. I have modified both and just created new
versions to sit within my application. Easy integration.

If I was to count on twisted, I would have another fetch fest collecting
all the pieces as well as increasing the complexity of building my
application.

If I could simply do: py-get twisted
and have all of the dependencies resolved, it would be wonderful. as a
prototype, I would suggest wrapping apt-get with a python envelope which
automatically invokes apt-get with a python specific configuration
file for .pydeb files as well as a private repository and database etc..
Since apt-get exists for almost every version of Linux and should be
portable to other platforms, this distribution method should be
relatively distribution portable.

One can also handle upgrades of the core image of Python and ancillary
packages as well by leaving a breadcrumb file indicating which packaging
system is native and the package currently installed. <<- idea needs work

yea? nay?

---eric

 

[Caleb Hattingh]

I heartily support something like this, but alas I have not the time to
help out with it. I like the Enthought python distribution because it
installs several packages in one shot. A pity there isn't a similar thing
for python addons in Linux (or is there?).

Something apt-get-like would be cool. Was there talk of having the python
package index support something like this at some stage?

 

[Michele Simionato]

this points to an important issue that should be addressed at some point
which is the ability to quickly and easily, on the level of rpm or
apt-get, import external frameworks.

Depending on your distribution, this may be already available. For instance
in Mandrake 10.1 I just tried

# urpmi python-twisted

and it downloaded twisted 1.3. Seems to work too.


Michele Simionato

 

[Anthony Baxter]

I heartily support something like this, but alas I have not the time to
help out with it. I like the Enthought python distribution because it
installs several packages in one shot. A pity there isn't a similar thing
for python addons in Linux (or is there?).

Plenty of people would like to see this. Alas, no-one has stepped forward
to do the work (or, if you have money but no time, to offer to fund the work).

Python runs on volunteers - no-one's done this yet, so it's not happened.

 

[Fredrik Lundh]

- work with other event loops, such as GUI toolkits

sure can.

but sure, if you don't want it to work, nothing stops you from writing
crappy code. we've been using asyncore/medusa on high-performance
servers for nearly 10 years; I think we'd noticed by now if the library
was seriously broken.

</F>