Questions about asyncore

[Frank Millman]

Hi all

I have been using my own home-brewed client/server technique for a
while, using socket and select. It seems to work ok. The server can
handle multiple clients. It does this by creating a new thread for
each connection. Each thread runs its own select loop.

I am making some fairly big changes, so I thought I would look at
asyncore. I modified my program to use asyncore without much trouble,
and it feels nicer. It uses async I/O instead of threading, and it
relieves me of having to run my own select loop.

I have two questions. They both relate to whether I am using the
module as intended. The documentation is rather sparse, and I know
from experience that just getting something working is no guarantee
that I am getting the full benefit.

Firstly, having got asyncore working, I had a look at asynchat. As far
as I can see I get very little benefit from using it. I have already
set up a 'messaging' protocol between server and client, where all
messages consist of 5 digits for the message length, followed by the
message. The message consists of a pickled tuple, where the first
element is a message identifier, and the rest is the message body.

This is how it works in asyncore -

def __init__(self,channel):
[...]
self.msgLength = 0
self.recvData = '' # temporary buffer

def handle_read(self):
self.recvData += self.recv(8192)
if not self.msgLength:
if len(self.recvData) < 5: # 1st 5 bytes = msg length
return
self.msgLength = int(self.recvData[:5])
self.recvData = self.recvData[5:]
if len(self.recvData) < self.msgLength:
return
data = loads(self.recvData[:self.msgLength])
self.recvData = self.recvData[self.msgLength:]
self.msgLength = 0
[handle data]

This is how I got it working in asynchat -

def __init__(self,channel):
[...]
self.recvData = '' # temporary buffer
self.set_terminator(5)
self.gotMsgLength = False

def collect_incoming_data(self, data):
self.recvData += data

def found_terminator(self):
if self.gotMsgLength: # what follows is the message
data = loads(self.recvData)
self.set_terminator(5)
self.gotMsgLength = False
[handle data]
else: # what follows is the message length
self.set_terminator(int(self.recvData))
self.gotMsgLength = True
self.recvData = ''

It may be slightly neater, but it does not seem worth adding an extra
layer just for that. Does asynchat give me any other benefits I may
have overlooked?

My second question relates to writing a dummy client program to test
the server. I just want to send it some messages and print the
responses. Some messages incorporate data extracted from previous
responses, so I have to wait for the reply before continuing.

I am using asyncore for this as well. However, the only way I can
think of to get it working is to run asyncore.dispatcher in a separate
thread.

To send messages, I append them to a list of messages to be sent. The
dispatcher method writable() returns True if there is anything in the
list, else False.

To receive messages, I run a 'while 1' loop in the main thread, with a
sleep of 0.1, until recvData has something in it.

It works, but it seems odd to use a separate thread, as one of the
points of asyncore is to avoid multi-threading. Is there a better way
to write the client program?

Thanks

Frank Millman

 

[Giampaolo Rodola']

Firstly, having got asyncore working, I had a look at asynchat. As far
as I can see I get very little benefit from using it. I have already
set up a 'messaging' protocol between server and client, where all
messages consist of 5 digits for the message length, followed by the
message. The message consists of a pickled tuple, where the first
element is a message identifier, and the rest is the message body.

This is how it works in asyncore -

def __init__(self,channel):
[...]
self.msgLength = 0
self.recvData = '' # temporary buffer

def handle_read(self):
self.recvData += self.recv(8192)
if not self.msgLength:
if len(self.recvData) < 5: # 1st 5 bytes = msg length
return
self.msgLength = int(self.recvData[:5])
self.recvData = self.recvData[5:]
if len(self.recvData) < self.msgLength:
return
data = loads(self.recvData[:self.msgLength])
self.recvData = self.recvData[self.msgLength:]
self.msgLength = 0
[handle data]

This is how I got it working in asynchat -

def __init__(self,channel):
[...]
self.recvData = '' # temporary buffer
self.set_terminator(5)
self.gotMsgLength = False

def collect_incoming_data(self, data):
self.recvData += data

def found_terminator(self):
if self.gotMsgLength: # what follows is the message
data = loads(self.recvData)
self.set_terminator(5)
self.gotMsgLength = False
[handle data]
else: # what follows is the message length
self.set_terminator(int(self.recvData))
self.gotMsgLength = True
self.recvData = ''

It may be slightly neater, but it does not seem worth adding an extra
layer just for that. Does asynchat give me any other benefits I may
have overlooked?

The benefit of asynchat is that it automatically handles the buffering
of both input and output.
Aside from set/found_terminator() the other two methods you could want
to look at are push() and push_with_producer().
push() is a buffered version of asyncore.send(), push_with_producer()
accepts a data-producer object you can use in case you want to deal
with something other than strings (e.g. files, lists, generators, and
so on...).

My second question relates to writing a dummy client program to test
the server. I just want to send it some messages and print the
responses. Some messages incorporate data extracted from previous
responses, so I have to wait for the reply before continuing.

I am using asyncore for this as well. However, the only way I can
think of to get it working is to run asyncore.dispatcher in a separate
thread.

To send messages, I append them to a list of messages to be sent. The
dispatcher method writable() returns True if there is anything in the
list, else False.

To receive messages, I run a 'while 1' loop in the main thread, with a
sleep of 0.1, until recvData has something in it.

It works, but it seems odd to use a separate thread, as one of the
points of asyncore is to avoid multi-threading. Is there a better way
to write the client program?

I'm not sure to understand but I doubt you have to use a thread.
If you "have to wait for the reply before continuing" just implement
this logic into handle_read() or found_terminator() method.


--- Giampaolo
http://code.google.com/p/pyftpdlib/

 

[Frank Millman]

On 29 Lug, 13:09, Frank Millman <[email protected]> wrote:

Thanks for the reply, Giampaolo.

The benefit of asynchat is that it automatically handles the buffering
of both input and output.
Aside from set/found_terminator() the other two methods you could want
to look at are push() and push_with_producer().
push() is a buffered version of asyncore.send(), push_with_producer()
accepts a data-producer object you can use in case you want to deal
with something other than strings (e.g. files, lists, generators, and
so on...).

I looked at push() and push_with_producer(). To be honest I don't
fully understand why I would want to use them yet - I will have to go
over them a few more times. However, my handle_write() method seems to
be working ok and is not complicated, so I will stick with that for
now.

I'm not sure to understand but I doubt you have to use a thread.
If you "have to wait for the reply before continuing" just implement
this logic into handle_read() or found_terminator() method.

Maybe I am a bit slow, but I cannot figure out how to do this without
adding a lot of complication. I will try to give a simple example.

My server is a database server. It sits between the actual database
and the client, and implements access control, automatic handling of
foreign keys, etc. It accepts messages to read, update, and write
data, and returns the results.

For testing purposes, I want the client to send and receive messages
such as the following (pseudocode) -

-> Read Customer record with CustNo = 'A001'.
<- Print data, check that it is correct. [1]
-> Read customer's Branch record.
<- Print data, check that it is correct.
-> Update Customer record with new Branch code. [2]
-> Read Branch code from Customer record.
<- Print code, check that it has changed.
-> Read customer's Branch record.
<- Print data, check that it belongs to the new Branch.

[1] Amongst other things, the server returns the id of the record
[2] The update request uses the record id to identify which record to
update

These are just examples of the tests I might want to throw at the
server. With my multi-threaded approach, the asyncore loop runs in the
background, and in the foreground I can easily send any message I like
and check the results. I cannot see how to implement this using
handle_read() and found_terminator().

Maybe you can give a simple example of an alternative approach.

Thanks

Frank

 

[Giampaolo Rodola']

Thanks for the reply, Giampaolo.

Glad to help.

I looked at push() and push_with_producer(). To be honest I don't
fully understand why I would want to use them yet - I will have to go
over them a few more times. However, my handle_write() method seems to
be working ok and is not complicated, so I will stick with that for
now.

I don't know whether you need to use them. I've just introduced the
benefits of asynchat over asyncore.
If you're already ok with your handle_read() implementation you
probably don't need anything different.

Maybe I am a bit slow

Or maybe my English is not good enough to properly understand what you
need (it wouldn't be the first time, after all... =)).

...but I cannot figure out how to do this without
adding a lot of complication. I will try to give a simple example.

My server is a database server. It sits between the actual database
and the client, and implements access control, automatic handling of
foreign keys, etc. It accepts messages to read, update, and write
data, and returns the results.

For testing purposes, I want the client to send and receive messages
such as the following (pseudocode) -

-> Read Customer record with CustNo = 'A001'.
<- Print data, check that it is correct. [1]
-> Read customer's Branch record.
<- Print data, check that it is correct.
-> Update Customer record with new Branch code. [2]
-> Read Branch code from Customer record.
<- Print code, check that it has changed.
-> Read customer's Branch record.
<- Print data, check that it belongs to the new Branch.

[1] Amongst other things, the server returns the id of the record
[2] The update request uses the record id to identify which record to
update

These are just examples of the tests I might want to throw at the
server. With my multi-threaded approach, the asyncore loop runs in the
background, and in the foreground I can easily send any message I like
and check the results. I cannot see how to implement this using
handle_read() and found_terminator().

Maybe you can give a simple example of an alternative approach.

Thanks

Frank

I pretty much have the same overview I had before.
As far as I can tell the only reason you want to use a thread is when
you have to do something which requires a consistent amount of time to
complete so that the asyncore loop gets blocked.
The question is: is there anything like that in your code?
If the answer is no then you don't need to use threads.
Maybe you are just not used to the asynchronous approach where "wait
for server to respond" or "wait for the response to be complete"
doesn't mean that YOU have to wait.
It means that when a specific condition occurs (e.g. some data is
received) a certain method of the framework you're using will be
called and then it will be up to you deciding what do to.
For example, if you're supposed to do something when a string of 5
digits is received, overwrite the handle_read() method and check if
that's happened.
If not instead of waiting for it to happen just "pass" and wait for
handle_read() to be called again.
This way you don't do anything which blocks the asynchronous loop and
every peer (clients, servers or both) you have in your list of
watchable channels will be served.


If my response hasn't helped you any further try to post some code so
that me or someone else could see what exactly you're trying to do.
Hope this helps, anyway.


--- Giampaolo
http://code.google.com/p/pyftpdlib

 

[Frank Millman]

Thanks again, Giampaolo, your input is really appreciated.

I pretty much have the same overview I had before.
As far as I can tell the only reason you want to use a thread is when
you have to do something which requires a consistent amount of time to
complete so that the asyncore loop gets blocked.
The question is: is there anything like that in your code?
If the answer is no then you don't need to use threads.
Maybe you are just not used to the asynchronous approach where "wait
for server to respond" or "wait for the response to be complete"
doesn't mean that YOU have to wait.
It means that when a specific condition occurs (e.g. some data is
received) a certain method of the framework you're using will be
called and then it will be up to you deciding what do to.

Maybe I am not being clear enough on my side.

I (hope I) understand the power of asnyc. I have written the server to
use async techniques, and when I get around to writing the full-blown
client, that will also use async techniques.

However, at this stage, all I want to do is write something quick and
dirty to check that the server is behaving as intended. I want to
throw some messages at it, more or less at random, and check the
responses. I found it much easier to do this with asyncore.loop
running in a separate thread.

To send a message from the main thread, I append it to a list called
self.sendData. In asyncore.dispatcher, writable() returns True if the
list contains anything, and handle_write() pops the message off the
list and sends it.

To receive messages, readable() always return True, and handle_read()
breaks up the input into individual messages and appends them to a
list called self.recvData. When the main thread is waiting for a
response, it simply loops until self.recvData contains something.

To do this asynchronously, for every test I would have to define the
detailed interaction between client and server, and write methods to
be called from within handle_read(). It could be done, but it would be
much more tedious.

Does this make sense?

Frank

 

[Giampaolo Rodola']

Thanks again, Giampaolo, your input is really appreciated.




Maybe I am not being clear enough on my side.

I (hope I) understand the power of asnyc. I have written the server to
use async techniques, and when I get around to writing the full-blown
client, that will also use async techniques.

However, at this stage, all I want to do is write something quick and
dirty to check that the server is behaving as intended. I want to
throw some messages at it, more or less at random, and check the
responses. I found it much easier to do this with asyncore.loop
running in a separate thread.

I don't know why you find more convenient running asyncore.loop in a
separate thread but if your purpose is writing a test suite in which a
client checks responses sent by server I *would not* recommend using
asyncore.
The common way to do that is starting the server into setUp() method
and shut it down in tearDown().
Every test consists in a client which uses the socket module to
connect() to the server, send() something and recv() the response.
That's all.
pyftpdlib, which consists of an asyncore-based FTP server, has a test
suite behaving exactly like that.
Try to take a look and see if it could fit your purposes:
http://code.google.com/p/pyftpdlib/source/browse/tags/release-0.4.0/test/test_ftpd.py

To send a message from the main thread, I append it to a list called
self.sendData. In asyncore.dispatcher, writable() returns True if the
list contains anything, and handle_write() pops the message off the
list and sends it.

To receive messages, readable() always return True, and handle_read()
breaks up the input into individual messages and appends them to a
list called self.recvData. When the main thread is waiting for a
response, it simply loops until self.recvData contains something.

To do this asynchronously, for every test I would have to define the
detailed interaction between client and server, and write methods to
be called from within handle_read(). It could be done, but it would be
much more tedious.

Does this make sense?

Now I see why you find it more tedious, in fact I wouldn't use that
kind of approach in the test suite.
Anyway, I still don't understand why running asyncore.loop into a
thread could make any difference.


--- Giampaolo
http://code.google.com/p/pyftpdlib/

 

[Frank Millman]

I don't know why you find more convenient running asyncore.loop in a
separate thread but if your purpose is writing a test suite in which a
client checks responses sent by server I *would not* recommend using
asyncore.
The common way to do that is starting the server into setUp() method
and shut it down in tearDown().
Every test consists in a client which uses the socket module to
connect() to the server, send() something and recv() the response.
That's all.
pyftpdlib, which consists of an asyncore-based FTP server, has a test
suite behaving exactly like that.
Try to take a look and see if it could fit your purposes:http://code.google.com/p/pyftpdlib/source/browse/tags/release-0.4.0/t...

Ok, I see where you are coming from. I had a look at your test suite,
and I can see how its approach differs from mine. I will study it
carefully.

Thanks very much for all the help.

Frank

 

[Josiah Carlson]

Hi all

I have been using my own home-brewed client/server technique for a
while, using socket and select. It seems to work ok. The server can
handle multiple clients. It does this by creating a new thread for
each connection. Each thread runs its own select loop.

I am making some fairly big changes, so I thought I would look atasyncore.. I modified my program to useasyncorewithout much trouble,
and it feels nicer. It uses async I/O instead of threading, and it
relieves me of having to run my own select loop.

I have two questions. They both relate to whether I am using the
module as intended. The documentation is rather sparse, and I know
from experience that just getting something working is no guarantee
that I am getting the full benefit.

Firstly, having gotasyncoreworking, I had a look at asynchat. As far
as I can see I get very little benefit from using it. I have already
set up a 'messaging' protocol between server and client, where all
messages consist of 5 digits for the message length, followed by the
message. The message consists of a pickled tuple, where the first
element is a message identifier, and the rest is the message body.

This is how it works inasyncore-

    def __init__(self,channel):
        [...]
        self.msgLength = 0
        self.recvData = ''  # temporary buffer

    def handle_read(self):
        self.recvData += self.recv(8192)
        if not self.msgLength:
            if len(self.recvData) < 5:  # 1st 5 bytes = msg length
                return
            self.msgLength = int(self.recvData[:5])
            self.recvData = self.recvData[5:]
        if len(self.recvData) < self.msgLength:
            return
        data = loads(self.recvData[:self.msgLength])
        self.recvData = self.recvData[self.msgLength:]
        self.msgLength = 0
        [handle data]

This is how I got it working in asynchat -

    def __init__(self,channel):
        [...]
        self.recvData = ''  # temporary buffer
        self.set_terminator(5)
        self.gotMsgLength = False

    def collect_incoming_data(self, data):
        self.recvData += data

    def found_terminator(self):
        if self.gotMsgLength:  # what follows is the message
            data = loads(self.recvData)
            self.set_terminator(5)
            self.gotMsgLength = False
            [handle data]
        else:  # what follows is the message length
            self.set_terminator(int(self.recvData))
            self.gotMsgLength = True
        self.recvData = ''

It may be slightly neater, but it does not seem worth adding an extra
layer just for that. Does asynchat give me any other benefits I may
have overlooked?

Giampaolo already offered up some information, but I will offer these
two little tidbits:
In your first client, your handle_read doesn't handle the case where
you have received multiple packets at one time. That is, imagine that
in your one .read(8192), you received exactly two messages (the prefix
length and the pickle itself times two). The first pass will pick up
the message and handle the data. But unless the socket becomes
readable again, the second message will never be processed. And even
if the socket becomes readable immediately in the next select() call,
the message will be delayed depending on what other sockets are up
to. The asynchat module handles that case just fine.

As for push() vs. send(); send() returns the number of bytes sent. If
it sends less than the total block of data (which can be the case with
large blocks of data, small TCP/IP windows over a slow connection, or
small TCP/IP buffers), you need to be aware of it and attempt to
resend the remaining. Again, the asynchat module handles that case
just fine with it's .push() method; when it returns, you know that the
data to be transferred will be transferred as long as the connection
stays alive. Without .push(), you need to implement that behavior
yourself (less boilerplate for every subclass = easier maintenance).

(I normally don't hop on this list to comment, so please cc me on any
replies)
- Josiah

 

[Frank Millman]

Giampaolo already offered up some information, but I will offer these
two little tidbits:
In your first client, your handle_read doesn't handle the case where
you have received multiple packets at one time. That is, imagine that
in your one .read(8192), you received exactly two messages (the prefix
length and the pickle itself times two). The first pass will pick up
the message and handle the data. But unless the socket becomes
readable again, the second message will never be processed. And even
if the socket becomes readable immediately in the next select() call,
the message will be delayed depending on what other sockets are up
to. The asynchat module handles that case just fine.

As for push() vs. send(); send() returns the number of bytes sent. If
it sends less than the total block of data (which can be the case with
large blocks of data, small TCP/IP windows over a slow connection, or
small TCP/IP buffers), you need to be aware of it and attempt to
resend the remaining. Again, the asynchat module handles that case
just fine with it's .push() method; when it returns, you know that the
data to be transferred will be transferred as long as the connection
stays alive. Without .push(), you need to implement that behavior
yourself (less boilerplate for every subclass = easier maintenance).

(I normally don't hop on this list to comment, so please cc me on any
replies)
- Josiah

Valuable insights. Much appreciated.

Frank