PS: Help pickling across sockets

[Jonathan Hayward]

Just after posting the previous request, I realised that my test case
worked after I added flushes. The next problem I have is that, in the
real program, the server gets an EOFError when it first tries to read
from the socket. Relevant portions follow the error message below:

Traceback (most recent call last):
File "/var/www/cgi-bin/datamine", line 1633, in ?
find_and_output_appropriate_page()
File "/var/www/cgi-bin/datamine", line 1461, in
find_and_output_appropriate_page
sys.stdout.write(get_output())
File "/var/www/cgi-bin/datamine", line 1497, in get_output
return multitasking.get_page_from_oracle()
File "/var/www/cgi-bin/datamine", line 976, in get_page_from_oracle
self.check_and_start_oracle()
File "/var/www/cgi-bin/datamine", line 972, in check_and_start_oracle
self.start_oracle()
File "/var/www/cgi-bin/datamine", line 1083, in start_oracle
self.run_oracle()
File "/var/www/cgi-bin/datamine", line 1056, in run_oracle
self.handle_oracle_query(newsocket, address)
File "/var/www/cgi-bin/datamine", line 1015, in handle_oracle_query
self.get_thread_specific_storage()["environment_variables"] = \
EOFError
Traceback (most recent call last):
File "/var/www/cgi-bin/datamine", line 1633, in ?
find_and_output_appropriate_page()
File "/var/www/cgi-bin/datamine", line 1461, in
find_and_output_appropriate_page
sys.stdout.write(get_output())
File "/var/www/cgi-bin/datamine", line 1497, in get_output
return multitasking.get_page_from_oracle()
File "/var/www/cgi-bin/datamine", line 988, in get_page_from_oracle
result = cPickle.load(sockIn)
IOError: [Errno 104] Connection reset by peer

From the server side, here are run_oracle() and handle_oracle_query():

def run_oracle(self):
#thread.start_new_thread(\
#self.initialize, ())
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(("", configuration.get_search_server_port()))
sock.listen(5)
while 1:
try:
newsocket, address = sock.accept()
self.handle_oracle_query(newsocket, address)
#thread.start_new_thread(self.handle_oracle_query,
(newsocket, \ #address))
except socket.error:
sock.close()
sock = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,
1)
sock.bind(("", configuration.get_search_server_port()))
sock.listen(5)

def handle_oracle_query(self, sock, address):
sockIn = sock.makefile("rb")
sockOut = sock.makefile("wb")
sockIn.flush()
### Line 1015, immediately below, is where the EOF error is reported.
self.get_thread_specific_storage()["environment_variables"] = \
cPickle.load(sockIn)
sockIn.flush()
self.get_thread_specific_storage()["cgi"] =
cPickle.load(sockIn)
generate_output()
print_output(sockOut)
sockOut.flush()
sock.close()
sockIn.close()
sockOut.close()

From the client side, here is get_page_from_oracle():

def get_page_from_oracle(self):
self.check_and_start_oracle()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
sock.connect((configuration.get_search_server_ip(), \
configuration.get_search_server_port()))
sockIn = sock.makefile("rb")
sockOut = sock.makefile("wb")
cPickle.dump(os.environ, sockOut)
sockOut.flush()
cPickle.dump(cgi.FieldStorage(), sockOut)
sockOut.flush()
sockIn.flush()
### Line 988, immediately below, also gets an error, but I believe
this is secondary damage.
result = cPickle.load(sockIn)
sock.close()
sockIn.close()
sockOut.close()
return result

TIA,
++ Jonathan Hayward, (e-mail address removed)
** To see an award-winning website with stories, essays, artwork,
** games, and a four-dimensional maze, why not visit my home page?
** All of this is waiting for you at http://JonathansCorner.com

 

[Irmen de Jong]

Just after posting the previous request, I realised that my test case
worked after I added flushes. The next problem I have is that, in the
real program, the server gets an EOFError when it first tries to read
from the socket.

I'm not sure, but perhaps pickle doesn't work on sockets directly?
Perhaps it needs a random-access file object.

So, read all the data from the socket yourself, into a string
buffer, and run pickle on that.

That's what I do in Pyro

HTH,
--Irmen de Jong