pickle passing client/server design

[DwBear75]

I am contemplating the need for a way to handle high speed data
passing between two processes. One process would act as a queue that
would 'buffer' data coming from another processes. Seems that the
easiest way to handle the data would be to just pass pickles. Further,
I'm thinking that using a unix domain socket would make this a simple
way to pass high volumes of pickles. Are there any examples of an
architecture like these, where a process is a client, sending pickles
to a server listening on a domain socket?

I'm am thinking there would be a need to have a semaphore, and some
ACK or NACK that the server process got the whole pickle.

 

[John Krukoff]

I am contemplating the need for a way to handle high speed data
passing between two processes. One process would act as a queue that
would 'buffer' data coming from another processes. Seems that the
easiest way to handle the data would be to just pass pickles. Further,
I'm thinking that using a unix domain socket would make this a simple
way to pass high volumes of pickles. Are there any examples of an
architecture like these, where a process is a client, sending pickles
to a server listening on a domain socket?

I'm am thinking there would be a need to have a semaphore, and some
ACK or NACK that the server process got the whole pickle.

Quick bit of advice, don't reinvent the wheel, use PYRO:
http://pyro.sourceforge.net/index.html

 

[castironpi]

I am contemplating the need for a way to handle high speed data
passing between two processes. One process would act as a queue that
would 'buffer' data coming from another processes. Seems that the
easiest way to handle the data would be to just pass pickles. Further,
I'm thinking that using a unix domain socket would make this a simple
way to pass high volumes of pickles. Are there any examples of an
architecture like these, where a process is a client, sending pickles
to a server listening on a domain socket?

I'm am thinking there would be a need to have a semaphore, and some
ACK or NACK that the server process got the whole pickle.

Use mmap for your data, and a pipe to wake the other side on changes.

With moderate extra work, you can even use ctypes structures to store
raw data directly, without pickle.