a question regarding call-by-reference

[enjoying the view]

I am working on a school project, trying to build a simple RPC stub
generator. The idea is that the generator takes a normal python file
with functions defined in it and produces a client and server stub. The
client stub has the same functions defined, but they just connect to
the server and ask it to call the desired functions. The server stub is
a server listening for incoming requests and dispatching them to the
appropriate functions and sending back the results. Extremely simple
and I've gotten it to mostly work, but I have one problem:
call-by-reference parameters. An example:

A function like this:
def sum(a, b):
return a + b

would yield a client stub like this (simplified):
def sum(a, b):
send_message({'funcname': 'sum', 'args': (a, b)})
result = receive_message()
return result

Now, this works just fine. I get problems when I try to do something
like this:
def add_elm(list):
list.append('elm')
return

Imported normally this would work fine. The list given as a parameter
would be one element larger. But when the stubs are generated, the
function doesn't return anything and the list is appended in the server
and the client-side list is left untouched. At first I thought the
solution was easy, just return the changed parameters and place them in
the parameter variables in the client stub:
def add_elm(list):
send_message({'funcname': 'add_elm', 'args': (list)})
response = receive_message()
(list) = response.get('args')
return response.get('result')

The problem is, this doesn't work. The original list doesn't point to
the changed list. I have been trying to figure this out and it seems
that if I just assign to the list variable it just modifies the local
(to the function) name space, and that those changes aren't reflected
in the list in the original name space.

I believe there are some ways around this, but I haven't found one that
would not require any special handling in either the code calling the
client stub or the original functions. I want to maintain transparency.

etv

 

[Paul McGuire]

I am working on a school project, trying to build a simple RPC stub
generator. The idea is that the generator takes a normal python file
with functions defined in it and produces a client and server stub. The
client stub has the same functions defined, but they just connect to
the server and ask it to call the desired functions. The server stub is
a server listening for incoming requests and dispatching them to the
appropriate functions and sending back the results.

<snip>

One thing you might try is iterating over the arguments and doing special
handling for lists and dicts, which can be updated within the called module.
Using your example (changing 'list' to 'listarg' to avoid confusion):

def add_elm(listarg):
listarg.append('elm')
return

Generate code like:

def add_elm(listarg):
send_message({'funcname': 'add_elm', 'args': (listarg)})
response = receive_message()
listarg[:] = response.get('args')[0][:]
return response.get('result')

If the listargs get long/complicated, you might be doing a lot of data
copying at the cost of performance (especially since this is a defensive
copy - you don't *know* that the list was updated in the server in the first
place - I guess you could test first with "if response.get('args')[0] !=
listarg:" or something).

For dicts, do:

dictarg.update(response.get('args')[n])

This will be a little cheaper than just total replacement of the contents,
which is what the list [:] slice will do.

But this special-case handling only works for lists and dicts (don't know
how you would handle tuples or scalar variables), you quickly run into other
problems with passing objects. I'm taking you at your word that you are
creating a *simple* RPC stub generator. At this point, you can generate
stubs for methods that do not update their arguments, or for methods that
pass lists or dicts. I've used CORBA in the past, and this is similar to
using a subset of the full parameter functionality, such as using only "in"
parameters (as opposed to "out" or "inout"). That is, instead of defining a
method that modifies its parameter value, have the new value returned by the
function. Python actually helps you here, since it is so easy to pass back
a tuple of values from a function.

This kind of parameter marshalling/unmarshalling gets complicated quickly.
You are already very near the edge of non-intrusiveness on the client and
server sides - to get more features, you'll need to start adding some
aspects of parameter passing mechanisms (such as CORBA's in/out/inout
keywords), and object marshalling conventions (such as requiring objects
that are passed to implement compare() and update() or copy() methods).

-- Paul

 

[Claudio Grondi]

I am working on a school project, trying to build a simple RPC stub
generator. The idea is that the generator takes a normal python file
with functions defined in it and produces a client and server stub. The
client stub has the same functions defined, but they just connect to
the server and ask it to call the desired functions. The server stub is
a server listening for incoming requests and dispatching them to the
appropriate functions and sending back the results. Extremely simple
and I've gotten it to mostly work, but I have one problem:
call-by-reference parameters. An example:

A function like this:
def sum(a, b):
return a + b

would yield a client stub like this (simplified):
def sum(a, b):
send_message({'funcname': 'sum', 'args': (a, b)})
result = receive_message()
return result

Now, this works just fine. I get problems when I try to do something
like this:
def add_elm(list):
list.append('elm')
return

Imported normally this would work fine. The list given as a parameter
would be one element larger. But when the stubs are generated, the
function doesn't return anything and the list is appended in the server
and the client-side list is left untouched. At first I thought the
solution was easy, just return the changed parameters and place them in
the parameter variables in the client stub:
def add_elm(list):
send_message({'funcname': 'add_elm', 'args': (list)})
response = receive_message()
(list) = response.get('args')
return response.get('result')

The problem is, this doesn't work. The original list doesn't point to
the changed list. I have been trying to figure this out and it seems
that if I just assign to the list variable it just modifies the local
(to the function) name space, and that those changes aren't reflected
in the list in the original name space.

I believe there are some ways around this, but I haven't found one that
would not require any special handling in either the code calling the
client stub or the original functions. I want to maintain transparency.

etv

I am not an expert, so maybe someone else can provide here a nice tricky
solution, but I am quite sure, that there is no simple way around the
described problem.
The only strategy which comes to my mind in this context is to use for
all function arguments and return values only pickled objects which need
to be unpickled on the other side. Probably some compression algorithm
can help to keep the network traffic down in case large amount of data
must be exchanged.

Claudio

 

[Kent Johnson]

Imported normally this would work fine. The list given as a parameter
would be one element larger. But when the stubs are generated, the
function doesn't return anything and the list is appended in the server
and the client-side list is left untouched. At first I thought the
solution was easy, just return the changed parameters and place them in
the parameter variables in the client stub:
def add_elm(list):
send_message({'funcname': 'add_elm', 'args': (list)})
response = receive_message()
(list) = response.get('args')
return response.get('result')

The problem is, this doesn't work. The original list doesn't point to
the changed list. I have been trying to figure this out and it seems
that if I just assign to the list variable it just modifies the local
(to the function) name space, and that those changes aren't reflected
in the list in the original name space.

Try
list[:] = response.get('args')
this will change the value of the list passed in.

Kent

 

[Bengt Richter]

I am working on a school project, trying to build a simple RPC stub
generator. The idea is that the generator takes a normal python file
with functions defined in it and produces a client and server stub. The
client stub has the same functions defined, but they just connect to
the server and ask it to call the desired functions. The server stub is
a server listening for incoming requests and dispatching them to the
appropriate functions and sending back the results. Extremely simple
and I've gotten it to mostly work, but I have one problem:
call-by-reference parameters. An example:

A function like this:
def sum(a, b):
return a + b

would yield a client stub like this (simplified):
def sum(a, b):
send_message({'funcname': 'sum', 'args': (a, b)})
result = receive_message()
return result

Now, this works just fine. I get problems when I try to do something
like this:
def add_elm(list):
list.append('elm')
return

Imported normally this would work fine. The list given as a parameter
would be one element larger. But when the stubs are generated, the
function doesn't return anything and the list is appended in the server
and the client-side list is left untouched. At first I thought the
solution was easy, just return the changed parameters and place them in
the parameter variables in the client stub:
def add_elm(list):
send_message({'funcname': 'add_elm', 'args': (list)})
response = receive_message()

if the response contains an updated list, you can update the content
of the add_elm caller's list by slice assignment, e.g.,
list[:] = response.get(<however you get the updated list>')

But you are in really tricky territory here I think. There is any number
of ways to mutate objects via function argument reference. Not just
if they're lists. And lists can be deeply nested, so you could be sending
deep copies of lists across a network, which could involve most of your memory
even if the first level has only a few elements. And if there are side effects besides
the direct mutations, you have additional problems. What if a function accesses
arg.prop and that's a property that keeps count or otherwise tracks instances of something?

The totally general solution is probably impossible. So if I were you I would go back to
the specifier of this problem and negotiate some severe restrictions on assumptions
about arguments (maybe just outlaw direct mutation and access side effects ;-)

(list) = response.get('args')
return response.get('result')

The problem is, this doesn't work. The original list doesn't point to
the changed list. I have been trying to figure this out and it seems
that if I just assign to the list variable it just modifies the local
(to the function) name space, and that those changes aren't reflected
in the list in the original name space.

I believe there are some ways around this, but I haven't found one that
would not require any special handling in either the code calling the
client stub or the original functions. I want to maintain transparency.

etv

Good luck ;-)

Regards,
Bengt Richter

 

[enjoying the view]

Thank you everyone for your helpful replies!

I think the problems that arise with nested and overly large lists and
dictionaries, and difficulties of handling other mutable datatypes will
make my little assignment just too difficult. I'll just specify that
call-by-reference isn't supported and leave it at that.

Between floors. Going down.
Enjoying the view.

 

[Casey Hawthorne]

Don't send the whole list or parts of the list to the server unless
actually needed!

If the server has to do an append, have the server send back the new
elements to be appended and then do the appending on the client side!

Or

Does the homework specify that the entire mutable type be sent to the
server?