How safe is modifying locals()?

Discussion in 'Python' started by Paul Paterson, Jul 25, 2003.

  1. I am trying to find a way to mimic by-reference argument passing for
    immutables in Python. I need to do this because I am writing an
    automated VB to Python converter.

    Here's an example of the VB code:

    Sub Change(ByVal x, ByRef y)
    x = x+1
    y = y+1
    End Sub

    x = 0: y = 0
    Change x, y
    ' Now x should be 0 and y should be 1

    One approach that has been suggested is to use locals() and indirect
    references to the immutable via the resulting dictionary,

    def change(x, y, refs):
    x = x + 1
    refs[y] = refs[y] + 1

    x = 0; y = 0;
    change(x, 'y', locals())

    The Python documentation gives a warning that modifying the contents of
    locals() may not affect local values. Despite the warning, the code
    seems to work as desired, at least on Python 2.2.

    Does anyone know how safe this approach is, particularly in a
    multithreaded environment? Will this work on Jython?

    Does anyone have any alternative strategies for trying to achieve the
    objective? A "Pythonic" approach such as,

    def change(x, y):
    x = x + 1
    y = y + 1
    return y

    x = 0; y = 0
    y = change(x, y)

    Works in a single threaded environment but, because the value of 'y'
    changes at the wrong time, with multiple threads there is a possibility
    that two users of 'y' could dissagree on its value.

    Paul


    Paul Paterson
    ()

    vb2py :: A Visual Basic to Python Conversion Toolkit
    http://vb2py.sourceforge.net
    Paul Paterson, Jul 25, 2003
    #1
    1. Advertising

  2. Paul Paterson

    Terry Reedy Guest

    "Paul Paterson" <> wrote in message
    news:BDcUa.100193$...
    > I am trying to find a way to mimic by-reference argument passing for
    > immutables in Python. I need to do this because I am writing an
    > automated VB to Python converter.
    >
    > Here's an example of the VB code:
    >
    > Sub Change(ByVal x, ByRef y)
    > x = x+1
    > y = y+1
    > End Sub
    >
    > x = 0: y = 0
    > Change x, y
    > ' Now x should be 0 and y should be 1
    >
    > One approach that has been suggested is to use locals() and indirect
    > references to the immutable via the resulting dictionary,
    >
    > def change(x, y, refs):
    > x = x + 1
    > refs[y] = refs[y] + 1
    >
    > x = 0; y = 0;
    > change(x, 'y', locals())
    >
    > The Python documentation gives a warning that modifying the contents

    of
    > locals() may not affect local values. Despite the warning, the code
    > seems to work as desired, at least on Python 2.2.


    At module scope, locals() == globals(). Within a function, locals()
    currently is a *copy* of the function's local namespace. So above
    only works because you made the call from the global (module) context
    and would not if you tested by making call from within a function.

    Terry J. Reedy
    Terry Reedy, Jul 25, 2003
    #2
    1. Advertising

  3. Terry Reedy wrote:
    > "Paul Paterson" <> wrote in message
    >>
    >>The Python documentation gives a warning that modifying the contents

    >
    > of
    >
    >>locals() may not affect local values. Despite the warning, the code
    >>seems to work as desired, at least on Python 2.2.

    >
    >
    > At module scope, locals() == globals(). Within a function, locals()
    > currently is a *copy* of the function's local namespace. So above
    > only works because you made the call from the global (module) context
    > and would not if you tested by making call from within a function.
    >


    Thanks Terry! I adjusted my tests and now see this behaviour exactly.
    The function itself works but the changes do not propogate out to the
    calling scope. So this approach of using locals() appears to be dead.

    Are there any other approaches?

    Paul
    Paul Paterson, Jul 25, 2003
    #3
  4. Quoth Paul Paterson:
    [...]
    > Does anyone have any alternative strategies for trying to achieve the
    > objective? A "Pythonic" approach such as,
    >
    > def change(x, y):
    > x = x + 1
    > y = y + 1
    > return y
    >
    > x = 0; y = 0
    > y = change(x, y)
    >
    > Works in a single threaded environment but, because the value of 'y'
    > changes at the wrong time, with multiple threads there is a possibility
    > that two users of 'y' could dissagree on its value.


    Even if the simple
    y = y + 1
    could change the caller's binding, there would be a race condition
    -- you'll need a lock anyway. (I'd be a bit surprised if this
    were not so in VB as well. Are statements atomic in VB?)

    --
    Steven Taschuk w_w
    ,-= U
    1 1
    Steven Taschuk, Jul 25, 2003
    #4
  5. Paul Paterson wrote:
    > I am trying to find a way to mimic by-reference argument passing for
    > immutables in Python. I need to do this because I am writing an
    > automated VB to Python converter.
    >
    > Here's an example of the VB code:
    >
    > Sub Change(ByVal x, ByRef y)
    > x = x+1
    > y = y+1
    > End Sub
    >
    > x = 0: y = 0
    > Change x, y
    > ' Now x should be 0 and y should be 1


    I might put all of the "ByRef" variables in a dictionary that gets
    passed back to the caller through a hidden argument.

    def change(x, y, _byref):
    x = x + 1
    y = _byref['y'] = y + 1

    x = 0; y = 0
    __byref = {'y': y}; change(x, y, __byref); y = __byref['y']

    At least this relies on nothing magical. One thing that's still wrong,
    though, is that you can't embed the change() call in an expression.
    AFAIK, in Python, assignment to a local variable absolutely requires a
    statement. You can get around this by splitting apart the expression
    and storing the result of the call in another hidden temporary variable.

    Even if you do that, though, exceptions will behave differently in
    Python than they do in VB. If an exception occurs in change() after a
    value has been assigned to 'y', the outer code won't get the value, and
    it just might affect the behavior of the outer code. You could fix that
    with a try: / finally:

    __byref = {'y': y}
    try:
    change(x, y, __byref)
    finally:
    y = __byref['y']

    Suddenly it's awfully verbose and ugly, but if variables by reference
    are rare (as they should be), maybe it's not so bad. :)

    You could go the extreme route and store all variables in dictionaries
    rather than use local variables, which would allow "true" references,
    but then you'd kill speed and readability. Better not do that.

    Shane
    Shane Hathaway, Jul 25, 2003
    #5
  6. Paul Paterson

    Ian Bicking Guest

    On Fri, 2003-07-25 at 12:28, Paul Paterson wrote:
    > Thanks Terry! I adjusted my tests and now see this behaviour exactly.
    > The function itself works but the changes do not propogate out to the
    > calling scope. So this approach of using locals() appears to be dead.
    >
    > Are there any other approaches?


    Think about what you are trying to do, and try to identify a (mutable)
    object that can encapsulate that. Then pass the object, and modify its
    instance variables, like:

    class Point:
    def __init__(self, x, y):
    self.x = x
    self.y = y

    def change(p):
    p.y = 10

    obj = Point(0, 0)
    change(obj)
    assert obj.y == 10


    Maybe this mutable object will simply be the main application object,
    and instead of functions you will use methods of that application
    object.

    Ian
    Ian Bicking, Jul 25, 2003
    #6
  7. Ian Bicking wrote:

    > On Fri, 2003-07-25 at 12:28, Paul Paterson wrote:
    >
    >>Thanks Terry! I adjusted my tests and now see this behaviour exactly.
    >>The function itself works but the changes do not propogate out to the
    >>calling scope. So this approach of using locals() appears to be dead.
    >>
    >>Are there any other approaches?

    >
    >
    > Think about what you are trying to do, and try to identify a (mutable)
    > object that can encapsulate that. Then pass the object, and modify its
    > instance variables, like:
    >
    > class Point:
    > def __init__(self, x, y):
    > self.x = x
    > self.y = y
    >
    > def change(p):
    > p.y = 10
    >
    > obj = Point(0, 0)
    > change(obj)
    > assert obj.y == 10
    >
    >
    > Maybe this mutable object will simply be the main application object,
    > and instead of functions you will use methods of that application
    > object.
    >
    > Ian


    This is a very interesting (and Pythonic) approach, thanks for
    suggesting it! This is certainly what I would try to do if I were
    writing the code from scratch. It may be possible to construct a
    "namespace" type object which gets passed to the function.

    VB does have multiple namespaces so a single App object is probably not
    feasible but since my parser knows the App structure it can determine
    which namespace a variable will be resolved from anyway and just
    translate all attempts to access it to the relevant namespace object lookup.

    Paul
    Paul Paterson, Jul 26, 2003
    #7
  8. Shane Hathaway wrote:

    > Paul Paterson wrote:
    >
    >> I am trying to find a way to mimic by-reference argument passing for
    >> immutables in Python. I need to do this because I am writing an
    >> automated VB to Python converter.
    >>
    >> Here's an example of the VB code:
    >>
    >> Sub Change(ByVal x, ByRef y)
    >> x = x+1
    >> y = y+1
    >> End Sub
    >>
    >> x = 0: y = 0
    >> Change x, y
    >> ' Now x should be 0 and y should be 1

    >
    >
    > I might put all of the "ByRef" variables in a dictionary that gets
    > passed back to the caller through a hidden argument.
    >
    > def change(x, y, _byref):
    > x = x + 1
    > y = _byref['y'] = y + 1
    >
    > x = 0; y = 0
    > __byref = {'y': y}; change(x, y, __byref); y = __byref['y']
    >


    The problem here is that you still have the delayed change to 'y' in the
    caller's scope. Other posts in this thread have made me wonder whether
    this is a problem sepcific to the ByRef topic.

    The interesting thing about your approach is that you have an explicit
    step for transmitting changes to y back to the calling scope - which is
    quite nice. In an automated translation, having this explicit is a good
    sign-post for the user to let them know that something important is
    going on.

    > At least this relies on nothing magical. One thing that's still wrong,
    > though, is that you can't embed the change() call in an expression.
    > AFAIK, in Python, assignment to a local variable absolutely requires a
    > statement. You can get around this by splitting apart the expression
    > and storing the result of the call in another hidden temporary variable.
    >
    > Even if you do that, though, exceptions will behave differently in
    > Python than they do in VB.


    Matching VB's exception behaviour is my worst nightmare - but that's
    another story ...

    > If an exception occurs in change() after a
    > value has been assigned to 'y', the outer code won't get the value, and
    > it just might affect the behavior of the outer code. You could fix that
    > with a try: / finally:
    >
    > __byref = {'y': y}
    > try:
    > change(x, y, __byref)
    > finally:
    > y = __byref['y']
    >
    > Suddenly it's awfully verbose and ugly, but if variables by reference
    > are rare (as they should be), maybe it's not so bad. :)


    Unfortunately (and to my shock), it seems that ByRef is the *default*!
    It may be the case that using the implications of ByRef are rare and it
    may be possible for me to detect when this is occuring.

    > You could go the extreme route and store all variables in dictionaries
    > rather than use local variables, which would allow "true" references,
    > but then you'd kill speed and readability. Better not do that.


    At the end of the day (see my response to Ian) this may in fact be the
    safest approach if you want to match the exact behaviour of the original
    VB code. There will almost certainly be a switch where you can turn this
    off, but then it is up to the programmer to go and fix things which get
    broken. Hopefully it will be possible to try to identify likely hot
    spots automatically.

    Paul
    Paul Paterson, Jul 26, 2003
    #8
  9. Hi Paul

    Paul Paterson wrote:
    > This is a very interesting (and Pythonic) approach, thanks for
    > suggesting it! This is certainly what I would try to do if I were
    > writing the code from scratch. It may be possible to construct a
    > "namespace" type object which gets passed to the function.


    I think the most common way to make a namespace in Python is

    class my_namespace: pass
    ....
    my_namespace.a = 'foo'
    my_namespace.b = 'bar'

    Such namespaces are very similar to dictionaries:

    my_namespace = {}
    ....
    my_namespace['a'] = 'foo'
    my_namespace['b'] = 'bar'

    but the first approach is a bit more readable.

    If you need many containers, you could use:

    class Container:
    pass

    c1 = Container()
    c2 = Container()
    ....
    c1.foo = 'foo'
    c2.foo = 'bar'

    Of course, if you have a better name for your container, that's even
    better. Think of Ian's Point example. :)

    Stefan
    Stefan Schwarzer, Jul 26, 2003
    #9
  10. Stefan Schwarzer wrote:

    > Hi Paul
    >
    > Paul Paterson wrote:
    >
    >> This is a very interesting (and Pythonic) approach, thanks for
    >> suggesting it! This is certainly what I would try to do if I were
    >> writing the code from scratch. It may be possible to construct a
    >> "namespace" type object which gets passed to the function.

    >
    >
    > I think the most common way to make a namespace in Python is


    <snip nice example of namespace class>

    Thanks, the bare namespace class is essentially what I am going to try.

    >
    > Of course, if you have a better name for your container, that's even
    > better. Think of Ian's Point example. :)


    I think you may have missed my original post; I am writing a generic VB
    to Python converter so generating good names based on code intent would
    be pretty tough! Perhaps in v2.0 ;)

    My current thinking is (for the orignal example of changing one
    variable, 'y' but not another, 'x'),

    class Namespace: pas

    # ByVal arguments passed directly, ByRef via a namespace
    def change(x, byrefs):
    x = x + 1
    byrefs.y = byrefs.y + 1

    ns = Namespace() # The local namespace
    ns.x = 0
    ns.y = 0
    change(ns.x, ns)
    # Now ns.x = 0, ns.y = 1


    Paul
    Paul Paterson, Jul 26, 2003
    #10
  11. On Sat, 26 Jul 2003 14:49:41 +0200, Stefan Schwarzer <> wrote:

    >Hi Paul
    >
    >Paul Paterson wrote:
    >> This is a very interesting (and Pythonic) approach, thanks for
    >> suggesting it! This is certainly what I would try to do if I were
    >> writing the code from scratch. It may be possible to construct a
    >> "namespace" type object which gets passed to the function.

    >
    >I think the most common way to make a namespace in Python is
    >
    >class my_namespace: pass
    >...
    >my_namespace.a = 'foo'
    >my_namespace.b = 'bar'
    >
    >Such namespaces are very similar to dictionaries:
    >
    >my_namespace = {}
    >...
    >my_namespace['a'] = 'foo'
    >my_namespace['b'] = 'bar'
    >
    >but the first approach is a bit more readable.
    >
    >If you need many containers, you could use:
    >
    >class Container:
    > pass
    >
    >c1 = Container()
    >c2 = Container()
    >...
    >c1.foo = 'foo'
    >c2.foo = 'bar'
    >
    >Of course, if you have a better name for your container, that's even
    >better. Think of Ian's Point example. :)
    >

    Nice summary. Just to round it out, we could add using the type builtin.
    I.e., type(cname, bases, cdict) works to create an instant class (new type),
    and this can be used for some interesting alternative spellings, e.g.,

    A one-off name space object:

    >>> nso = type('',(),{})()
    >>> nso

    <__main__. object at 0x007AC1E0>

    BTW, funny that you can have a '' class name this way ;-) (I wonder if it can trigger a bug?)
    >>> `nso.__class__.__name__`

    "''"

    >>> nso.x = 123
    >>> vars(nso)

    {'x': 123}

    Anyway, if you want to initialize some default name bindings in the form of class variables,
    you can do that simply, even in the one-off one-liner, e.g.,

    >>> nso = type('',(),{'x':123, 'y':'wye'})()
    >>> nso.x, nso.y

    (123, 'wye')

    Note that they are class attributes ("variables"), not instance attributes, though:

    >>> nso.x = 'instance attribute' # shadows class attr
    >>> nso.x

    'instance attribute'
    >>> del nso.x # but can be unshadowed
    >>> nso.x

    123

    If you want to generate a cdict for class attributes by calling some function, you could, e.g.,

    >>> nso = type('',(),(lambda **kw:kw)(

    ... x = 456,
    ... y = 'wye',
    ... etc = 'and so forth'
    ... ))()

    Of course, that's a pretty silly substitute (other than the '' vs 'nso' class name) for

    >>> class nso(object):

    ... x = 456
    ... y = 'wye'
    ... etc = 'and so forth'
    ...
    >>> nso = nso()


    but it does show a function (lambda **kw:kw) being called to generate the cdict.
    Checking results:

    >>> nso.x

    456
    >>> nso.x, nso.y, nso.etc

    (456, 'wye', 'and so forth')
    >>> vars(nso.__class__).keys()

    ['__dict__', '__module__', 'etc', 'y', 'x', '__weakref__', '__doc__']

    BTW, I find it useful to use vars(foo).keys() rather than dir(foo) to avoid inherited stuff.

    Of course, if you bind the class to a name instead of instantly ()'ing it, note

    >>> ClassAlias = type('ClassName',(),{})
    >>> ClassAlias

    <class '__main__.ClassName'>

    doesn't match very well, and

    >>> ClassName = type('ClassName',(),{})
    >>> ClassName

    <class '__main__.ClassName'>

    is a silly substitute for

    >>> class ClassName(object): pass

    ...
    >>> ClassName

    <class '__main__.ClassName'>

    Even if you don't bind the name locally, it is bound to the instance
    as inst.__class__.__name__, so it may be a good idea to call type with
    a reasonable name as the first param, not ''.

    Hm, got to rambling again ;-/

    Regards,
    Bengt Richter
    Bengt Richter, Jul 26, 2003
    #11
  12. On Sat, 26 Jul 2003 15:20:24 GMT, Paul Paterson <> wrote:

    >Stefan Schwarzer wrote:
    >
    >> Hi Paul
    >>
    >> Paul Paterson wrote:
    >>
    >>> This is a very interesting (and Pythonic) approach, thanks for
    >>> suggesting it! This is certainly what I would try to do if I were
    >>> writing the code from scratch. It may be possible to construct a
    >>> "namespace" type object which gets passed to the function.

    >>
    >>
    >> I think the most common way to make a namespace in Python is

    >
    ><snip nice example of namespace class>
    >
    >Thanks, the bare namespace class is essentially what I am going to try.
    >
    >>
    >> Of course, if you have a better name for your container, that's even
    >> better. Think of Ian's Point example. :)

    >
    >I think you may have missed my original post; I am writing a generic VB
    >to Python converter so generating good names based on code intent would
    >be pretty tough! Perhaps in v2.0 ;)
    >
    >My current thinking is (for the orignal example of changing one
    >variable, 'y' but not another, 'x'),
    >
    >class Namespace: pas
    >
    ># ByVal arguments passed directly, ByRef via a namespace
    >def change(x, byrefs):
    > x = x + 1
    > byrefs.y = byrefs.y + 1
    >
    >ns = Namespace() # The local namespace
    >ns.x = 0
    >ns.y = 0
    >change(ns.x, ns)
    ># Now ns.x = 0, ns.y = 1
    >

    This looks readable, to me ;-)

    But it is not pointer semantics, since you have to know the name for y inside change.
    It will be nice and fast if you can use the above, but if not, in some cases, class NSP
    below will let you use it just like Namespace above, though with pointer features if
    you need them.

    I just wanted to throw in here that the namespace approach will also let you
    use properties (there's a handy builtin to create them from accessor methods,
    or you can instantiate them from your own descriptor classes) as your
    byrefs "variables" if you like.

    You just have to mane the NameSpace class inherit from object, to make it
    a new-style class. Then you can add properties either right in the class definition
    or later by setting class attributes , e.g.,

    >>> class Namespace(object): pass

    ...
    >>> ns = Namespace()
    >>> ns.x = 123
    >>> Namespace.p = property(lambda self:'Read is only access function specified here')


    Note that the above was *not* setting an instance attribute with ns.p = property(...)

    >>> ns.x

    123
    >>> ns.p

    'Read is only access function specified here'
    >>> ns.x = 456
    >>> ns.p = 789

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    AttributeError: can't set attribute
    >>> del ns.p

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    AttributeError: can't delete attribute

    So you can define a kind of constant, or have any side effect you want for setting, getting,
    and deleting.

    Note that normal class attributes are shadowed by instance attributes, e.g.,

    >>> ns.x # from last assignment

    456
    >>> Namespace.x = '456000'
    >>> ns.x

    456
    >>> del ns.x # deletes ordinary instance attribute
    >>> ns.x # and if instance attribute is absent, it will be looked for in class

    '456000'

    Actually it is a little more complicated than that, since it has to be determined that
    there is not a data descriptor/property in the class overriding the instance variable access.
    That's why I said "normal" ;-)

    A namespace for variables also gives you the potential to define methods for the name space
    itself. Here is an example with a name space that has an internal pointer class and lets
    you create "pointers" to the "variables" in the namespace, using ns[vname] syntax, which
    goes to the __getitem__ method of the class to make a "pointer" (__slots__ hopefully just
    makes pointer instances better optimized):

    >>> class NSP(object):

    ... """NSP defines name space with "pointer" creation via p = ns[vname]"""
    ... class Ptr(object):
    ... """Ptr instance holds ns ref and vname for access to ns.vname"""
    ... __slots__ = 'ns', 'vname'
    ... def __init__(self, ns, vname): self.ns=ns; self.vname=vname
    ... def __getitem__(self, i):
    ... """typical: x=p[:]"""
    ... return getattr(self.ns, self.vname)
    ... def __setitem__(self, i, v):
    ... """Typical: p[:] = x"""
    ... setattr(self.ns, self.vname, v)
    ... def __delitem__(self, i):
    ... """Typical: del p[:] # deletes what's pointed to"""
    ... delattr(self.ns, self.vname)
    ... def __getitem__(self, name):
    ... """Make "pointer." Typical: p2x = ns['x']; p2x[:] => ns.x"""
    ... return self.Ptr(self, name)
    ...
    >>> ns = NSP()
    >>> ns.x = 123
    >>> ns.y = 456
    >>> def change(x, py):

    ... x = 'new x?'
    ... py[:] = 'new via py[:]'
    ...
    >>> ns.x

    123
    >>> ns.y

    456
    >>> change(ns.x, ns['y']) # on the fly &y
    >>> ns.x

    123
    >>> ns.y

    'new via py[:]'
    >>>


    You can also save a "pointer" and use it later, or pass it around:

    >>> px = ns['x']
    >>> px[:]

    123

    ( [:] is just sugar, since the arg is ignored. [0] will run faster, since no slice object is needed)

    >>> px[0]

    123
    >>> px[0] = 'new x value'
    >>> ns.x

    'new x value'
    >>>


    Establish some new values:

    >>> ns.x = 123; ns.y = 456
    >>> ns.x, ns.y

    (123, 456)

    An important distinction from using explicit byref attribute names (e.g. byref.y) in change():

    >>> change('byvalue', px) # note we are passing px in the py parameter position
    >>> ns.x, ns.y

    ('new via py[:]', 456)

    I.e., change was passed a "pointer" it thought of as pointing to y (i.e., was named "py"
    by the programmer to suggest the meaning), but it pointed to x, and
    so x changed when py[:] was assigned to. Changes to 'byvalue' were just local to change, as expected.

    We can also delete an attribute via the pointer:
    >>> del px[0]
    >>> ns.x

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    AttributeError: 'NSP' object has no attribute 'x'

    which doesn't affect the "pointer" itself:
    >>> px[0]='an x again'
    >>> ns.x

    'an x again'

    but of course deleting the pointer itself as opposed to px[0] will:
    >>> del px
    >>> ns.x

    'an x again'
    >>> px

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    NameError: name 'px' is not defined

    i.e., the pointer got deleted, not the thing pointed to.

    I'm not necessarily recommending any of the above, but I thought it might give you some ideas.
    There is definitely a performance hit in px[:] (or better, px[0]) vs byref.x to consider.

    Anyway, I thought properties and descriptors might also come in handy somewhere in your quest ;-)

    Raymond Hettinger has written a nice "How-To Guide for Descriptors":

    http://users.rcn.com/python/download/Descriptor.htm

    and for properties (follow Table Of Contents link in below page) and much more there is

    http://www.python.org/2.2/descrintro.html

    HTH

    Regards,
    Bengt Richter
    Bengt Richter, Jul 26, 2003
    #12
  13. Bengt Richter wrote:

    > On Sat, 26 Jul 2003 15:20:24 GMT, Paul Paterson <> wrote:
    >
    >
    >>Stefan Schwarzer wrote:
    >>
    >>
    >>>Hi Paul
    >>>
    >>>Paul Paterson wrote:
    >>>
    >>>
    >>>>This is a very interesting (and Pythonic) approach, thanks for
    >>>>suggesting it! This is certainly what I would try to do if I were
    >>>>writing the code from scratch. It may be possible to construct a
    >>>>"namespace" type object which gets passed to the function.
    >>>
    >>>
    >>>I think the most common way to make a namespace in Python is

    >>
    >><snip nice example of namespace class>
    >>
    >>Thanks, the bare namespace class is essentially what I am going to try.
    >>
    >>
    >>>Of course, if you have a better name for your container, that's even
    >>>better. Think of Ian's Point example. :)

    >>
    >>I think you may have missed my original post; I am writing a generic VB
    >>to Python converter so generating good names based on code intent would
    >>be pretty tough! Perhaps in v2.0 ;)
    >>
    >>My current thinking is (for the orignal example of changing one
    >>variable, 'y' but not another, 'x'),
    >>
    >>class Namespace: pas
    >>
    >># ByVal arguments passed directly, ByRef via a namespace
    >>def change(x, byrefs):
    >> x = x + 1
    >> byrefs.y = byrefs.y + 1
    >>
    >>ns = Namespace() # The local namespace
    >>ns.x = 0
    >>ns.y = 0
    >>change(ns.x, ns)
    >># Now ns.x = 0, ns.y = 1
    >>

    >
    > This looks readable, to me ;-)
    >
    > But it is not pointer semantics, since you have to know the name for y inside change.


    For my application this is always the case ... but sometimes treating
    the problem as harder than it really is leads to a solution that is
    better than you would get otherwise!

    > It will be nice and fast if you can use the above, but if not, in some cases, class NSP
    > below will let you use it just like Namespace above, though with pointer features if
    > you need them.
    >
    > I just wanted to throw in here that the namespace approach will also let you
    > use properties (there's a handy builtin to create them from accessor methods,
    > or you can instantiate them from your own descriptor classes) as your
    > byrefs "variables" if you like.
    >
    > You just have to mane the NameSpace class inherit from object, to make it
    > a new-style class. Then you can add properties either right in the class definition
    > or later by setting class attributes , e.g.,


    <snip interesting development of the property approach>

    > A namespace for variables also gives you the potential to define methods for the name space
    > itself. Here is an example with a name space that has an internal pointer class and lets
    > you create "pointers" to the "variables" in the namespace, using ns[vname] syntax, which
    > goes to the __getitem__ method of the class to make a "pointer" (__slots__ hopefully just
    > makes pointer instances better optimized):
    >
    > >>> class NSP(object):

    > ... """NSP defines name space with "pointer" creation via p = ns[vname]"""
    > ... class Ptr(object):
    > ... """Ptr instance holds ns ref and vname for access to ns.vname"""
    > ... __slots__ = 'ns', 'vname'
    > ... def __init__(self, ns, vname): self.ns=ns; self.vname=vname
    > ... def __getitem__(self, i):
    > ... """typical: x=p[:]"""
    > ... return getattr(self.ns, self.vname)
    > ... def __setitem__(self, i, v):
    > ... """Typical: p[:] = x"""
    > ... setattr(self.ns, self.vname, v)
    > ... def __delitem__(self, i):
    > ... """Typical: del p[:] # deletes what's pointed to"""
    > ... delattr(self.ns, self.vname)
    > ... def __getitem__(self, name):
    > ... """Make "pointer." Typical: p2x = ns['x']; p2x[:] => ns.x"""
    > ... return self.Ptr(self, name)
    > ...
    > >>> ns = NSP()
    > >>> ns.x = 123
    > >>> ns.y = 456
    > >>> def change(x, py):

    > ... x = 'new x?'
    > ... py[:] = 'new via py[:]'
    > ...
    > >>> ns.x

    > 123
    > >>> ns.y

    > 456
    > >>> change(ns.x, ns['y']) # on the fly &y
    > >>> ns.x

    > 123
    > >>> ns.y

    > 'new via py[:]'
    > >>>


    To my eye, the [:] or [0] spelling of this makes the code look more
    complex than necessary, but I think you are on to something because if
    you spell it,

    def change(x, y):
    x = 'new x'
    y.update('new y')

    with the relevant changes to the Ptr class then it could certainly grow
    on me. The things I like are,

    - no new variable names in the 'change' function so it looks similar to
    the original code
    - the mechanism for propogating changes to the caller scope is explicit
    - 'y' can be passed on to another function if needed and things are
    still clear

    eg,

    def change(x, y):
    x = 'new x'
    change2(y)

    def change2(y):
    y.update('deep change in y')

    To do this using the original namespace approach gets a little tricky
    because you have to merge the namespaces as you go. The pointer idea
    flattens that structure.

    <snip stuff about deleting a pointer>


    > I'm not necessarily recommending any of the above, but I thought it might give you some ideas..
    > There is definitely a performance hit in px[:] (or better, px[0]) vs byref.x to consider.
    >
    > Anyway, I thought properties and descriptors might also come in handy somewhere in your quest ;-)


    This is certainly food for thought. My main criteria is this: I don't
    want to translate clearly written code in an ugly language (VB) into
    ugly code in a clear language (Python). If this happens I might as well
    pack up and go home!

    So I will trade speed for clarity almost always because someone can
    always optimize clear code later.

    Thanks for these thoughts and the time it took to post them, they really
    made me think! (I mean that in a good way, of course ;) )

    Paul
    Paul Paterson, Jul 27, 2003
    #13
  14. On Sun, 27 Jul 2003 04:51:11 GMT, Paul Paterson <> wrote:

    [...]

    >To my eye, the [:] or [0] spelling of this makes the code look more
    >complex than necessary, but I think you are on to something because if
    >you spell it,
    >
    >def change(x, y):
    > x = 'new x'
    > y.update('new y')
    >


    For a general pointer, ISTM you want to be able to dereference it for both getting and setting.
    The [:] or [0] syntax gets you that, and you could do it with p() for getting and p(val) for setting,
    or p.getval() and p.update(), but all these get clumsy when you want to use the "pointer" on
    both sides of an assignment statement, e.g.,

    p.update(p.getval()+' added text for target')

    where you could write more readably (IMO),

    p.value = p.value + 'added text for target'

    or
    p.v += 'added text for target'


    Below is a class PNS that lets you spell as immediately above, using .value
    (and .v for concise use) bound to a property that implements the accessing of
    what's pointed/referred to. I gave it a somewhat informative __repr__ method also,
    so the test prints better. As you will note, this is now separate from any particular
    name space. Any object that supports getattr and/or setattr can be used. I also
    threw in a permissions parameter to control read/write/delete. See nsother and math
    in examples. Note also that a pointer may point to another pointer, allowing cascaded
    dereferencing spelled p.v.v etc.

    >with the relevant changes to the Ptr class then it could certainly grow
    >on me. The things I like are,
    >
    >- no new variable names in the 'change' function so it looks similar to
    >the original code
    >- the mechanism for propogating changes to the caller scope is explicit
    >- 'y' can be passed on to another function if needed and things are
    >still clear
    >
    >eg,
    >
    >def change(x, y):
    > x = 'new x'
    > change2(y)
    >
    >def change2(y):
    > y.update('deep change in y')
    >

    If you use PNS, that will be spelled

    def change(x, y):
    x = 'new x'
    change2(y)

    def change2(y):
    y.value = 'deep change in y'

    See test() code below. I used your original class Namespace: pass as the main namespace.

    >
    >To do this using the original namespace approach gets a little tricky
    >because you have to merge the namespaces as you go. The pointer idea
    >flattens that structure.
    >


    ====< pns.py >==========================================================
    class PNS(object):
    """
    Pointer to Name Space
    PNS instance holds ns ref and vname for access to ns.vname
    Read, write, delete access permitted if r,w,d in perms, respectively.
    Typical: ptr_to_x = PNS(ns, 'x')
    """
    __slots__ = 'ns vname ok_r ok_w ok_d'.split()
    class PNSError(Exception):
    def __init__(self, ptr, msg):
    Exception.__init__(self, '%s for %r' %(msg, ptr))

    def __init__(self, ns, vname, perms='rwd'):
    self.ns=ns; self.vname=vname
    self.ok_r = 'r' in perms
    self.ok_w = 'w' in perms
    self.ok_d = 'd' in perms
    def _getv(self):
    """Typical read access: x = ptr.value (x = ptr.v works too)"""
    if self.ok_r: return getattr(self.ns, self.vname)
    raise self.PNSError(self, 'Value read prohibited')
    def _setv(self, v):
    """Typical write access: ptr.value = 'new x' (ptr.v = 'new x' works too)"""
    if self.ok_w: setattr(self.ns, self.vname, v)
    else: raise self.PNSError(self, 'Value write prohibited')
    def _delv(self):
    """Typical del access: del ptr.value (del ptr.v works too)"""
    if self.ok_d: delattr(self.ns, self.vname)
    else: raise self.PNSError(self, 'Value deletion prohibited')
    value = v = property(_getv, _setv, _delv) # .v for short
    def __repr__(self): return '<PNS ptr to %r of %r>'%(self.vname, self.ns)

    class Namespace(object): pass

    def test():
    ns = Namespace()
    ns.x = 'x value'
    ns.y = 'y value'
    print 'Before change:'
    print 'ns.x=%r, ns.y=%r' %(ns.x, ns.y)
    print 'Making pointer py point to ns.y ...'
    py = PNS(ns, 'y') # prefixing pointer names with 'p' is not mandatory, just mnemonic
    print 'ptr py=%r' % py

    def change(x, y): # prefixing pointer names with 'p' is not mandatory, just mnemonic
    x = 'new x'
    y.value = 'new y'

    print 'Calling change(ns.x, py) ...'
    change(ns.x, py)
    print 'After change:'
    print 'ns.x=%r, ns.y=%r' %(ns.x, ns.y)

    def change1(x, y):
    x = 'new x'
    change2(y)

    def change2(y):
    y.v = 'deep change in y'

    print 'Before change1/change2:'
    print 'ns.x=%r, ns.y=%r' %(ns.x, ns.y)
    change1(ns.x, py)
    print 'After calling change1(ns.x, py):'
    print 'ns.x=%r, ns.y=%r' %(ns.x, ns.y)

    pz = PNS(ns, 'z')
    print '\nNew pointer to non-existent ns.z:\n%r' % pz
    print 'Trying to access as yet nonexistent ns.z ...'
    try: ns.z
    except Exception, e: print '%s: %s'% (e.__class__.__name__,e)
    else: print 'ns.z accessed ok'

    print 'Passing pz to change(ns.x, pz)...'
    change(ns.x, pz)
    print 'Result: ns.x=%r, ns.z=%r' %(ns.x, ns.z)
    print '\nBefore deleting ns.y via py.v:'
    print 'ns.y=%r, py.v=%r' %(ns.y, py.v)

    print '\nDeleting ns.y by del py.value ...'
    del py.value
    print 'Trying to access ns.y ...'
    try: ns.y
    except Exception, e: print '%s: %s'% (e.__class__.__name__,e)
    else: print 'ns.y accessed ok'

    print '\nCreating nsother name space to put pz in as a value ...'
    nsother = type('AnotherNS',(),{})()
    print nsother
    nsother.pz = pz
    print 'Creating pointer ppz pointing to nsother.pz'
    ppz = PNS(nsother,'pz')
    print 'ppz = %r'% ppz
    print 'ppz.value = %r'% ppz.value
    print 'ppz.value.value = %r'% ppz.value.value
    print 'ppz.v.v= %r'% ppz.v.v

    print '\nDemo read-only pointer to pi in namespace math (the module) ...'
    import math
    ppi = PNS(math,'pi', 'r') # read only
    print 'math = %r' % math
    print 'ppi = %r' % ppi
    print 'ppi.v = %s' % ppi.v

    print '\nAttempting to set math.pi via ppi.v=3.14 (will work via math.pi, BTW !)'
    try: ppi.v = 3.14
    except Exception, e: print '%s: %s'% (e.__class__.__name__,e)
    else: print 'ppi.v set ok: %r' % ppi.v

    if __name__ == '__main__':
    test()
    ========================================================================

    Result of run:

    [10:03] C:\pywk\clp>pns.py
    Before change:
    ns.x='x value', ns.y='y value'
    Making pointer py point to ns.y ...
    ptr py=<PNS ptr to 'y' of <__main__.Namespace object at 0x007F9EC0>>
    Calling change(ns.x, py) ...
    After change:
    ns.x='x value', ns.y='new y'
    Before change1/change2:
    ns.x='x value', ns.y='new y'
    After calling change1(ns.x, py):
    ns.x='x value', ns.y='deep change in y'

    New pointer to non-existent ns.z:
    <PNS ptr to 'z' of <__main__.Namespace object at 0x007F9EC0>>
    Trying to access as yet nonexistent ns.z ...
    AttributeError: 'Namespace' object has no attribute 'z'
    Passing pz to change(ns.x, pz)...
    Result: ns.x='x value', ns.z='new y'

    Before deleting ns.y via py.v:
    ns.y='deep change in y', py.v='deep change in y'

    Deleting ns.y by del py.value ...
    Trying to access ns.y ...
    AttributeError: 'Namespace' object has no attribute 'y'

    Creating nsother name space to put pz in as a value ...
    <__main__.AnotherNS object at 0x007F92C0>
    Creating pointer ppz pointing to nsother.pz
    ppz = <PNS ptr to 'pz' of <__main__.AnotherNS object at 0x007F92C0>>
    ppz.value = <PNS ptr to 'z' of <__main__.Namespace object at 0x007F9EC0>>
    ppz.value.value = 'new y'
    ppz.v.v= 'new y'

    Demo read-only pointer to pi in namespace math (the module) ...
    math = <module 'math' (built-in)>
    ppi = <PNS ptr to 'pi' of <module 'math' (built-in)>>
    ppi.v = 3.14159265359

    Attempting to set math.pi via ppi.v=3.14 (will work via math.pi, BTW !)
    PNSError: Value write prohibited for <PNS ptr to 'pi' of <module 'math' (built-in)>>

    [10:03] C:\pywk\clp>

    >
    >Thanks for these thoughts and the time it took to post them, they really
    >made me think! (I mean that in a good way, of course ;) )
    >

    You're welcome. Hope this adds another useful angle.

    Regards,
    Bengt Richter
    Bengt Richter, Jul 27, 2003
    #14
  15. Bengt Richter wrote:

    > On Sun, 27 Jul 2003 04:51:11 GMT, Paul Paterson <> wrote:
    >
    > [...]
    >
    >
    >>To my eye, the [:] or [0] spelling of this makes the code look more
    >>complex than necessary, but I think you are on to something because if
    >>you spell it,
    >>
    >>def change(x, y):
    >> x = 'new x'
    >> y.update('new y')
    >>

    >
    >
    > For a general pointer, ISTM you want to be able to dereference it for both getting and setting.
    > The [:] or [0] syntax gets you that, and you could do it with p() for getting and p(val) for setting,
    > or p.getval() and p.update(), but all these get clumsy when you want to use the "pointer" on
    > both sides of an assignment statement, e.g.,
    >
    > p.update(p.getval()+' added text for target')
    >
    > where you could write more readably (IMO),
    >
    > p.value = p.value + 'added text for target'


    Yes, this feels right. Visually you have only one element to recognize
    (p.value) which makes it not much more complex than the original code. A
    single search and replace could also get rid of the pointer semantics if
    it was later decided that they were not required.

    >
    > or
    > p.v += 'added text for target'
    >
    >
    > Below is a class PNS that lets you spell as immediately above, using .value
    > (and .v for concise use) bound to a property that implements the accessing of
    > what's pointed/referred to. I gave it a somewhat informative __repr__ method also,
    > so the test prints better. As you will note, this is now separate from any particular
    > name space. Any object that supports getattr and/or setattr can be used. I also
    > threw in a permissions parameter to control read/write/delete. See nsother and math
    > in examples. Note also that a pointer may point to another pointer, allowing cascaded
    > dereferencing spelled p.v.v etc.
    >
    >
    >>with the relevant changes to the Ptr class then it could certainly grow
    >>on me. The things I like are,
    >>
    >>- no new variable names in the 'change' function so it looks similar to
    >>the original code
    >>- the mechanism for propogating changes to the caller scope is explicit
    >>- 'y' can be passed on to another function if needed and things are
    >>still clear
    >>
    >>eg,
    >>
    >>def change(x, y):
    >> x = 'new x'
    >> change2(y)
    >>
    >>def change2(y):
    >> y.update('deep change in y')
    >>

    >
    > If you use PNS, that will be spelled
    >
    > def change(x, y):
    > x = 'new x'
    > change2(y)
    >
    > def change2(y):
    > y.value = 'deep change in y'


    Looks good!

    <snip PNS code + tests>

    I like this a lot - it is much clearer than a bare namespace when you
    start passing these things around. In fact, I don't think the namespace
    can actually work in the change/change2 example above. Consider,

    def change(x, y):
    x = 'new x'
    change2(y)

    def change2(z):
    z = 'deep change to y'

    Using the bare namespace you would have to pass ns to 'change' and then
    address 'y' as ns.y ... but then when you pass it to change2 it has to
    be called z, but you can't call it ns.z ...

    Using the pointer system (or sisters?!) you pass the "pointer" and then
    always refer to it as localname.value. Neat!

    I'm not sure if I would recommend this in general but for my application
    it seems to be a nice approach.

    >
    >>Thanks for these thoughts and the time it took to post them, they really
    >>made me think! (I mean that in a good way, of course ;) )
    >>

    >
    > You're welcome. Hope this adds another useful angle.


    Absolutely! Thanks again.

    Paul
    Paul Paterson, Jul 28, 2003
    #15
  16. Interesting thread here, I'm enjoying reading this. Of course, I know
    next to nothing about Visual Basic, so I don't know if I can really
    help all that much. But it doesn't sound like implementing something
    like pointers is that much trouble, just a matter of getting some
    syntax you like. Now if you were translating C into Python, I suspect
    you might need a more complicated pointer model. I mean, half the use
    of pointers in C is to iterate down simple arrays, like strings, so
    you'd have to make something more like this:

    class CStack(object):
    def __init__():
    self.stack = []
    self.names = {}
    def malloc(sizeinbytes, object_name):
    self.names[object_name] = len(self.stack)
    for i in range(sizeinbytes):
    self.stack.append(0)
    def pobj(pointer_index):
    return self.stack[pointer_index]
    def paddr(object_name):
    return self.names[object_name]

    or something like that, obviously this isn't fleshed out, but given
    the breathtakingly simple memory model of C, it should be a snap to
    code up in Python. Hmmm.... I wonder if the Python-In-Python people
    are trying anything like this to convert the C library code for
    CPython. Or would it even be worth it? I think I'll meditate on
    that......
    Corey Coughlin, Jul 28, 2003
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Gabriel Rossetti
    Replies:
    0
    Views:
    1,312
    Gabriel Rossetti
    Aug 29, 2008
  2. John [H2O]

    modifying locals

    John [H2O], Oct 30, 2008, in forum: Python
    Replies:
    6
    Views:
    287
    M.-A. Lemburg
    Oct 31, 2008
  3. Helmut Jarausch
    Replies:
    5
    Views:
    241
    Helmut Jarausch
    Dec 3, 2008
  4. Ed Anuff
    Replies:
    2
    Views:
    209
    Ed Anuff
    Sep 14, 2009
  5. Onion Knight
    Replies:
    1
    Views:
    93
    Onion Knight
    Mar 5, 2007
Loading...

Share This Page