Beginner question - How to effectively pass a large list

J

J.R.

Hi folks,

The python can only support passing value in function call (right?), I'm
wondering how to effectively pass a large parameter, such as a large list or
dictionary?

It could achieved by pointer in C++, is there such way in Python?

Thansk in advance.
J.R.
 
B

Bruno Desthuilliers

J.R. said:
Hi folks,

The python can only support passing value in function call (right?),

let's see :
>>> l = range(5)
>>> def modif(alist): alist[0] = 'allo' ....
>>> l [0, 1, 2, 3, 4]
>>> modif(l)
>>> l
['allo', 1, 2, 3, 4]

Er... Ok, let's try something else :

.... pass
....'Titi'


Well... You may want to read more about bindings (not 'assignements') in
Python.

(hint : it already *does* work like pointers in C[++] - at least for
mutable objects).

Bruno
 
B

Ben Finney

The python can only support passing value in function call (right?)

Wrong. Function parameters in Python are always passed by reference,
not by value. The local function parameter gets a binding to the same
object that was passed, so there are not two copies of the object in
memory.

Of course, because of the way that Python treats assignment, you can
then change that local parameter within the function and it will re-bind
to the new value, without altering the original value passed.
 
D

Donn Cave

Bruno Desthuilliers said:
(hint : it already *does* work like pointers in C[++] - at least for
mutable objects).

For any and all objects, irrespective of mutability.

Donn Cave, (e-mail address removed)
 
J

J.R.

Thanks for the response.

I got following conclusion after reading your reply and other documents:

Actually, the python is passing the identity (i.e. memory address) of each
parameter, and it will bind to a local name within the function.

Right?

Thanks,
J.R.

Donn Cave said:
Bruno Desthuilliers said:
(hint : it already *does* work like pointers in C[++] - at least for
mutable objects).

For any and all objects, irrespective of mutability.

Donn Cave, (e-mail address removed)
 
T

Terry Reedy

J.R. said:
I got following conclusion after reading your reply and other documents:

Actually, the python is passing the identity (i.e. memory address) of each
parameter, and it will bind to a local name
within the function.

Some months ago, there was a long thread on
whether Python function calling is 'call by name',
'call by value', or something else. Without
reiterating long discussion, I think from a user's
viewpoint, it is best considered 'call by
name-binding', with the value-carrying object
'passing' being by cross-namespace binding. One
can think of the return process as being similar
in that the return object is substituted for the
function name as if it had been bound to that
name.

The usage I believe promoted by Knuth and
supported by some here defines 'parameter' as the
within-function local name and 'argument' as the
outside value/object bound to that name at a
particular function call. Yes, CPython does that
with ids that are memory addresses, but that is
implementation rather than part of the language
definition itself. Who know what we do when we
act as Python interpreters!

Terry J. Reedy
 
D

Donn Cave

"J.R. said:
Thanks for the response.

I got following conclusion after reading your reply and other documents:

Actually, the python is passing the identity (i.e. memory address) of each
parameter, and it will bind to a local name within the function.

That should work. This notion of binding a object to a name or
data structure is all over Python, as you have probably noticed,
and it will be time well spent if you experiment with it a little.

For example, how could you verify that arrays are passed without
copying? Unless you are unusually dense for a programmer or have
no access to a Python interpreter, that could be no more than a
couple of minutes work. As penance for having failed to do this,
I assign a more mysterious problem to you:

def f(d=[]):
d.append(0)
print d
f()
f()

Explain results. When is d bound?

Here's an easier one:

a = {'z': 0}
b = [a] * 4
b[0]['z'] = 1
print b

If the result is anywhere near a surprise to you, then you would
do well to stick to this line of inquiry if you want to do anything
more than the most trivial Python programming.

Donn Cave, (e-mail address removed)
 
J

J.R.

As penance for having failed to do this,
I assign a more mysterious problem to you:

def f(d=[]):
d.append(0)
print d
f()
f()

Explain results. When is d bound?

I found that the "problem" showed above is caused by the python principle
"everything
is object".

Here the function "f" is an object as well, the object function f is created
once this function is defined. And there is tuple attribute(func_defaults)
inside
function object to record the default arguments.
print id(d)
d.append(0)
print d
f.func_defaults ([],)
id(f.func_defaults[0]) 11279792
f()
11279792
[0]
11279792
[0, 0]
11279952
[1, 0]

1. There is no value passed to the default argument
The name "d" is bound to the first element of the f.func_defaults. Since the
function "f" is an
object, which will be kept alive as long as there is name (current is "f")
refered to it, the
list in the func_defaults shall be accumulated by each invoking.

2. There is a value passed to the default argument
The name "d" will be bound to the passed object, it's proven from the
different identity showd above.

I think we could eliminate such accumulation effect by changing the function
as follow:
d = d+[0]
print d

J.R.
 
S

Stian =?iso-8859-1?Q?S=F8iland?=

* J.R. spake thusly:
def f(d=[]):
d.append(0)
print d
f()
f()
Explain results. When is d bound?

When is this issue going to be resolved? Enough newbie-pythoners have
made this mistake now.

Why not evaluate the parameter lists at calltime instead of definition
time? This should work the same way as lambdas.
f = lambda: []
a=f()
b=f()
a.append(1)
print a, b
[1] []


Maybe this could be defined in a similar way to remind of the
"lazy-evaluation":

def getvalue(cls):
return "OK"

class SomeClass:
def blapp(something: getvalue(), other: []):
print something, other

This way, the lambda forms defined after 'something' and 'other' are
evaluated each time the function is called without supplying those
parameters.

The lambda forms could be evaluated as if within the class-block, and
therefore they might actually use other values defined within that
namespace. However, this might again confuse users, as subclassed
attributes and instance attributes would not be resolved that way.

(Note that default-parameters-lambdas by today does NOT resolve this way:
.... ting = 15
.... def fix(self, per=lambda: ting):
.... print per()
....Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 4, in fix
File "<stdin>", line 3, in <lambda>
NameError: global name 'ting' is not defined



Finally, things could also be done like this, to avoid confusion to all
those new folks (our main goal):

def blapp(something=getvalue(), other=[]):

getvalue() should be called if something-paramer is not specified (ie.
expression something=getvalue() is evaluated), and likewise for other.


Although this would break existing code and need to be delayed to at
least 3.0 and implemented in the __future__.

I must say I can't see the reason to not delay evalution now that we
have nested scopes. This way, even this would work:


class A:
default_height=100
default_width=200
def make_picture(self, height=default_height,
width=default_width):
self.gui.do_blabla(height, width)
set_default_width = classmethod(set_default_width)

a = A()
a.make_picture()
A.default_width = 150
a.make_picture() # Now with new default width


One might argue that this could could benefit from resolving
self.default_width instead, that would still require setting
height=None and testing inside make_picture.
 
A

Alan Gauld

Actually, the python is passing the identity (i.e. memory address) of each
parameter, and it will bind to a local name within the function.

Right?

Nope.
This is one case where understanding something of the insides of
Python helps. Basically Python variables are dictionary entries.
The variable values are the the dictionary values associated with
the variable names which are the dictionary keys.

Thus when you pass an argument to a function you are passing a
dictionary key. When the function uses the argument it looks up
the dictionary and uses the value found there.

This applies to all sorts of things in Python including modules -
a local dictionary associated with the module, and classes -
another dictionary. Dictionaries are fundamental to how Python
works and memory addresses per se play no part in the procedings.

HTH

Alan G.

Author of the Learn to Program website
http://www.freenetpages.co.uk/hp/alan.gauld
 
T

Terry Reedy

message
When is this issue going to be resolved? Enough newbie-pythoners have
made this mistake now.

I am puzzled as to why. When I learned Python, I
read something to the effect that default value
expressions are evaluated at definition time. I
understood that the resulting objects were saved
for later use (parameter binding) when needed (as
default for value not given). I believed this and
that was that.
Why not evaluate the parameter lists at calltime instead of definition
time?

Run time code belongs in the body of the function
and the corresponding byte code in the code
object. To me anyway.

Terry J. Reedy
 
P

Paul Rubin

Greg Ewing (using news.cis.dfn.de) said:
In this case, evaluating the default args at call time would
have a negative payoff, since it would slow down every call to
the function in cases where the default value doesn't need
to be evaluated more than once.

In those cases the compiler should notice it and generate appropriate
code to evaluate the default arg just once. In many of the cases it
can put a static value into the .pyc file.
 
G

Greg Ewing (using news.cis.dfn.de)

Stian said:
When is this issue going to be resolved? Enough newbie-pythoners have
made this mistake now.

Changes are rarely if ever made to Python for the sole reason
of reducing newbie mistakes. There needs to be a payoff for
long-term use of the language as well.

In this case, evaluating the default args at call time would
have a negative payoff, since it would slow down every call to
the function in cases where the default value doesn't need
to be evaluated more than once.
 
B

Bengt Richter

* J.R. spake thusly:
def f(d=[]):
d.append(0)
print d
f()
f()
Explain results. When is d bound?

When is this issue going to be resolved? Enough newbie-pythoners have
made this mistake now.
It works as designed. The default parameter value bindings are made a def time.
If you want to do them at call time, the idiom is

def f(d=None):
if d is None: d = []
d.append(0)
print d
Why not evaluate the parameter lists at calltime instead of definition
time? This should work the same way as lambdas.
Lambdas do work the same as defs, except for the automatic name binding
and the limitation to an expression as the body.
f = lambda: []
a=f()
b=f()
a.append(1)
print a, b
[1] []
The comparable def code to your lambda would be
def f(): return []
The above is misguiding attention away from your point, IMO.
Maybe this could be defined in a similar way to remind of the
"lazy-evaluation":

def getvalue(cls):
return "OK"

class SomeClass:
def blapp(something: getvalue(), other: []):
print something, other
This might be an interesting concise spelling to accomplish what the following does:
(BTW, you need a self for normal methods)
... def __init__(self, fun): self.fun = fun
... ... def blapp(self, something=Defer(lambda: getvalue()), other=Defer(lambda:[])):
... if isinstance(something, Defer): something = something.fun()
... if isinstance(other, Defer): other = other.fun()
... print something, other
...
>>> def getvalue(): return 'gotten_value' ...
>>> sc = SomeClass()
>>> sc.blapp() gotten_value []
>>> sc.blapp(1) 1 []
>>> sc.blapp('one', 'two')
one two
This way, the lambda forms defined after 'something' and 'other' are
evaluated each time the function is called without supplying those
parameters.

The lambda forms could be evaluated as if within the class-block, and
therefore they might actually use other values defined within that
namespace. However, this might again confuse users, as subclassed
attributes and instance attributes would not be resolved that way.
Yes, messy. ISTM better to let them be defined in the same scope
as the def, as in my example above.
(Note that default-parameters-lambdas by today does NOT resolve this way:

... ting = 15
... def fix(self, per=lambda: ting):
... print per()
...
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 4, in fix
File "<stdin>", line 3, in <lambda>
NameError: global name 'ting' is not defined
Right, though you could get the functionality if you wanted to.
Finally, things could also be done like this, to avoid confusion to all
those new folks (our main goal):
If they're confused because their preconceptions are filtering out
anything discordant, pandering to them would not serve the language.
def blapp(something=getvalue(), other=[]):

getvalue() should be called if something-paramer is not specified (ie.
expression something=getvalue() is evaluated), and likewise for other.
No. I like your version with the lambda-colons much better.
Although this would break existing code and need to be delayed to at
least 3.0 and implemented in the __future__.

I must say I can't see the reason to not delay evalution now that we
Are you sure you counted your negatives there? ;-)
have nested scopes. This way, even this would work:
You're assuming default_height will refer to A.default_height
class A:
default_height=100
default_width=200
def make_picture(self, height=default_height,
width=default_width):
self.gui.do_blabla(height, width)
set_default_width = classmethod(set_default_width)
IMO better:

class A:
default_height=100
default_width=200
def make_picture(self, height: A.default_height,
width: A.default_width):
self.gui.do_blabla(height, width)
set_default_width = classmethod(set_default_width)
a = A()
a.make_picture()
A.default_width = 150
a.make_picture() # Now with new default width


One might argue that this could could benefit from resolving
self.default_width instead, that would still require setting
height=None and testing inside make_picture.

Or maybe have the implict lambda have an implicit self arg bound like a method
if it is the default arg of a method, ie., so you could write

class A:
default_height=100
default_width=200
def make_picture(self, height: self.default_height,
width: self.default_width):
self.gui.do_blabla(height, width)

Then when the function was called, it would be like getting an implicit something like
... default_height=100
... default_width=200
... def make_picture(self, height=Defer(lambda self: self.default_height),
... width=Defer(lambda self: self.default_width)):
... if isinstance(height, Defer): height = height.fun.__get__(self)()
... if isinstance(width, Defer): width = width.fun.__get__(self)()
... self.gui.do_blabla(height, width)
... class gui(object): # something to catch the above ;-/
... def do_blabla(h,w): print 'h=%r, w=%r'%(h,w)
... do_blabla = staticmethod(do_blabla)
... h='one', w='two'

Obviously the nested class gui is just to make the self.gui.do_blabla call work as spelled ;-)

I doubt if this is going to make it, but I think it's feasible without backwards breakage.
Write a PEP if you want to pursue something like that, but don't get overexcited ;-)

Regards,
Bengt Richter
 
C

Carl Banks

Terry said:
message


I am puzzled as to why. When I learned Python, I
read something to the effect that default value
expressions are evaluated at definition time. I
understood that the resulting objects were saved
for later use (parameter binding) when needed (as
default for value not given). I believed this and
that was that.


I am puzzled as to why you're puzzled. Not everyone who reads the
manual pays attention to the time of evaluation explanation, if the
manual they're using even covers. Not everyone stops and says, "Oh my
God, I don't know whether this is evaluated when the function is
defined or called. I better find out." (And of course, not everyone
reads the manual.)

It seems that most people who haven't thought about time of evaluation
tend to expect it to be evaluated when the function is called; I know
I would expect this. (I think I even made the mistake.)
 
C

Carl Banks

Paul said:
In those cases the compiler should notice it and generate appropriate
code to evaluate the default arg just once. In many of the cases it
can put a static value into the .pyc file.

In a perfect world, that would be a good way to do it. However,
Python is NOT in the business of deciding whether an arbitrary object
is constant or not, except maybe in the parsing stages. Internally,
it's just not built that way.

If I were designing, I would definitely make it the language's (and
extension writers') business, because there is a lot of opportunity
for optimization.
 
C

Carl Banks

Stian said:
* J.R. spake thusly:
def f(d=[]):
d.append(0)
print d
f()
f()
Explain results. When is d bound?

When is this issue going to be resolved? Enough newbie-pythoners have
made this mistake now.

Why not evaluate the parameter lists at calltime instead of definition
time? This should work the same way as lambdas.


Consider something like this:

def func(param=((1,2),(3,4),(5,6),(7,8))):
whatever

Do you really want to be building a big-ass nested tuple every time
the function is called?

Python evaluates default args at time of definition mostly for
performance reasons (and maybe so we could simulate closures before we
had real closures). My gut feeling is, moving the evaluation to call
time would be too much of a performance hit to justify it.
 
P

Paul Rubin

Carl Banks said:
It seems that most people who haven't thought about time of evaluation
tend to expect it to be evaluated when the function is called; I know
I would expect this. (I think I even made the mistake.)

The principle of least astonishment then suggests that Python made
a suboptical choice.
 
P

Paul Rubin

Carl Banks said:
Consider something like this:

def func(param=((1,2),(3,4),(5,6),(7,8))):
whatever

Do you really want to be building a big-ass nested tuple every time
the function is called?

Come on, the compiler can easily recognize that that list is constant.
Python evaluates default args at time of definition mostly for
performance reasons (and maybe so we could simulate closures before we
had real closures). My gut feeling is, moving the evaluation to call
time would be too much of a performance hit to justify it.

Python takes so many other performance hits for the sake of
convenience and/or clarity that this particular one would be miniscule
by comparison.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
474,262
Messages
2,571,055
Members
48,769
Latest member
Clifft

Latest Threads

Top