Ruby blocks in Python, a suggestion

J

John Roth

Ville Vainio said:
John> The reason for saying "inject" is that, under some
John> circumstances, the block can access the host function's
John> locals, which you wouldn't expect from a normal function
John> that's passed as a parameter.

I think that's what they call 'dynamic scope', and it has been
generally considered a bad idea.

Like operator overloading, multiple inheritance and other
things, it depends on what you're using it for. There are very
few language features that can't be used to confuse rather
than clarify.

John Roth
 
M

Michael Walter

Ville said:
John> The reason for saying "inject" is that, under some
John> circumstances, the block can access the host function's
John> locals, which you wouldn't expect from a normal function
John> that's passed as a parameter.

I think that's what they call 'dynamic scope', and it has been
generally considered a bad idea.
For instance, "global variables" in Common Lisp usually have dynamic
extent. This allows temporarily modifying a global variable/parameter
rather easily, as in:

(let ((*print-readably* nil))
<body>)

in contrast to:

(let ((old-value *print-readably*))
(unwind-protect
(progn
(setf *print-readably* nil)
<body>)
(setf *print-readably* old-value)))

(in python:
global print_readably
old_value = print_readably
try:
print_readably = 0
<body>
finally:
print_readably = old_value)

So, 'generally' could be not as 'generally' as one might think.

Cheers,
Michael

PS: my CL is rusty
 
J

Jim Weirich

Ville Vainio said:
John> The reason for saying "inject" is that, under some
John> circumstances, the block can access the host function's
John> locals, which you wouldn't expect from a normal function
John> that's passed as a parameter.

I think that's what they call 'dynamic scope', and it has been
generally considered a bad idea.

Just to be clear: Ruby blocks can access locals declared in the same
scope that the block/closure is created in. This is static scoping.

Blocks do not access the locals of the function in which they are
executed. That would be dynamic scoping.

I think John is saying the first, although the meaning of "host
function" is a little vague to me.
 
J

John Roth

Jim Weirich said:
John> The reason for saying "inject" is that, under some
John> circumstances, the block can access the host function's
John> locals, which you wouldn't expect from a normal function
John> that's passed as a parameter.

I think that's what they call 'dynamic scope', and it has been
generally considered a bad idea.

Just to be clear: Ruby blocks can access locals declared in the same
scope that the block/closure is created in. This is static scoping.

Blocks do not access the locals of the function in which they are
executed. That would be dynamic scoping.

I think John is saying the first, although the meaning of "host
function" is a little vague to me.[/QUOTE]

Actually, I was saying the second, and I stand (or rather, sit)
corrected.

John Roth
 
L

Lonnie Princehouse

Yes!

def my_codeblock:
assert(y == x*2)

for (x,y) in [(1,2), (2,4), (3,6)]:
exec my_codeblock

This is by far the prettiest syntax for blocks presented thus far =)
A function really is just a special case of a code block that (a)
operates within its own scope and (b) defines a mapping between that
scope and the calling scope. This syntax preserves that relationship,
while also avoiding adding new keywords like "defblock" to the
language. I would go so far as to say the code block should be
callable just like a function that takes no arguments-

def my_codeblock:
assert(y == x*2)

for (x,y) in [(1,2), (2,4), (3,6)]:
my_codeblock()

Someone should put in a PEP for this... it seems like these code
blocks could also help to speed up execution in certain circumstances,
since the interpreter doesn't need to create and then dispose of a new
scope for their execution.

And now for something completely different-
Why not make "def" into an expression that returns a
function/block, instead of a statement? This could obviate lambda by
providing anonymous functions that aren't limited to one line.

e.g.

my_function = def (x,y):
return sqrt(x**2 + y**2)

my_codeblock = def:
assert(y == x*2)

(Note- I'm not advocating the replacement of the current syntax, just
an extension) Of course, the indentation requirement might be a
problem when one tries to embed these expressions inside of other
expressions. Not sure how that would work. Semicolons, perhaps, but
then we start to look like Perl :p



-ljp
 
J

Joe Mason

This is by far the prettiest syntax for blocks presented thus far =)
A function really is just a special case of a code block that (a)
operates within its own scope and (b) defines a mapping between that
scope and the calling scope. This syntax preserves that relationship,
while also avoiding adding new keywords like "defblock" to the
language. I would go so far as to say the code block should be
callable just like a function that takes no arguments-

def my_codeblock:
assert(y == x*2)

for (x,y) in [(1,2), (2,4), (3,6)]:
my_codeblock()

What's the point? If you've got to make a named item before the for
loop anyway, why not just use a function? The point of code blocks is
to be anonymous, and declared inline.

Joe
 
G

Glenn Andreas

Joe Mason said:
Lonnie said:
This is by far the prettiest syntax for blocks presented thus far =)
A function really is just a special case of a code block that (a)
operates within its own scope and (b) defines a mapping between that
scope and the calling scope. This syntax preserves that relationship,
while also avoiding adding new keywords like "defblock" to the
language. I would go so far as to say the code block should be
callable just like a function that takes no arguments-

def my_codeblock:
assert(y == x*2)

for (x,y) in [(1,2), (2,4), (3,6)]:
my_codeblock()

What's the point? If you've got to make a named item before the for
loop anyway, why not just use a function? The point of code blocks is
to be anonymous, and declared inline.

Joe

Is it?

Or is the point of the codeblock to execute in the same scope as where
it is called (i.e., dynamic scoping).

It's fairly easy to emulate the anonymous part (by not having it inline,
and just using a dummy name, which makes it less anonymous), but it's
much harder to emulate the dynamic scoping (though with some magic of
the stack frame it could probably be done, but it might be pretty ugly).

Dynamic scoping be partially simulated easily enough:

def block(b):
import sys
caller_frame = sys._getframe().f_back
exec b.func_code in caller_frame.f_locals

def test():
def my_codeblock():
assert(y == x * 2)

for (x,y) in [(1,2), (2,4), (3,6)]:
block(my_codeblock)


(note that this won't actually work since you can't exec a function with
"free variables", so obviously the "injection" in block would need to be
a bit more clever - perhaps building a code object dynamically).

It may be possible to use function decorater to achieve the dynamic
scoping nature (and thus achieving one of those goals without having to
make further changes to the python "core" like changing the grammar
would):

def test():
def my_codeblock() [block]:
assert(y == x * 2)

for (x,y) in [(1,2), (2,4), (3,6)]:
my_codeblock()
 
J

Joe Mason

Or is the point of the codeblock to execute in the same scope as where
it is called (i.e., dynamic scoping).

Why not just pass parameters? Dynamic scoping is a pain because it
introduces rules you need to remember. Ruby's already decided their
rules were wrong and they need to change them for version 2.0.
def test():
def my_codeblock() [block]:
assert(y == x * 2)

for (x,y) in [(1,2), (2,4), (3,6)]:
my_codeblock()

I still don't see the advantage over

def test():
def my_func(x, y):
assert(y == x * 2)

for (x, y) in [(1,2), (2,4), (3,6)]:
my_func(x, y)

With more parameters it'd be more convenient, but also much easier to be
sloppy.

Joe
 
V

Ville Vainio

Joe> I still don't see the advantage over

Joe> def test():
Joe> def my_func(x, y):
Joe> assert(y == x * 2)

Joe> for (x, y) in [(1,2), (2,4), (3,6)]:
Joe> my_func(x, y)

Joe> With more parameters it'd be more convenient, but also much
Joe> easier to be sloppy.

Add z = 10 to the would-be block my_func, and you'll see the
difference. With statically scoped block semantics (instead of
function semantics), it would bind z as a local variable in test,
instead of local variable in my_func. It's not possible with functions
currently, because binding it in my_func makes it a local variable in
my_func.

Of course the cleanest option of all would be the ability to
explicitly specify variables as 'free variables' so that binding them
wouldn't make them local variables, i.e.:

def f():
x = 10
def inner():
freevar x
x = 4 # bind in f
y = 5 # bind in inner

def promiscuous(param):
freevar * # all vars except param are free - so this
# is essentially a "block"!
x = 45 # bind in f
y = 12 # bind in f
z = 12323 # bind in f
 
G

Glenn Andreas

Joe Mason said:
Glenn Andreas said:
Or is the point of the codeblock to execute in the same scope as where
it is called (i.e., dynamic scoping).

Why not just pass parameters? Dynamic scoping is a pain because it
introduces rules you need to remember. Ruby's already decided their
rules were wrong and they need to change them for version 2.0.
def test():
def my_codeblock() [block]:
assert(y == x * 2)

for (x,y) in [(1,2), (2,4), (3,6)]:
my_codeblock()

I still don't see the advantage over

def test():
def my_func(x, y):
assert(y == x * 2)

for (x, y) in [(1,2), (2,4), (3,6)]:
my_func(x, y)

With more parameters it'd be more convenient, but also much easier to be
sloppy.

Joe

how about:


def test():
def my_codeblock() [block]:
z += x * y

z = 0
for (x,y) in [(1,2), (2,4), (3,6)]:
my_codeblock()
print z

In this way, code blocks can be used like macros (except not expanded at
compile time), in that they can change the context that they are called
in.

If one figured out a good, clean way to make anonymous code blocks that
could be passed as expressions, you could then make new control
structures (after all, in Smalltalk, all control structures are
implemented using codeblocks):

def switch(expr, cases, default = None):
if cases.has_key(expr):
cases[expr]()
elif default:
default()


z = 0
switch(x+y,
{ 4 : def[block]:
z = 4
5 : def[block]:
z = 10
}
default = def[block]:
z = 20
)
print z
 
L

Lonnie Princehouse

Joe Mason said:
I still don't see the advantage over

def test():
def my_func(x, y):
assert(y == x * 2)

for (x, y) in [(1,2), (2,4), (3,6)]:
my_func(x, y)

In this particular example, there's no advantage to using codeblocks
over functions. However, dynamically scoped blocks would facilitate a
range of things that are difficult to do with functions, particularly
in the metaprogramming arena. If you're trying to write code that
writes code, it's really handy to have blocks that execute in the
current scope instead of in a new one.

If dynamically scoped blocks were added to the language, it would give
metaprogrammers a much cleaner syntax that could replace a lot of
exec, eval, and compile calls, not to mention it would reduce the need
to tinker directly with frames.

.... Speaking of which, it's too bad that Glenn's sys._getframe() hack
doesn't quite work. Can anybody figure a way to inject locals created
these "blocks" back into the calling frame's locals? Illustration of
the problem-

class block:
def __init__(self, f):
self.f = f
def __call__(self):
import sys
exec self.f.func_code in sys._getframe(1).f_locals

def next_fibonacci_term():
fibonacci_sequence.append(fibonacci_sequence[-2] +
fibonacci_sequence[-1])
i = len(fibonacci_sequence) - 1

nft_block = block(next_fibonacci_term)

fibonacci_sequence = [0,1]

[nft_block() for n in range(20)]

print fibonacci_sequence
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987,
1597, 2584, 4181, 6765, 10946]

print i
# Traceback (most recent call last):
# File "<stdin>", line 1, in ?
# NameError: name 'i' is not defined

Conclusion-
"exec foo in scope" appears to execute foo in a _copy_ of scope, s.t.
we can modify objects that are contained in the scope, but assignment
won't work.
 
J

Joe Mason

Joe> I still don't see the advantage over

Joe> def test():
Joe> def my_func(x, y):
Joe> assert(y == x * 2)

Joe> for (x, y) in [(1,2), (2,4), (3,6)]:
Joe> my_func(x, y)

Joe> With more parameters it'd be more convenient, but also much
Joe> easier to be sloppy.

Add z = 10 to the would-be block my_func, and you'll see the
difference. With statically scoped block semantics (instead of
function semantics), it would bind z as a local variable in test,
instead of local variable in my_func. It's not possible with functions
currently, because binding it in my_func makes it a local variable in
my_func.

Yes, it would. I consider that a bad thing, because now if I add a new
variable before and after the "block call", the block can fiddle with
it accidentally. If the block's in-line this isn't (as much of) a
problem since it's easy to scan.

If you actually want to update the value of z in the block, what's wrong
with this?

def test():
def my_func(x, y, z):
assert(y == x * 2)
z = 10
return z

for (x, y) in [(1, 2), (2, 4), (3, 6)]:
z = my_func(x, y, z)

(Of course, passing z to my_func and then not using it doesn't make much
sense, but maybe it's just leaving it open for expansion.

Joe
 
J

Joe Mason

In this particular example, there's no advantage to using codeblocks
over functions. However, dynamically scoped blocks would facilitate a
range of things that are difficult to do with functions, particularly
in the metaprogramming arena. If you're trying to write code that
writes code, it's really handy to have blocks that execute in the
current scope instead of in a new one.

If dynamically scoped blocks were added to the language, it would give
metaprogrammers a much cleaner syntax that could replace a lot of
exec, eval, and compile calls, not to mention it would reduce the need
to tinker directly with frames.

Now that I can buy. I'd prefer to add explicit metaprogramming-support
facilities than a generic "block" construct, though.

Joe
 
H

Hung Jung Lu

def next_fibonacci_term():
fibonacci_sequence.append(fibonacci_sequence[-2] +
fibonacci_sequence[-1])
i = len(fibonacci_sequence) - 1

nft_block = block(next_fibonacci_term)

fibonacci_sequence = [0,1]

[nft_block() for n in range(20)]

print fibonacci_sequence
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987,
1597, 2584, 4181, 6765, 10946]

print i
# Traceback (most recent call last):
# File "<stdin>", line 1, in ?
# NameError: name 'i' is not defined

Conclusion-
"exec foo in scope" appears to execute foo in a _copy_ of scope, s.t.
we can modify objects that are contained in the scope, but assignment
won't work.
-----------------------------

Run-time codeblocks DO ALREADY EXIST in Python, via the compile()
function. So we are basically talking about the need for COMPILE-TIME
codeblocks.

Python run-time codeblocks are fully functional. Before making any
comment/question about codeblocks in Python, the first thing is to
check it with the compile() function. (Questions may include: Can you
assign to local variables? Can you assign to global variables? Can you
have break/continue statements in codeblocks? etc. etc.)

#------------------------------------------------
nft_block = compile('''

fibonacci_sequence.append(
fibonacci_sequence[-2] + fibonacci_sequence[-1])
i = len(fibonacci_sequence) - 1

''', '<string>', 'exec')

fibonacci_sequence = [0, 1]
for n in range(20):
exec nft_block

print fibonacci_sequence
# [0, 1, ..., 10946]

print i
# 21
#------------------------------------------------

The compile() approach has a scope problem, too. That is, when used
inside functions, it won't bind to global variables automatically. But
one can always use the 'global' statement explicitly inside the
codeblock, if necessary. The compile() function is not a new thing. It
has been battle-tested from early days of Python. It's just an
inconceivable mistake that Python does not port the run-time codeblock
into compile-time.

regards,

Hung Jung
 
L

Lonnie Princehouse

Run-time codeblocks DO ALREADY EXIST in Python, via the compile()
function. So we are basically talking about the need for COMPILE-TIME
codeblocks.

True =) Guess I was a little carried away with trying to fake
compile-time codeblocks. It just seems so kludgy, having to write
code in a string...



# But now it's an idee fixe. Here we go again!
import inspect, re

def block(f):
# won't work from stdin, function code must be in a file
# decompyle could help?
source = inspect.getsource(f)
source = source[source.index('\n')+1:]
indent = re.search('^(\s+)\S',source).group(1)
source = re.sub('(?m)^%s' % indent, '', source)
return compile(source, '<string>', 'exec')

def my_codeblock():
z = x + y

my_codeblock = block(my_codeblock)

for (x,y) in [(1,2), (2,4), (3,6)]:
exec my_codeblock
 
L

Lonnie Princehouse

(my apologies if this message appears more than once... mozilla is
behaving strangely)

Run-time codeblocks DO ALREADY EXIST in Python, via the compile()
function. So we are basically talking about the need for COMPILE-TIME
codeblocks.

True =) Guess I was a little carried away with trying to fake
compile-time codeblocks. It just seems so kludgy, having to write
code in a string...



# But now it's an idee fixe. Here we go again!
import inspect, re

def block(f):
# won't work from stdin, function code must be in a file
# decompyle could help?
source = inspect.getsource(f)
source = source[source.index('\n')+1:]
indent = re.search('^(\s+)\S',source).group(1)
source = re.sub('(?m)^%s' % indent, '', source)
return compile(source, '<string>', 'exec')

def my_codeblock():
z = x + y

my_codeblock = block(my_codeblock)

for (x,y) in [(1,2), (2,4), (3,6)]:
exec my_codeblock
 
H

Hung Jung Lu

# But now it's an idee fixe. Here we go again!
import inspect, re

def block(f):
# won't work from stdin, function code must be in a file
# decompyle could help?
source = inspect.getsource(f)
source = source[source.index('\n')+1:]
indent = re.search('^(\s+)\S',source).group(1)
source = re.sub('(?m)^%s' % indent, '', source)
return compile(source, '<string>', 'exec')

def my_codeblock():
z = x + y

my_codeblock = block(my_codeblock)

for (x,y) in [(1,2), (2,4), (3,6)]:
exec my_codeblock

Man, it's a hack all right, but what a brilliant one! :) Never ever
seen anyone coming up with it.

Sure, it's not debuggable in the sense that you can't place break
points, and when there is an exception you won't see the exact line.
However, this part can be solved by generating small files in the
block() function, one per codeblock. Then, instead of '<string>', pass
the file name. Not only you will be able to see the line where error
happened, but you will also be able to put break points, or to step
through the code lines. The only thing to remember is to know where
you need to make the changes: not in the codeblock files, but in the
original place of definition. :) This can be helped by pre-pending
some explanatory messages in the codeblock files (pointing out the
location of the original file, i.e., via inspect.getsourcefile() and
inspect.getsourcefilelines()). Of course, in shipped products you'll
suppress the generation of the codeblock files.

It's functional. It's usable. It's compiler-checked. The refined
version should be published as a recipe!

Man! Wow! Triple Wow!

regards,

Hung Jung
 
M

Michele Simionato

Python's lack of compile-time, easily-debuggable codeblock is one of
the obvious major shortcomings/mistakes of the language.

Why ??
Codeblocks are absolutely wonderful for metaprogramming, to the point of
being essential.

Why ??
This topic has been discussed before, and I am not going
back there. When people are short-sighted, there is no cure. :) Other
people can take over the issue.

I have not followed in detail your discussion about code blocks, scattered
in different threads in different times, so please point out to me the posts
where your substain your claims.

My experience with code blocks has been the following:

1. At least one year ago I discovered on my own that Python has already
run time code blocks.

2. More or less at the same time I discovered the hack proposed by
Lonnie Princehouse which I used to implement macros in Python as
a proof of concept.

3. I didn't find any compelling use case for both 1 and 2; actually I
thought they were confusing and un-needed constructs, so after having
implemented them, I forgot them.

I may have been wrong about point 3. If so, please tell me why.
Otherwise, please stop making unsubstained claims and give
explanations and/or references to previous posts.


Michele Simionato
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,755
Messages
2,569,536
Members
45,020
Latest member
GenesisGai

Latest Threads

Top