List comprehension - NameError: name '_[1]' is not defined ?

[Peter Otten]

I do not believe they did in 2.4. Not sure of 2.5.

As Mario said, 2.4, 2.5, and 2.6 all show the same behaviour.

There is certainly
a very different implementation in 3.0 and, I think, 2.6. OP
neglected to mention Python version he tested on. Code meant to run on
2.4 to 3.0 cannot depend on subtle listcomp details.

3.0 behaves different. Like generator expressions listcomps no longer leak
the loop variable, and this is implemented by having each listcomp execute
as a nested function:

In 3.0

def f(): [i for i in [1]]
import dis
dis.dis(f)

1 0 LOAD_CONST 1 (<code object <listcomp> at
0x01349BF0, file "<pyshell#12>", line 1>)
3 MAKE_FUNCTION 0
6 LOAD_CONST 2 (1)
9 BUILD_LIST 1
12 GET_ITER
13 CALL_FUNCTION 1
16 POP_TOP
17 LOAD_CONST 0 (None)
20 RETURN_VALUE

This is more robust (at least I can't think of a way to break it like the
2.x approach) but probably slower due to the function call overhead. The
helper variable is still there, but the possibility of a clash with another
helper is gone (disclaimer: I didn't check this in the Python source) so
instead of

# 2.5 and 2.6 (2.4 has the names in a different order)
.... [[i for i in ()] for k in ()]
....('_[1]', 'k', '_[2]', 'i')

we get

# 3.0
.... [[i for i in ()] for k in ()]
....()

The variables are gone from f's scope, as 3.x listcomps no longer leak their
loop variables.

f.__code__.co_consts

outer = f.__code__.co_consts[1]
outer.co_varnames

('.0', '_[1]', 'k')

Again the inner listcomp is separated from the outer.

outer.co_consts

inner = outer.co_consts[0]
inner.co_varnames

('.0', '_[1]', 'i')


Peter

 

[ajaksu]

Laboriously doing all these
checks on each expr eval will be very performance heavy, so I hope to
be able to limit access to all these more efficiently. Suggestions?

None regarding the general issue, a try:except to handle this one:

'(x for x in ()).throw("bork")'

 

[mario ruggier]

None regarding the general issue, a try:except to handle this one:

'(x for x in ()).throw("bork")'

What is the potential security risk with this one?

To handle this and situations like the ones pointed out above on this
thread, I will probably affect the following change to the
evoque.evaluator.RestrictedEvaluator class, and that is to replace the
'if name.find("__")!=-1:' with an re.search... where the re is defined
as:

restricted = re.compile(r"|\.".join([
"__", "func_", "f_", "im_", "tb_", "gi_", "throw"]))

and the test becomes simply:

if restricted.search(name):

All the above attempts will be blocked this way. Any other disallow-
sub-strings to add to the list above?

And thanks a lot Daniel, need to find a way to get somebeer over to
ya... ;-)

mario

 

[Paul Rubin]

All the above attempts will be blocked this way. Any other disallow-
sub-strings to add to the list above?

I think what you are trying to do is fundamentally hopeless. You
might look at web.py (http://webpy.org) for another approach, that
puts a complete interpreter for a Python-like language into the
template engine.

 

[ajaksu]

What is the potential security risk with this one?

I don't see a concrete issue, just found it tempting... raising hand-
crafted objects

All the above attempts will be blocked this way. Any other disallow-
sub-strings to add to the list above?

None that I know of, but I suggest testing with dir, globals, locals
and '__' enabled (which I haven't done yet), as spotting possible
flaws should be easier. If you can get BOM+encoded garbage tested (see
http://tinyurl.com/72d98y ), it might be worth it too.

This one fails in lots of interesting ways when you juggle keyword-
args around:
exprs = [
'evoque("hmm", filters=[unicode.upper ] ,src="/etc/python2.5/
site.py")',
]

And thanks a lot Daniel, need to find a way to get somebeer over to
ya... ;-)

You're welcome! Don't worry about the beer, I'd only consider a real
promise if it involved chocolate

Regards,
Daniel

 

[mario ruggier]

I don't see a concrete issue, just found it tempting... raising hand-
crafted objects

OK, I can think of no good reson why anyone would want to do that from
within a temlate, so I'd be fine with blocking out any attribute whose
name starts with "throw" to block this out.

None that I know of, but I suggest testing with dir, globals, locals
and '__' enabled (which I haven't done yet), as spotting possible
flaws should be easier. If you can get BOM+encoded garbage tested (seehttp://tinyurl.com/72d98y), it might be worth it too.

The BOM stuff is interesting... from that discussion, I think it would
be also a good idea to blacklist "object" out of the restricted
builtins. I played with this, and prepared a file template as well as
a little script to run it... see below.

To tweak any disallwoed builtins back into the restricted namespace
for testing, you can just do something like:

d.set_on_globals("dir", dir)

for each name you'd like to add, when setting up the domain (see
script below).

To re-enable "__" lookups, you'd need to tweak the regexp above, in
the RestrictedEvaluator class.

This one fails in lots of interesting ways when you juggle keyword-
args around:
exprs = [
    'evoque("hmm", filters=[unicode.upper ] ,src="/etc/python2.5/
site.py")',
]

Not sure what you mean... it just renders that source code file
uppercased (if it finds it as per the domain setup) ?!?


Here's (a) a mini testing py2-py3 script, similar to previous one
above but to read a template from a file (there may be additional
tricks possible that way), and (b) a sample companion test template.

evoque_restricted_file_test.py
----
# in lieu of print, py2/py3
import sys
def pr(*args):
sys.stdout.write(" ".join([str(arg) for arg in args])+'\n')
#
from os.path import abspath, join, dirname
from evoque import domain, template

# set the base for for the defualt collection
DEFAULT_DIR = abspath((dirname(__file__)))

# a restricted domain instance
d = domain.Domain(DEFAULT_DIR, restricted=True, errors=3,
quoting='str')
# errors: 3 -> renders, 4 -> raises any evaluation errors,
# see: http://evoque.gizmojo.org/usage/errors/

# Tweak domain.globals to add specific callables for testing:
d.set_on_globals("dir", dir)
d.set_on_globals("gobals", globals)
d.set_on_globals("locals", locals)

pr("domain", d.default_collection.dir,
d.restricted and "RESTRICTED" or "*** unrestricted ***")

t = d.get_template(restricted_exprs.txt)
pr(t.evoque())
----

restricted_exprs.txt
----
#[
BOM + encoded trickery

Note:
when evaluated in python interpreter:+AG8AYgBqAGUAYwB0AC4AXwBfAHMAdQBiAGMAbABhAHMAcwBlAHMAXwBf-")
<built-in method __subclasses__ of type object at 0x1f1860>

but when specified within a template here as:
${# coding: utf7\n
+AG8AYgBqAGUAYwB0AC4AXwBfAHMAdQBiAGMAbABhAHMAcwBlAHMAXwBf-}
gives **pre-evaluation**:
SyntaxError: unknown encoding: utf7
]#
${"# coding: utf7\n
+AG8AYgBqAGUAYwB0AC4AXwBfAHMAdQBiAGMAbABhAHMAcwBlAHMAXwBf-"},


#[
Attempt to subversively build string expressions
]#
Explicitly written target expression: ().__class__.mro()
[1].__subclasses__()
evaluates: ${().__class__.mro()[1].__subclasses__()}

Subversive variation: "()."+"_"*2+"class"+"_"*2+".mro()
[1]."+"_"*2+"subclasses"+"_"*2+"()"
evaluates (to just the str!): ${"()."+"_"*2+"class"+"_"*2+".mro()
[1]."+"_"*2+"subclasses"+"_"*2+"()"}

Attempt to "set" same subsersively built expr to a loop variable
and then "evaluate" that variable:
$for{
expr in [
str("()."+"_"*2+"class"+"_"*2+".mro()
[1]."+"_"*2+"subclasses"+"_"*2+"()")
] }
evaluates (to just the str!): ${expr}
attempt eval(...): ${eval(expr)}
$rof
(Note: evoque does not explicitly allow arbitrary setting of
variables, except within for loops.)

 

[mario ruggier]

Just to add that a further potential subversion possibility could have
been to build the expr in some way from within a template, and then
dynamically setting that string as the source of a new template with
from_string=True. This is precisely the reason why **from within a
template** evoque has never supported the from_string parameter i.e.
you *cannot* "load/create" a template from a string source from within
another template -- you may *only* do that from within the python app.
To illustrate, an extension of the previous example template could
thus become:

$for{expr in [
str("()."+"_"*2+"class"+"_"*2+".mro()
[1]."+"_"*2+"subclasses"+"_"*2+"()")
]}
${% evoque("test", src="$${"+expr+"}", from_string=True) %}
$rof

That would fail, as it would simply take the value of src i.e. "$
{().__class__.mro()[1].__subclasses__()}" to mean the sub-path to the
template file within the collection (from_string would be simply
interpreted as an evaluation data parameter). See:
http://evoque.gizmojo.org/directives/evoque/

 

[mario ruggier]

I think what you are trying to do is fundamentally hopeless.  You
might look at web.py (http://webpy.org) for another approach, that
puts a complete interpreter for a Python-like language into the
template engine.

Well, that is a bold statement... but maybe it is explained by what
you refer to, so I did a cursory look. But I miss to notice any
reference of an embedded "python-like language -- is there some sort
of overview of how web.py implements this e.g. something like the
equivalent of the doc describing how evoque implements it's sandbox:
http://evoque.gizmojo.org/usage/restricted/

I get the feeling you may also be ignoring contextual factors...
restricting the full python interpreter is not what we are talking
about here, but templating systems (such as web.py?) that just allow
embedding of any and all python code will require exactly that. And
*that* may well seem fundamentally hopeless.

Evoque chooses to allow only expressions, and those under a *managed*
context. To make that secure is a whole different (smaller) task.