List comprehension - NameError: name '_[1]' is not defined ?

Discussion in 'Python' started by mario ruggier, Jan 15, 2009.

  1. Hello,

    I would like to evaluate list comprehension expressions, from within
    which I'd like to call a function. For a first level it works fine but
    for second level it seems to lose the "_[1]" variable it uses
    internally to accumulate the results. Some sample code is:

    class GetItemEvaluator(object):
    def __init__(self):
    self.globals = globals() # some dict (never changes)
    self.globals["ts"] = self.ts
    self.globals["join"] = "".join
    self.locals = {} # changes on each evaluation
    def __getitem__(self, expr):
    return eval(expr, self.globals, self.locals)
    def ts(self, ts, name, value):
    self.locals[name] = value
    #print ts, name, value, "::::", self.locals, "::::", ts % self
    return ts % self

    gie = GetItemEvaluator()
    gie.locals["inner"] = ("a","b","c","d")
    print """
    pre %(join([ts("%s."%(j)+'%(k)s ', 'k', k) for j,k in enumerate
    (inner)]))s post
    """ % gie
    # OK, outputs: pre 0.a 1.b 2.c 3.d post

    gie = GetItemEvaluator()
    gie.locals["outer"] = [ ("m","n","o","p"), ("q","r","s","t")]
    print """
    pre %(join([ts(
    '''inner pre
    %(join([ts("%s.%s."%(i, j)+'%(k)s ', 'k', k) for j,k in enumerate
    (inner)]))s
    inner post''',
    "inner", inner) # END CALL outer ts()
    for i,inner in enumerate(outer)])
    )s post
    """ % gie

    The second 2-level comprehension gives:

    File "scratch/eval_test.py", line 8, in __getitem__
    return eval(expr, self.globals, self.locals)
    File "<string>", line 4, in <module>
    NameError: name '_[1]' is not defined

    If the print was to be enable, the last line printed out is:

    0.3.%(k)s k p :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r', 's',
    't')], 'i': 0, 'k': 'p', 'j': 3, '_[1]': ['0.0.m ', '0.1.n ', '0.2.o
    '], 'inner': ('m', 'n', 'o', 'p')} :::: 0.3.p

    i.e. it has correctly processed the first inner sequence, until the
    (last) "p" element. But on exit of the last inner ts() call, it seems
    to lose the '_[1]' on self.locals.

    Any ideas why?

    Note, i'd like that the first parameter to ts() is as independent as
    possible from teh context in expression context, a sort of independent
    mini-template. Thus, the i,j enumerate counters would normally not be
    subbed *within* the comprehension itself, but in a similar way to how
    k is evaluated, within the call to ts() -- I added them this way here
    to help follow easier what the execution trail is. Anyhow, within that
    mini-template, i'd like to embed other expressions for the % operator,
    and that may of course also be list comprehensions.

    Thanks!
    mario ruggier, Jan 15, 2009
    #1
    1. Advertising

  2. mario ruggier

    Guest

    mario ruggier, that's a hack that you can forget. Your code can't be
    read. Don't use list comps for that purpose. Write code that can be
    read.

    Bye,
    bearophile
    , Jan 15, 2009
    #2
    1. Advertising

  3. On Jan 15, 12:29 pm, mario ruggier <> wrote:
    > Any ideas why?
    >
    > Note, i'd like that the first parameter to ts() is as independent as
    > possible from the context in expression context, a sort of independent
    > mini-template. Thus, the i,j enumerate counters would normally not be
    > subbed *within* the comprehension itself, but in a similar way to how
    > k is evaluated, within the call to ts() -- I added them this way here
    > to help follow easier what the execution trail is. Anyhow, within that
    > mini-template, i'd like to embed other expressions for the % operator,
    > and that may of course also be list comprehensions.


    OK, here's the same sample code somewhat simplified
    and maybe be easier to follow what may be going on:


    class GetItemEvaluator(object):
    def __init__(self):
    self.globals = globals() # some dict (never changes)
    self.globals["ts"] = self.ts
    self.globals["join"] = " ".join
    self.locals = {} # changes on each evaluation
    def __getitem__(self, expr):
    return eval(expr, self.globals, self.locals)
    def ts(self, ts):
    print "ts:", ts, "::::", self.locals
    return ts % self

    # one level
    gie = GetItemEvaluator()
    gie.locals["inner"] = ("a","b","c","d")
    TS1 = """
    pre %(join([
    ts('%(j)s.%(k)s')
    for j,k in enumerate(inner)]))s post
    """
    OUT1 = TS1 % gie
    print "Output 1:", OUT1

    # two level
    gie = GetItemEvaluator()
    gie.locals["outer"] = [ ("m","n","o","p"), ("q","r","s","t")]
    TS2 = """
    leading %(join([
    ts(
    '''
    pre %(join([
    ts('%(i)s.%(j)s.%(k)s')
    for j,k in enumerate(inner)]))s post
    ''' # identical to TS1, except for additional '%(s)s.'
    )
    for i,inner in enumerate(outer)])
    )s trailing
    """
    OUT2 = TS2 % gie
    print "Output 2:", OUT2


    As the gie.locals dict is being automagically
    updated from within the list comprehension
    expression, I simplified the previous call to ts().
    Thus, executing this with the prints enabled as
    shown will produce the following output:


    $ python2.6 scratch/eval_test_4.py
    ts: %(j)s.%(k)s :::: {'_[1]': [], 'k': 'a', 'j': 0, 'inner': ('a',
    'b', 'c', 'd')}
    ts: %(j)s.%(k)s :::: {'_[1]': ['0.a'], 'k': 'b', 'j': 1, 'inner':
    ('a', 'b', 'c', 'd')}
    ts: %(j)s.%(k)s :::: {'_[1]': ['0.a', '1.b'], 'k': 'c', 'j': 2,
    'inner': ('a', 'b', 'c', 'd')}
    ts: %(j)s.%(k)s :::: {'_[1]': ['0.a', '1.b', '2.c'], 'k': 'd', 'j': 3,
    'inner': ('a', 'b', 'c', 'd')}
    Output 1:
    pre 0.a 1.b 2.c 3.d post

    ts:
    pre %(join([
    ts('%(i)s.%(j)s.%(k)s')
    for j,k in enumerate(inner)]))s post
    :::: {'_[1]': [], 'i': 0, 'outer': [('m', 'n', 'o', 'p'), ('q', 'r',
    's', 't')], 'inner': ('m', 'n', 'o', 'p')}
    ts: %(i)s.%(j)s.%(k)s :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r',
    's', 't')], 'i': 0, 'k': 'm', 'j': 0, '_[1]': [], 'inner': ('m', 'n',
    'o', 'p')}
    ts: %(i)s.%(j)s.%(k)s :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r',
    's', 't')], 'i': 0, 'k': 'n', 'j': 1, '_[1]': ['0.0.m'], 'inner':
    ('m', 'n', 'o', 'p')}
    ts: %(i)s.%(j)s.%(k)s :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r',
    's', 't')], 'i': 0, 'k': 'o', 'j': 2, '_[1]': ['0.0.m', '0.1.n'],
    'inner': ('m', 'n', 'o', 'p')}
    ts: %(i)s.%(j)s.%(k)s :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r',
    's', 't')], 'i': 0, 'k': 'p', 'j': 3, '_[1]': ['0.0.m', '0.1.n',
    '0.2.o'], 'inner': ('m', 'n', 'o', 'p')}
    Traceback (most recent call last):
    File "scratch/eval_test.py", line 40, in <module>
    OUT2 = TS2 % gie
    File "scratch/eval_test_4.py", line 8, in __getitem__
    return eval(expr, self.globals, self.locals)
    File "<string>", line 9, in <module>
    NameError: name '_[1]' is not defined


    Anyone can help clarify what may be going on?

    m.
    mario ruggier, Jan 15, 2009
    #3
  4. On Jan 15, 1:48 pm, wrote:
    > mario ruggier, that's a hack that you can forget. Your code can't be
    > read. Don't use list comps for that purpose. Write code that can be
    > read.


    Ya, agree with you whole-heartily, but then so are most
    optimizations ;-) It is just an idea I am exploring, and that code
    would be never be humanly written (that's why it seems more convoluted
    than necessary). I hope the simplified boiled down sample gets the
    intention out better... i'd still would like to understand why the '_
    [1]' variable is disappearing after first inner loop!

    > Bye,
    > bearophile
    mario ruggier, Jan 15, 2009
    #4
  5. mario ruggier

    Peter Otten Guest

    mario ruggier wrote:

    > Hello,
    >
    > I would like to evaluate list comprehension expressions, from within
    > which I'd like to call a function. For a first level it works fine but
    > for second level it seems to lose the "_[1]" variable it uses
    > internally to accumulate the results. Some sample code is:
    >
    > class GetItemEvaluator(object):
    > def __init__(self):
    > self.globals = globals() # some dict (never changes)
    > self.globals["ts"] = self.ts
    > self.globals["join"] = "".join
    > self.locals = {} # changes on each evaluation
    > def __getitem__(self, expr):
    > return eval(expr, self.globals, self.locals)
    > def ts(self, ts, name, value):
    > self.locals[name] = value
    > #print ts, name, value, "::::", self.locals, "::::", ts % self
    > return ts % self
    >
    > gie = GetItemEvaluator()
    > gie.locals["inner"] = ("a","b","c","d")
    > print """
    > pre %(join([ts("%s."%(j)+'%(k)s ', 'k', k) for j,k in enumerate
    > (inner)]))s post
    > """ % gie
    > # OK, outputs: pre 0.a 1.b 2.c 3.d post
    >
    > gie = GetItemEvaluator()
    > gie.locals["outer"] = [ ("m","n","o","p"), ("q","r","s","t")]
    > print """
    > pre %(join([ts(
    > '''inner pre
    > %(join([ts("%s.%s."%(i, j)+'%(k)s ', 'k', k) for j,k in enumerate
    > (inner)]))s
    > inner post''',
    > "inner", inner) # END CALL outer ts()
    > for i,inner in enumerate(outer)])
    > )s post
    > """ % gie
    >
    > The second 2-level comprehension gives:
    >
    > File "scratch/eval_test.py", line 8, in __getitem__
    > return eval(expr, self.globals, self.locals)
    > File "<string>", line 4, in <module>
    > NameError: name '_[1]' is not defined
    >
    > If the print was to be enable, the last line printed out is:
    >
    > 0.3.%(k)s k p :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r', 's',
    > 't')], 'i': 0, 'k': 'p', 'j': 3, '_[1]': ['0.0.m ', '0.1.n ', '0.2.o
    > '], 'inner': ('m', 'n', 'o', 'p')} :::: 0.3.p
    >
    > i.e. it has correctly processed the first inner sequence, until the
    > (last) "p" element. But on exit of the last inner ts() call, it seems
    > to lose the '_[1]' on self.locals.
    >
    > Any ideas why?
    >
    > Note, i'd like that the first parameter to ts() is as independent as
    > possible from teh context in expression context, a sort of independent
    > mini-template. Thus, the i,j enumerate counters would normally not be
    > subbed *within* the comprehension itself, but in a similar way to how
    > k is evaluated, within the call to ts() -- I added them this way here
    > to help follow easier what the execution trail is. Anyhow, within that
    > mini-template, i'd like to embed other expressions for the % operator,
    > and that may of course also be list comprehensions.


    I have no idea what you are trying to do. Please reread the Zen of Python ;)

    What happens is:

    List comprehensions delete the helper variable after completion:

    >>> def f(): [i for i in [1]]

    ....
    >>> dis.dis(f)

    1 0 BUILD_LIST 0
    3 DUP_TOP
    4 STORE_FAST 0 (_[1])
    7 LOAD_CONST 1 (1)
    10 BUILD_LIST 1
    13 GET_ITER
    >> 14 FOR_ITER 13 (to 30)

    17 STORE_FAST 1 (i)
    20 LOAD_FAST 0 (_[1])
    23 LOAD_FAST 1 (i)
    26 LIST_APPEND
    27 JUMP_ABSOLUTE 14
    >> 30 DELETE_FAST 0 (_[1])

    33 POP_TOP
    34 LOAD_CONST 0 (None)
    37 RETURN_VALUE

    If you manage to run two nested listcomps in the same namespace you get a
    name clash and the inner helper variable overwrites/deletes the outer:

    >>> def xeval(x): return eval(x, ns)

    ....
    >>> ns = dict(xeval=xeval)
    >>> xeval("[xeval('[k for k in ()]') for i in (1,)]")

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 1, in xeval
    File "<string>", line 1, in <module>
    NameError: name '_[1]' is not defined

    Peter
    Peter Otten, Jan 15, 2009
    #5
  6. On Jan 15, 2:02 pm, Peter Otten <> wrote:
    > mario ruggier wrote:
    > > Hello,

    >
    > > I would like to evaluate list comprehension expressions, from within
    > > which I'd like to call a function. For a first level it works fine but
    > > for second level it seems to lose the "_[1]" variable it uses
    > > internally to accumulate the results. Some sample code is:

    >
    > > class GetItemEvaluator(object):
    > >     def __init__(self):
    > >         self.globals = globals() # some dict (never changes)
    > >         self.globals["ts"] = self.ts
    > >         self.globals["join"] = "".join
    > >         self.locals = {} # changes on each evaluation
    > >     def __getitem__(self, expr):
    > >         return eval(expr, self.globals, self.locals)
    > >     def ts(self, ts, name, value):
    > >         self.locals[name] = value
    > >         #print ts, name, value, "::::", self.locals, "::::", ts % self
    > >         return ts % self

    >
    > > gie = GetItemEvaluator()
    > > gie.locals["inner"] = ("a","b","c","d")
    > > print """
    > > pre %(join([ts("%s."%(j)+'%(k)s ', 'k', k) for j,k in enumerate
    > > (inner)]))s post
    > > """ % gie
    > > # OK, outputs: pre 0.a 1.b 2.c 3.d  post

    >
    > > gie = GetItemEvaluator()
    > > gie.locals["outer"] = [ ("m","n","o","p"), ("q","r","s","t")]
    > > print """
    > > pre %(join([ts(
    > >   '''inner pre
    > >   %(join([ts("%s.%s."%(i, j)+'%(k)s ', 'k', k) for j,k in enumerate
    > > (inner)]))s
    > >    inner post''',
    > >   "inner", inner) # END CALL outer ts()
    > >   for i,inner in enumerate(outer)])
    > > )s post
    > > """ % gie

    >
    > > The second 2-level comprehension gives:

    >
    > >   File "scratch/eval_test.py", line 8, in __getitem__
    > >     return eval(expr, self.globals, self.locals)
    > >   File "<string>", line 4, in <module>
    > > NameError: name '_[1]' is not defined

    >
    > > If the print was to be enable, the last line printed out is:

    >
    > > 0.3.%(k)s  k p :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r', 's',
    > > 't')], 'i': 0, 'k': 'p', 'j': 3, '_[1]': ['0.0.m ', '0.1.n ', '0.2.o
    > > '], 'inner': ('m', 'n', 'o', 'p')} :::: 0.3.p

    >
    > > i.e. it has correctly processed the first inner sequence, until the
    > > (last) "p" element. But on exit of the last inner ts() call, it seems
    > > to lose the '_[1]' on self.locals.

    >
    > > Any ideas why?

    >
    > > Note, i'd like that the first parameter to ts() is as independent as
    > > possible from teh context in expression context, a sort of independent
    > > mini-template. Thus, the i,j enumerate counters would normally not be
    > > subbed *within* the comprehension itself, but in a similar way to how
    > > k is evaluated, within the call to ts() -- I added them this way here
    > > to help follow easier what the execution trail is. Anyhow, within that
    > > mini-template, i'd like to embed other expressions for the % operator,
    > > and that may of course also be list comprehensions.

    >
    > I have no idea what you are trying to do. Please reread the Zen of Python ;)
    >
    > What happens is:
    >
    > List comprehensions delete the helper variable after completion:
    >
    > >>> def f(): [i for i in [1]]

    > ...
    > >>> dis.dis(f)

    >
    >   1           0 BUILD_LIST               0
    >               3 DUP_TOP
    >               4 STORE_FAST               0 (_[1])
    >               7 LOAD_CONST               1 (1)
    >              10 BUILD_LIST               1
    >              13 GET_ITER
    >         >>   14 FOR_ITER                13 (to 30)
    >              17 STORE_FAST               1 (i)
    >              20 LOAD_FAST                0 (_[1])
    >              23 LOAD_FAST                1 (i)
    >              26 LIST_APPEND
    >              27 JUMP_ABSOLUTE           14
    >         >>   30 DELETE_FAST              0 (_[1])
    >              33 POP_TOP
    >              34 LOAD_CONST               0 (None)
    >              37 RETURN_VALUE
    >
    > If you manage to run two nested listcomps in the same namespace you get a
    > name clash and the inner helper variable overwrites/deletes the outer:
    >
    > >>> def xeval(x): return eval(x, ns)

    > ...
    > >>> ns = dict(xeval=xeval)
    > >>> xeval("[xeval('[k for k in ()]') for i in (1,)]")

    >
    > Traceback (most recent call last):
    >   File "<stdin>", line 1, in <module>
    >   File "<stdin>", line 1, in xeval
    >   File "<string>", line 1, in <module>
    > NameError: name '_[1]' is not defined
    >
    > Peter


    Ah, brilliant, thanks for the clarification!

    To verify if I understood you correctly, I have modified
    the ts() method above to:

    def ts(self, ts):
    _ns = self.locals
    self.locals = self.locals.copy()
    print "ts:", ts, "::::", self.locals
    try:
    return ts % self
    finally:
    self.locals = _ns

    And, it executes correctly, thus the 2nd output is:

    Output 2:
    leading
    pre 0.0.m 0.1.n 0.2.o 0.3.p post

    pre 1.0.q 1.1.r 1.2.s 1.3.t post
    trailing

    But, the need to do a copy() will likely kill any potential
    optimization gains... so, I will only be forced to rite more readable
    code ;-)

    Thanks!
    mario ruggier, Jan 15, 2009
    #6
  7. On Thu, 15 Jan 2009 03:29:59 -0800, mario ruggier wrote:

    > Hello,
    >
    > I would like to evaluate list comprehension expressions, from within
    > which I'd like to call a function. For a first level it works fine but
    > for second level it seems to lose the "_[1]" variable it uses internally
    > to accumulate the results. Some sample code is:
    >
    > class GetItemEvaluator(object):
    > def __init__(self):
    > self.globals = globals() # some dict (never changes)


    Would you like to put a small wager on that?

    >>> len(gie.globals)

    64
    >>> something_new = 0
    >>> len(gie.globals)

    65



    > self.globals["ts"] = self.ts
    > self.globals["join"] = "".join
    > self.locals = {} # changes on each evaluation
    > def __getitem__(self, expr):
    > return eval(expr, self.globals, self.locals)


    Can you say "Great Big Security Hole"?

    >>> gie = GetItemEvaluator()
    >>> gie['__import__("os").system("ls")']

    dicttest dumb.py rank.py sorting
    startup.py
    0


    http://cwe.mitre.org/data/definitions/95.html



    --
    Steven
    Steven D'Aprano, Jan 15, 2009
    #7
  8. On Jan 15, 4:06 pm, Steven D'Aprano <st...@REMOVE-THIS-
    cybersource.com.au> wrote:

    Hi Steve!

    > > class GetItemEvaluator(object):
    > >     def __init__(self):
    > >         self.globals = globals() # some dict (never changes)


    Ya, this is just a boiled down sample, and for simplicity I set to to
    the real globals(), so of course it will change when that changes...
    but in the application this is a distinct dict, that is entirely
    managed by the application, and it never changes as a result of an
    *evaluation*.

    > Would you like to put a small wager on that?
    >
    > >>> len(gie.globals)

    > 64
    > >>> something_new = 0
    > >>> len(gie.globals)

    >
    > 65



    > >         self.globals["ts"] = self.ts
    > >         self.globals["join"] = "".join
    > >         self.locals = {} # changes on each evaluation
    > >     def __getitem__(self, expr):
    > >         return eval(expr, self.globals, self.locals)

    >
    > Can you say "Great Big Security Hole"?


    With about the same difficulty as "Rabbit-Proof Fence" ;-)
    Again, it is just a boiled down sample, for communication purposes. As
    I mentioned in another thread, the real application behind all this is
    one of the *few* secure templating systems around. Some info on its
    security is at: http://evoque.gizmojo.org/usage/restricted/
    Tell you what, if you find a security hole there (via exposed template
    source on a Domain(restricted=True) setup) I'll offer you a nice
    dinner (including the beer!) somewhere, maybe at some py conference,
    but even remotely if that is not feasible... ;-) The upcoming 0.4
    release will run on 2.4 thru to 3.0 -- you can have some fun with that
    one (the current 0.3 runs on 2.5 and 2.6).

    > --
    > Steven


    Cheers, mario
    mario ruggier, Jan 15, 2009
    #8
  9. The listcomps exploration above was primarily an attempt
    (unsuccessful) to make this piece of code go a little faster:

    s = " text %(item)s text "
    acc = []
    for value in iterator:
    some_dict["item"] = value
    acc.append(s % evaluator)
    "".join(acc)

    The item=value pair is essentially a loop variable, and the evaluator
    (something like the gie instance above) uses it via the updated
    some_dict.

    Is there any way to express the above as a list comp or so? Any ideas
    how it might be made to go faster?

    m.
    mario ruggier, Jan 15, 2009
    #9
  10. mario ruggier

    Mark Wooding Guest

    mario ruggier <> writes:

    > Some info on its security is at:
    > http://evoque.gizmojo.org/usage/restricted/


    > Tell you what, if you find a security hole there (via exposed template
    > source on a Domain(restricted=True) setup) I'll offer you a nice
    > dinner (including the beer!) somewhere, maybe at some py conference,
    > but even remotely if that is not feasible... ;-) The upcoming 0.4
    > release will run on 2.4 thru to 3.0 -- you can have some fun with that
    > one (the current 0.3 runs on 2.5 and 2.6).


    I'm pretty sure I can break this on 3.0, because the f_restricted frame
    flag has gone. Here's how:

    >>> import template, domain
    >>> dom = domain.Domain('/tmp/mdw/', restricted = True, quoting = 'str')
    >>> t = template.Template(dom, 'evil', from_string = True, src =
    >>> "${inspect.func_globals['_'*2+'builtins'+'_'*2].open('/tmp/mdw/target').read()}")

    2009-01-15 20:30:29,177 ERROR [evoque] RuntimeError: restricted
    attribute: File "<string>", line 1, in <module>
    : EvalError(inspect.func_globals['_'*2+'builtins'+'_'*2].open('/tmp/mdw/target').read())
    u'[RuntimeError: restricted attribute: File "<string>", line 1, in
    <module>\n:
    EvalError(inspect.func_globals[\'_\'*2+\'builtins\'+\'_\'*2].open(\'/tmp/mdw/target\').read())]'

    which means that it's depending on the func_globals attribute being
    rejected by the interpreter -- which it won't be because 3.0 doesn't
    have restricted evaluation any more.

    Python is very leaky. I don't think trying to restrict Python execution
    is a game that's worth playing.

    -- [mdw]
    Mark Wooding, Jan 15, 2009
    #10
  11. mario ruggier

    ajaksu Guest

    On Jan 15, 1:56 pm, mario ruggier <> wrote:
    > As
    > I mentioned in another thread, the real application behind all this is
    > one of the *few* secure templating systems around. Some info on its
    > security is at:http://evoque.gizmojo.org/usage/restricted/
    > Tell you what, if you find a security hole there (via exposed template
    > source on a Domain(restricted=True) setup) I'll offer you a nice
    > dinner (including the beer!) somewhere, maybe at some py conference,
    > but even remotely if that is not feasible... ;-)


    If you could provide a bare-bones instance of your evaluator to test
    against, without using the whole evoque (I get DUMMY MODE ON from
    'self.template.collection.domain.globals'), it'd be more interesting
    to try :)
    ajaksu, Jan 15, 2009
    #11
  12. On Jan 15, 9:36 pm, Mark Wooding <> wrote:
    > mario ruggier <> writes:
    > > Some info on its security is at:
    > >http://evoque.gizmojo.org/usage/restricted/
    > > Tell you what, if you find a security hole there (via exposed template
    > > source on a Domain(restricted=True) setup) I'll offer you a nice
    > > dinner (including the beer!) somewhere, maybe at some py conference,
    > > but even remotely if that is not feasible... ;-) The upcoming 0.4
    > > release will run on 2.4 thru to 3.0 -- you can have some fun with that
    > > one (the current 0.3 runs on 2.5 and 2.6).

    >
    > I'm pretty sure I can break this on 3.0, because the f_restricted frame
    > flag has gone.  Here's how:
    >
    > >>> import template, domain
    > >>> dom = domain.Domain('/tmp/mdw/', restricted = True, quoting = 'str')
    > >>> t = template.Template(dom, 'evil', from_string = True, src =
    > >>> "${inspect.func_globals['_'*2+'builtins'+'_'*2].open('/tmp/mdw/target').read()}")

    >
    > 2009-01-15 20:30:29,177 ERROR [evoque] RuntimeError: restricted
    > attribute:   File "<string>", line 1, in <module>
    > : EvalError(inspect.func_globals['_'*2+'builtins'+'_'*2].open('/tmp/mdw/target').read())
    > u'[RuntimeError: restricted attribute:   File "<string>", line 1, in
    > <module>\n:
    > EvalError(inspect.func_globals[\'_\'*2+\'builtins\'+\'_\'*2].open(\'/tmp/mdw/target\').read())]'
    >
    > which means that it's depending on the func_globals attribute being
    > rejected by the interpreter -- which it won't be because 3.0 doesn't
    > have restricted evaluation any more.



    $ touch /tmp/mdw.test
    mr:evoque mario$ python3.0
    Python 3.0 (r30:67503, Dec 8 2008, 18:45:31)
    [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from evoque import domain, template
    >>> d = domain.Domain("/", restricted=True, quoting="str")
    >>> t = template.Template(d, "mdw1", from_string=True, src="${inspect..func_globals['_'*2+'builtins'+'_'*2].open('/tmp/mdw.test').read()}")
    >>> t.evoque()

    2009-01-15 22:26:18,704 ERROR [evoque] AttributeError: 'function'
    object has no attribute 'func_globals': File "<string>", line 1, in
    <module>
    : EvalError(inspect.func_globals['_'*2+'builtins'+'_'*2].open('/tmp/
    mdw.test').read())
    '[AttributeError: \'function\' object has no attribute \'func_globals
    \': File "<string>", line 1, in <module>\n: EvalError
    (inspect.func_globals[\'_\'*2+\'builtins\'+\'_\'*2].open(\'/tmp/
    mdw.test\').read())]'

    But even if inspect did have the func_globals attribute, the "open"
    builtin will not be found on __builtins__ (that is cleaned out when
    restricted=True).

    But, I guess it is necessary to keep an eye on what is available/
    allowed by the different python versions, and adjust as needed,
    probably to the lowest common denominator. In addition to what is
    mentioned on the doc on evoque's resticted mode at the above URL, do
    you have specific suggestions what may be a good idea to also block
    out?

    > Python is very leaky.  I don't think trying to restrict Python execution
    > is a game that's worth playing.


    It may not be worth risking your life on it, but it is certainly worth
    playing ;-)

    Thanks.. with you permission I am adding your evil expression to the
    restricted tests?

    Cheers, mario

    > -- [mdw]
    mario ruggier, Jan 15, 2009
    #12
  13. On Jan 15, 10:35 pm, ajaksu <> wrote:
    > On Jan 15, 1:56 pm, mario ruggier <> wrote:
    >
    > > As
    > > I mentioned in another thread, the real application behind all this is
    > > one of the *few* secure templating systems around. Some info on its
    > > security is at:http://evoque.gizmojo.org/usage/restricted/
    > > Tell you what, if you find a security hole there (via exposed template
    > > source on a Domain(restricted=True) setup) I'll offer you a nice
    > > dinner (including the beer!) somewhere, maybe at some py conference,
    > > but even remotely if that is not feasible... ;-)

    >
    > If you could provide a bare-bones instance of your evaluator to test
    > against, without using the whole evoque (I get DUMMY MODE ON from
    > 'self.template.collection.domain.globals'), it'd be more interesting
    > to try :)


    OK! Here's a small script to make it easier...
    Just accumulate any expression you can dream of,
    and pass it to get_expr_template() to get the template,
    and on that then call evoque()... i guess you'd have to
    test with 0.3, but 0.4 (also runs on py3) is just
    around the corner....

    Let it rip... the beer'd be on me ;-!


    # evoque_restricted_test.py

    from os.path import abspath, join, dirname
    from evoque import domain, template

    import logging
    # uncomment to hide the plentiful ERROR logs:
    #logging_level = logging.CRITICAL

    # set the base for for the defualt collection
    DEFAULT_DIR = abspath("/")

    # 3 -> renders, 4 -> raises any evaluation errors,
    # see: http://evoque.gizmojo.org/usage/errors/
    ERRORS=2

    # a restricted domain instance
    d = domain.Domain(DEFAULT_DIR, restricted=True, errors=ERRORS,
    quoting='str')
    count = 0

    # utility to easily init a template from any expression
    def get_expr_template(expr):
    global count
    count += 1
    name = "test%s"%(count)
    src = "${%s}" % (expr)
    d.set_template(name, src=src, from_string=True)
    return d.get_template(name)

    # some test expressions
    exprs = [
    "open('test.txt', 'w')",
    "getattr(int, '_' + '_abs_' + '_')",
    "().__class__.mro()[1].__subclasses__()",
    "inspect.func_globals['_'*2+'builtins'+'_'*2]",
    ]

    # execute
    for expr in exprs:
    print
    print expr
    print get_expr_template(expr).evoque()
    mario ruggier, Jan 15, 2009
    #13
  14. mario ruggier

    Terry Reedy Guest

    Peter Otten wrote:

    > List comprehensions delete the helper variable after completion:


    I do not believe they did in 2.4. Not sure of 2.5. There is certainly
    a very different implementation in 3.0 and, I think, 2.6. OP
    neglected to mention Python version he tested on. Code meant to run on
    2.4 to 3.0 cannot depend on subtle listcomp details.

    >>>> def f(): [i for i in [1]]

    > ...
    >>>> dis.dis(f)

    > 1 0 BUILD_LIST 0
    > 3 DUP_TOP
    > 4 STORE_FAST 0 (_[1])
    > 7 LOAD_CONST 1 (1)
    > 10 BUILD_LIST 1
    > 13 GET_ITER
    > >> 14 FOR_ITER 13 (to 30)

    > 17 STORE_FAST 1 (i)
    > 20 LOAD_FAST 0 (_[1])
    > 23 LOAD_FAST 1 (i)
    > 26 LIST_APPEND
    > 27 JUMP_ABSOLUTE 14
    > >> 30 DELETE_FAST 0 (_[1])

    > 33 POP_TOP
    > 34 LOAD_CONST 0 (None)
    > 37 RETURN_VALUE
    >


    In 3.0
    >>> def f(): [i for i in [1]]


    >>> import dis
    >>> dis.dis(f)

    1 0 LOAD_CONST 1 (<code object <listcomp> at
    0x01349BF0, file "<pyshell#12>", line 1>)
    3 MAKE_FUNCTION 0
    6 LOAD_CONST 2 (1)
    9 BUILD_LIST 1
    12 GET_ITER
    13 CALL_FUNCTION 1
    16 POP_TOP
    17 LOAD_CONST 0 (None)
    20 RETURN_VALUE

    Running OP code in 3.0 with print ()s added gives

    pre 0.a 1.b 2.c 3.d post

    Traceback (most recent call last):
    File "C:\Programs\Python30\misc\temp7.py", line 32, in <module>
    """ % gie)
    File "C:\Programs\Python30\misc\temp7.py", line 8, in __getitem__
    return eval(expr, self.globals, self.locals)
    File "<string>", line 7, in <module>
    File "<string>", line 7, in <listcomp>
    File "C:\Programs\Python30\misc\temp7.py", line 12, in ts
    return ts % self
    File "C:\Programs\Python30\misc\temp7.py", line 8, in __getitem__
    return eval(expr, self.globals, self.locals)
    File "<string>", line 2, in <module>
    File "<string>", line 1, in <listcomp>
    NameError: global name 'i' is not defined

    > If you manage to run two nested listcomps in the same namespace you get a
    > name clash and the inner helper variable overwrites/deletes the outer:
    >
    >>>> def xeval(x): return eval(x, ns)

    > ...
    >>>> ns = dict(xeval=xeval)
    >>>> xeval("[xeval('[k for k in ()]') for i in (1,)]")

    > Traceback (most recent call last):
    > File "<stdin>", line 1, in <module>
    > File "<stdin>", line 1, in xeval
    > File "<string>", line 1, in <module>
    > NameError: name '_[1]' is not defined


    Which Python? 3.0 prints "[[]]"! But I think the nested listcomp *is*
    in a separate namespace here. I will leave it to you or OP to disect
    how his and your code essentially differ from 3.0 (and maybe 2.6)
    implementation's viewpoint.

    Terry Jan Reedy
    Terry Reedy, Jan 15, 2009
    #14
  15. mario ruggier

    Terry Reedy Guest

    mario ruggier wrote:
    > On Jan 15, 4:06 pm, Steven D'Aprano <st...@REMOVE-THIS-
    > cybersource.com.au> wrote:
    >
    > Hi Steve!
    >
    >>> class GetItemEvaluator(object):
    >>> def __init__(self):
    >>> self.globals = globals() # some dict (never changes)

    >
    > Ya, this is just a boiled down sample, and for simplicity I set to to
    > the real globals(), so of course it will change when that changes...
    > but in the application this is a distinct dict, that is entirely
    > managed by the application, and it never changes as a result of an
    > *evaluation*.


    It would have been less confusing if you had written

    self.globals = {} # a constant dict
    or even
    self.constants = {} # empty here only for simplicity

    This might also make the 3.0 error message clearer (see other post).

    tjr
    Terry Reedy, Jan 15, 2009
    #15
  16. On Thu, 15 Jan 2009 07:56:02 -0800, mario ruggier wrote:

    > On Jan 15, 4:06 pm, Steven D'Aprano <st...@REMOVE-THIS-
    > cybersource.com.au> wrote:
    >
    > Hi Steve!
    >
    >> > class GetItemEvaluator(object):
    >> >     def __init__(self):
    >> >         self.globals = globals() # some dict (never changes)

    >
    > Ya, this is just a boiled down sample, and for simplicity I set to to
    > the real globals(),



    You should make that more clear when posting, in the code snippet as well
    as the descriptive text.

    And if you *did* make it clear, then *I* should read your post more
    carefully.



    Regards,


    --
    Steven
    Steven D'Aprano, Jan 15, 2009
    #16
  17. On Jan 15, 11:35 pm, Terry Reedy <> wrote:
    > Peter Otten wrote:
    > > List comprehensions delete the helper variable after completion:

    >
    > I do not believe they did in 2.4.  Not sure of 2.5.  There is certainly
    >   a very different implementation in 3.0 and, I think, 2.6.  OP
    > neglected to mention Python version he tested on. Code meant to run on
    > 2.4 to 3.0 cannot depend on subtle listcomp details.
    >
    >
    >
    > >>>> def f(): [i for i in [1]]

    > > ...
    > >>>> dis.dis(f)

    > >   1           0 BUILD_LIST               0
    > >               3 DUP_TOP
    > >               4 STORE_FAST               0 (_[1])
    > >               7 LOAD_CONST               1 (1)
    > >              10 BUILD_LIST               1
    > >              13 GET_ITER
    > >         >>   14 FOR_ITER                13 (to 30)
    > >              17 STORE_FAST               1 (i)
    > >              20 LOAD_FAST                0 (_[1])
    > >              23 LOAD_FAST                1 (i)
    > >              26 LIST_APPEND
    > >              27 JUMP_ABSOLUTE           14
    > >         >>   30 DELETE_FAST              0 (_[1])
    > >              33 POP_TOP
    > >              34 LOAD_CONST               0 (None)
    > >              37 RETURN_VALUE

    >
    > In 3.0
    >  >>> def f(): [i for i in [1]]
    >
    >  >>> import dis
    >  >>> dis.dis(f)
    >    1           0 LOAD_CONST               1 (<code object <listcomp> at
    > 0x01349BF0, file "<pyshell#12>", line 1>)
    >                3 MAKE_FUNCTION            0
    >                6 LOAD_CONST               2 (1)
    >                9 BUILD_LIST               1
    >               12 GET_ITER
    >               13 CALL_FUNCTION            1
    >               16 POP_TOP
    >               17 LOAD_CONST               0 (None)
    >               20 RETURN_VALUE
    >
    > Running OP code in 3.0 with print ()s added gives
    >
    > pre 0.a 1.b 2.c 3.d  post
    >
    > Traceback (most recent call last):
    >    File "C:\Programs\Python30\misc\temp7.py", line 32, in <module>
    >      """ % gie)
    >    File "C:\Programs\Python30\misc\temp7.py", line 8, in __getitem__
    >      return eval(expr, self.globals, self.locals)
    >    File "<string>", line 7, in <module>
    >    File "<string>", line 7, in <listcomp>
    >    File "C:\Programs\Python30\misc\temp7.py", line 12, in ts
    >      return ts % self
    >    File "C:\Programs\Python30\misc\temp7.py", line 8, in __getitem__
    >      return eval(expr, self.globals, self.locals)
    >    File "<string>", line 2, in <module>
    >    File "<string>", line 1, in <listcomp>
    > NameError: global name 'i' is not defined
    >
    > > If you manage to run two nested listcomps in the same namespace you get a
    > > name clash and the inner helper variable overwrites/deletes the outer:

    >
    > >>>> def xeval(x): return eval(x, ns)

    > > ...
    > >>>> ns = dict(xeval=xeval)
    > >>>> xeval("[xeval('[k for k in ()]') for i in (1,)]")

    > > Traceback (most recent call last):
    > >   File "<stdin>", line 1, in <module>
    > >   File "<stdin>", line 1, in xeval
    > >   File "<string>", line 1, in <module>
    > > NameError: name '_[1]' is not defined

    >
    > Which Python?  3.0 prints "[[]]"! But I think the nested listcomp *is*
    > in a separate namespace here.  I will leave it to you or OP to disect
    > how his and your code essentially differ from 3.0 (and maybe 2.6)
    > implementation's viewpoint.


    I was testing on 2.6, but running it thru 2.4 and 2.5 it seems
    behaviour is the same there. For 3.0 it does change... and there seems
    not to be the "_[1]" key defined, and, what's more, it gives a:
    NameError: name 'j' is not defined.

    In any case, that was an exploration to get a feeling for how the
    listcomps behave (performance) if evaluated directly as opposed to
    doing the equivalent from within a function. It turned out to be
    slower, so I moved on... but, should it have been faster, then
    differences between how the different python versions handle list
    comps internally would have been a next issue to address.

    I think the globals dict is not touched by eval'ing a list comp... it
    is any not "constant" as such, just that it is not affected by
    evaluations (unless the python application decides to affect it in
    some way or another). But, evaluating a template by definition does
    not change the globals dict.

    > Terry Jan Reedy
    mario ruggier, Jan 15, 2009
    #17
  18. mario ruggier

    ajaksu Guest

    On Jan 15, 8:21 pm, mario ruggier <> wrote:
    > OK! Here's a small script to make it easier...


    Thanks! I think I found a quick way around the restrictions (correct
    me if I borked it), but I think you can block this example by
    resetting your globals/builtins:

    exprs = [
    '(x for x in range(1)).gi_frame.f_globals.clear()',
    'open("where_is_ma_beer.txt", "w").write("Thanks for the fun ")'
    ]

    Regards,
    Daniel
    ajaksu, Jan 16, 2009
    #18
  19. mario ruggier

    Mark Wooding Guest

    mario ruggier <> writes:

    > 2009-01-15 22:26:18,704 ERROR [evoque] AttributeError: 'function'
    > object has no attribute 'func_globals': File "<string>", line 1, in
    > <module>


    Damn. So that doesn't work. :-(

    > But even if inspect did have the func_globals attribute, the "open"
    > builtin will not be found on __builtins__ (that is cleaned out when
    > restricted=True).


    Irrelevant. I wasn't trying to get at my __builtins__ but the one
    attached to a function I was passed in, which has a different
    environment.

    You define a function (a method, actually, but it matters little). The
    function's globals dictionary is attached as an attribute. You didn't
    do anything special here, so the globals have the standard __builtins__
    binding. It contains open.

    You now run my code in a funny environment with a stripped-down
    __builtins__. But that doesn't matter, because my environment contains
    your function. And that function has your __builtins__ hanging off the
    side of it.

    .... but I can't actually get there because of f_restricted. You don't
    mention the fact that Python magically limits access to these attributes
    if __builtins__ doesn't match the usual one, so I think you got lucky.

    -- [mdw]
    Mark Wooding, Jan 16, 2009
    #19
  20. On Jan 16, 2:30 am, ajaksu <> wrote:
    > On Jan 15, 8:21 pm, mario ruggier <> wrote:
    >
    > > OK! Here's a small script to make it easier...

    >
    > Thanks! I think I found a quick way around the restrictions (correct
    > me if I borked it), but I think you can block this example by
    > resetting your globals/builtins:
    >
    > exprs = [
    >     '(x for x in range(1)).gi_frame.f_globals.clear()',
    >     'open("where_is_ma_beer.txt", "w").write("Thanks for the fun  ")'
    > ]


    Cool, the beer that is ;) Under 2.6... why does python allow the
    f_globals lookup in this case, but for the previous example for
    func_globals it does not?

    If you look at the top of the file test/test_restricted.py, there is:

    # Attempt at accessing these attrs under restricted execution on an
    object
    # that has them should raise a RuntimeError
    RESTRICTED_ATTRS = [
    'im_class', 'im_func', 'im_self', 'func_code', 'func_defaults',
    'func_globals', #'func_name',
    #'tb_frame', 'tb_next',
    #'f_back', 'f_builtins', 'f_code', 'f_exc_traceback',
    'f_exc_type',
    #'f_exc_value', 'f_globals', 'f_locals'
    ]

    I have not yet finished working this list off to ensure that any
    lookup of these attrs wherever they occur will be refused, but I guess
    that would block this kind of lookup out. I should also block any
    attempt to access any "gi_*" attribute... Laboriously doing all these
    checks on each expr eval will be very performance heavy, so I hope to
    be able to limit access to all these more efficiently. Suggestions?

    Cheers, Mario

    > Regards,
    > Daniel
    mario ruggier, Jan 16, 2009
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. jolly
    Replies:
    2
    Views:
    572
    jolly
    Dec 19, 2007
  2. Replies:
    3
    Views:
    596
    Dan Bishop
    Mar 22, 2008
  3. Lalit
    Replies:
    1
    Views:
    744
    Diez B. Roggisch
    Apr 24, 2008
  4. Mathieu Prevot
    Replies:
    3
    Views:
    376
    Mathieu Prevot
    May 24, 2008
  5. Vedran Furac(
    Replies:
    4
    Views:
    328
    Marc 'BlackJack' Rintsch
    Dec 19, 2008
Loading...

Share This Page