Unexpected python exception

Discussion in 'Python' started by Richard Purdie, Nov 11, 2009.

  1. I've been having problems with an unexpected exception from python which
    I can summarise with the following testcase:

    def A():
    import __builtin__
    import os

    __builtin__.os = os

    def B():
    os.stat("/")
    import os

    A()
    B()

    which results in:

    Traceback (most recent call last):
    File "./test.py", line 12, in <module>
    B()
    File "./test.py", line 8, in B
    os.stat("/")
    UnboundLocalError: local variable 'os' referenced before assignment

    If I remove the "import os" from B(), it works as expected.

    >From what I've seen, its very unusual to have something operate

    "backwards" in scope in python. Can anyone explain why this happens?

    Cheers,

    Richard
    Richard Purdie, Nov 11, 2009
    #1
    1. Advertising

  2. Richard Purdie schrieb:
    > I've been having problems with an unexpected exception from python which
    > I can summarise with the following testcase:
    >
    > def A():
    > import __builtin__
    > import os
    >
    > __builtin__.os = os
    >
    > def B():
    > os.stat("/")
    > import os
    >
    > A()
    > B()
    >
    > which results in:
    >
    > Traceback (most recent call last):
    > File "./test.py", line 12, in <module>
    > B()
    > File "./test.py", line 8, in B
    > os.stat("/")
    > UnboundLocalError: local variable 'os' referenced before assignment
    >
    > If I remove the "import os" from B(), it works as expected.
    >
    >>From what I've seen, its very unusual to have something operate

    > "backwards" in scope in python. Can anyone explain why this happens?


    As the import-statement in a function/method-scope doesn't leak the
    imported names into the module scope, python treats them as locals.
    Which makes your code equivalent to


    x = 1000

    def foo():
    print x
    x = 10

    Throws the same error. The remedy is to inform python that a specific
    name belongs to global scope, using the "global"-statement.

    def foo():
    global x
    print x
    x = 10


    Beware though that then of course *assigning* to x is on global level.
    This shouldn't be of any difference in your case though, because of the
    import-only-once-mechanics of python.

    Diez
    Diez B. Roggisch, Nov 11, 2009
    #2
    1. Advertising

  3. Hi,
    unfortunatley I cannot reproduce your error. Which Python Version do you
    use?

    The expected case in this scenario is that the exception is thrown, as
    you import os in A() where it is stored in the local namespace of the
    function.

    I tested it with Python 2.4, 2.5 and 2.6 and in both cases an exception
    is thrown.

    Best,
    Andreas
    Andreas Löscher, Nov 11, 2009
    #3
  4. Richard Purdie

    Chris Rebert Guest

    On Wed, Nov 11, 2009 at 8:49 AM, Eduardo Lenz <> wrote:
    > Em Qua 11 Nov 2009, às 03:21:55, Diez B. Roggisch escreveu:
    >> Richard Purdie schrieb:
    >> > I've been having problems with an unexpected exception from python which
    >> > I can summarise with the following testcase:
    >> >
    >> > def A():
    >> >     import __builtin__
    >> >     import os
    >> >
    >> >     __builtin__.os = os
    >> >
    >> > def B():
    >> >     os.stat("/")
    >> >     import os
    >> >
    >> > A()
    >> > B()
    >> >
    >> > which results in:
    >> >
    >> > Traceback (most recent call last):
    >> >   File "./test.py", line 12, in <module>
    >> >     B()
    >> >   File "./test.py", line 8, in B
    >> >     os.stat("/")
    >> > UnboundLocalError: local variable 'os' referenced before assignment
    >> >
    >> > If I remove the "import os" from B(), it works as expected.
    >> >
    >> >>From what I've seen, its very unusual to have something operate
    >> >
    >> > "backwards" in scope in python. Can anyone explain why this happens?

    >>
    >> As the import-statement in a function/method-scope doesn't leak the
    >> imported names into the module scope, python treats them as locals.
    >> Which makes your code equivalent to
    >>
    >>
    >> x = 1000
    >>
    >> def foo():
    >>      print x
    >>      x = 10
    >>
    >> Throws the same error. The remedy is to inform python that a specific
    >> name belongs to global scope, using the "global"-statement.
    >>
    >> def foo():
    >>      global x
    >>      print x
    >>      x = 10
    >>
    >>
    >> Beware though that then of course *assigning* to x is on global level.
    >> This shouldn't be of any difference in your case though, because of the
    >> import-only-once-mechanics of python.
    >>
    >> Diez
    >>

    >
    > So...it should not work
    >
    > def A():
    >     import __builtin__
    >     import os
    >     __builtin__.os = os
    >
    > A()
    > os.stat("/")
    >
    > but it does.  Why ? B() cannot see the import, but the global level can ?


    The optimization which results in the behavior in question is only
    done on functions scopes, not global scope.

    Cheers,
    Chris
    --
    http://blog.rebertia.com
    Chris Rebert, Nov 11, 2009
    #4
  5. Python searches for Variables not only in local or global scoop but also
    in __builtins__. If you do something like __builtins__.os = os, than
    this variable should be accessible global.

    If you then write something like:
    def B():
    os.stat("/")
    import os

    Python recognises on compile time, that os is a local variable in B and
    allocates memory for it. All reading or writing now goes to the local
    variable and not the one in __builtins__. And the local variable has
    never been assigned to a value and an Exception is thrown.

    Best
    Andreas Löscher, Nov 11, 2009
    #5
  6. On Wed, 2009-11-11 at 12:21 +0100, Diez B. Roggisch wrote:
    > As the import-statement in a function/method-scope doesn't leak the
    > imported names into the module scope, python treats them as locals.
    > Which makes your code equivalent to
    >
    >
    > x = 1000
    >
    > def foo():
    > print x
    > x = 10


    Aha, thanks. This makes it clear whats happening.

    > Throws the same error. The remedy is to inform python that a specific
    > name belongs to global scope, using the "global"-statement.
    >
    > def foo():
    > global x
    > print x
    > x = 10
    >
    >
    > Beware though that then of course *assigning* to x is on global level.
    > This shouldn't be of any difference in your case though, because of the
    > import-only-once-mechanics of python.


    Is there a way to make the "global x" apply to all functions without
    adding it to each one?

    I suspect this equates to intentionally "leaking the imported names into
    the module scope"? :)

    What I'm trying to do is to avoid having "import X" statements
    everywhere by changing __builtin__. It seems my approach doesn't have
    quite the same effect as a true import though.

    Cheers,

    Richard
    Richard Purdie, Nov 11, 2009
    #6
  7. Richard Purdie

    Chris Rebert Guest

    On Wed, Nov 11, 2009 at 4:37 AM, Richard Purdie <> wrote:
    <snip>
    > Is there a way to make the "global x" apply to all functions without
    > adding it to each one?


    Thankfully, no.

    > What I'm trying to do is to avoid having "import X" statements
    > everywhere by changing __builtin__. It seems my approach doesn't have
    > quite the same effect as a true import though.


    And you can't just put all your imports together at the top of the
    file because...?
    If you're importing the same stuff across multiple modules, I'd say
    you have a code smell on your hands. Injecting stuff into the builtin
    namespace purely for convenience's sake is Just Evil (tm). At the
    least, you can put all the imports in one module and then use `from
    all_imports import *`, which is just slightly less evil.

    Cheers,
    Chris
    --
    http://blog.rebertia.com
    Chris Rebert, Nov 11, 2009
    #7
  8. Richard Purdie

    Ralax Guest

    On Nov 11, 6:59 pm, Richard Purdie <> wrote:
    > I've been having problems with an unexpected exception from python which
    > I can summarise with the following testcase:
    >
    > def A():
    >     import __builtin__
    >     import os
    >
    >     __builtin__.os = os
    >
    > def B():
    >     os.stat("/")
    >     import os
    >
    > A()
    > B()
    >
    > which results in:
    >
    > Traceback (most recent call last):
    >   File "./test.py", line 12, in <module>
    >     B()
    >   File "./test.py", line 8, in B
    >     os.stat("/")
    > UnboundLocalError: local variable 'os' referenced before assignment
    >
    > If I remove the "import os" from B(), it works as expected.
    >
    > >From what I've seen, its very unusual to have something operate

    >
    > "backwards" in scope in python. Can anyone explain why this happens?
    >
    > Cheers,
    >
    > Richard


    One word, Python treat objects in a function or method-scope as
    locals.
    So os is not defined in B(), is it right?
    Ralax, Nov 11, 2009
    #8
  9. On Wed, 2009-11-11 at 05:04 -0800, Chris Rebert wrote:
    > On Wed, Nov 11, 2009 at 4:37 AM, Richard Purdie <> wrote:
    > <snip>
    > > Is there a way to make the "global x" apply to all functions without
    > > adding it to each one?

    >
    > Thankfully, no.


    Hmm :(.

    > > What I'm trying to do is to avoid having "import X" statements
    > > everywhere by changing __builtin__. It seems my approach doesn't have
    > > quite the same effect as a true import though.

    >
    > And you can't just put all your imports together at the top of the
    > file because...?


    The application in question is bitbake, the parsing tool for the
    OpenEmbedded and Poky projects.

    We're talking about python code fragments spread across about 8,000
    files which make up the "metadata" some of which is public and some of
    which is not. Rightly or wrongly bitbake does some interesting things
    with its execution contexts for these code fragments.

    Lets just say its not a traditional use case, we know the way bitbake
    does some things is evil but it does other things rather well and we
    can't break those.

    Cheers,

    Richard
    Richard Purdie, Nov 11, 2009
    #9
  10. Richard Purdie

    Eduardo Lenz Guest

    Em Qua 11 Nov 2009, às 03:21:55, Diez B. Roggisch escreveu:
    > Richard Purdie schrieb:
    > > I've been having problems with an unexpected exception from python which
    > > I can summarise with the following testcase:
    > >
    > > def A():
    > > import __builtin__
    > > import os
    > >
    > > __builtin__.os = os
    > >
    > > def B():
    > > os.stat("/")
    > > import os
    > >
    > > A()
    > > B()
    > >
    > > which results in:
    > >
    > > Traceback (most recent call last):
    > > File "./test.py", line 12, in <module>
    > > B()
    > > File "./test.py", line 8, in B
    > > os.stat("/")
    > > UnboundLocalError: local variable 'os' referenced before assignment
    > >
    > > If I remove the "import os" from B(), it works as expected.
    > >
    > >>From what I've seen, its very unusual to have something operate

    > >
    > > "backwards" in scope in python. Can anyone explain why this happens?

    >
    > As the import-statement in a function/method-scope doesn't leak the
    > imported names into the module scope, python treats them as locals.
    > Which makes your code equivalent to
    >
    >
    > x = 1000
    >
    > def foo():
    > print x
    > x = 10
    >
    > Throws the same error. The remedy is to inform python that a specific
    > name belongs to global scope, using the "global"-statement.
    >
    > def foo():
    > global x
    > print x
    > x = 10
    >
    >
    > Beware though that then of course *assigning* to x is on global level.
    > This shouldn't be of any difference in your case though, because of the
    > import-only-once-mechanics of python.
    >
    > Diez
    >


    So...it should not work

    def A():
    import __builtin__
    import os
    __builtin__.os = os

    A()
    os.stat("/")

    but it does. Why ? B() cannot see the import, but the global level can ?

    Thanks.

    Eduardo.

    --
    Esta mensagem foi verificada pelo sistema de antivírus e
    acredita-se estar livre de perigo.
    Eduardo Lenz, Nov 11, 2009
    #10
  11. En Wed, 11 Nov 2009 11:11:31 -0300, Ralax <> escribió:
    > On Nov 11, 6:59 pm, Richard Purdie <> wrote:


    >> def B():
    >> os.stat("/")
    >> import os
    >>
    >> Traceback (most recent call last):
    >> File "./test.py", line 12, in <module>
    >> B()
    >> File "./test.py", line 8, in B
    >> os.stat("/")
    >> UnboundLocalError: local variable 'os' referenced before assignment
    >>
    >> If I remove the "import os" from B(), it works as expected.
    >> From what I've seen, its very unusual to have something operate
    >> "backwards" in scope in python. Can anyone explain why this happens?

    >
    > One word, Python treat objects in a function or method-scope as
    > locals.


    Not exactly; from the Language Reference [1]:

    "The following constructs bind names: formal parameters to functions,
    import statements, class and function definitions (these bind the class or
    function name in the defining block), and targets that are identifiers if
    occurring in an assignment, for loop header, or in the second position of
    an except clause header. The import statement of the form “from ...import
    *†binds all names defined in the imported module, except those beginning
    with an underscore. This form may only be used at the module level. [...]
    If a name binding operation occurs anywhere within a code block, all uses
    of the name within the block are treated as references to the current
    block. This can lead to errors when a name is used within a block before
    it is bound. This rule is subtle. Python lacks declarations and allows
    name binding operations to occur anywhere within a code block. The local
    variables of a code block can be determined by scanning the entire text of
    the block for name binding operations."

    > So os is not defined in B(), is it right?


    "os" is a local name due to import, and starts uninitialized; using it
    before it is initialized raises UnboundLocalError

    [1] http://docs.python.org/reference/executionmodel.html

    --
    Gabriel Genellina
    Gabriel Genellina, Nov 13, 2009
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. nicolas edel
    Replies:
    5
    Views:
    3,045
    Gordon Beaton
    Jul 31, 2003
  2. James Ly
    Replies:
    3
    Views:
    671
    James
    Dec 3, 2003
  3. Replies:
    2
    Views:
    12,643
  4. Teddy

    unexpected exception

    Teddy, May 25, 2005, in forum: C++
    Replies:
    3
    Views:
    1,555
    Razzer
    May 25, 2005
  5. R.A.M.
    Replies:
    4
    Views:
    9,879
    R.A.M.
    May 19, 2006
Loading...

Share This Page