Doubley imported module caused devastating bug

Discussion in 'Python' started by Zac Burns, Sep 24, 2009.

  1. Zac Burns

    Zac Burns Guest

    Currently it is possible to import a file of one path to more than one
    'instance' of a module. One notable example is "import __init__" from
    a package. See http://stackoverflow.com/questions/436497/python-import-the-containing-package

    This recently caused a devastating bug in some of my code. What I have
    is support for the Perforce global options as a context for a perforce
    module. http://www.perforce.com/perforce/doc.072/manuals/cmdref/o.gopts.html#1040647
    This way one can call functions that call many perforce command and
    have them execute on a different client for example.

    So, in module A and module B both imported the Perforce module, but
    they turned out not to be the same module. Module A did "with
    Perforce.GlobalOptions(client=client): B.function()"

    B.function did not receive the new GlobalOptions because of this
    problem. As a result important files on the original client were
    overwritten (OUCH).

    I would like to propose that it be made impossible in the Python
    source to import two instances of the same module.

    --
    Zachary Burns
    (407)590-4814
    Aim - Zac256FL
    Production Engineer (Digital Overlord)
    Zindagi Games
     
    Zac Burns, Sep 24, 2009
    #1
    1. Advertising

  2. Zac Burns

    Carl Banks Guest

    On Sep 24, 10:26 am, Zac Burns <> wrote:
    > Currently it is possible to import a file of one path to more than one
    > 'instance' of a module. One notable example is "import __init__" from
    > a package. Seehttp://stackoverflow.com/questions/436497/python-import-the-containin...
    >
    > This recently caused a devastating bug in some of my code. What I have
    > is support for the Perforce global options as a context for a perforce
    > module.http://www.perforce.com/perforce/doc.072/manuals/cmdref/o.gopts.html#...
    > This way one can call functions that call many perforce command and
    > have them execute on a different client for example.
    >
    > So, in module A and module B both imported the Perforce module, but
    > they turned out not to be the same module. Module A did "with
    > Perforce.GlobalOptions(client=client): B.function()"
    >
    > B.function did not receive the new GlobalOptions because of this
    > problem. As a result important files on the original client were
    > overwritten (OUCH).
    >
    > I would like to propose that it be made impossible in the Python
    > source to import two instances of the same module.


    Impossible's a pretty strong word.

    It's a reasonable request, but with Python's importing the way it is
    it'd be kind of hard to do. A Python file can be visible in multiple
    ways.

    However, anyone who does "import __init__" (or "from . import
    __init__" with relative import) is asking for trouble, I can't think
    of any valid reason to do it, and I wouldn't mind seeing that
    forbidden, but it's simple to avoid. Someone probably did that
    because they didn't know how to import a containing package from one
    of its modules, failing to realize that it created a new module.


    Carl Banks
     
    Carl Banks, Sep 24, 2009
    #2
    1. Advertising

  3. Zac Burns

    Zac Burns Guest

    On Thu, Sep 24, 2009 at 1:38 PM, Carl Banks <> wrote:
    > On Sep 24, 10:26 am, Zac Burns <> wrote:
    >> Currently it is possible to import a file of one path to more than one
    >> 'instance' of a module. One notable example is "import __init__" from
    >> a package. Seehttp://stackoverflow.com/questions/436497/python-import-the-containin...
    >>
    >> This recently caused a devastating bug in some of my code. What I have
    >> is support for the Perforce global options as a context for a perforce
    >> module.http://www.perforce.com/perforce/doc.072/manuals/cmdref/o.gopts.html#...
    >> This way one can call functions that call many perforce command and
    >> have them execute on a different client for example.
    >>
    >> So, in module A and module B both imported the Perforce module, but
    >> they turned out not to be the same module. Module A did "with
    >> Perforce.GlobalOptions(client=client): B.function()"
    >>
    >> B.function did not receive the new GlobalOptions because of this
    >> problem. As a result important files on the original client were
    >> overwritten (OUCH).
    >>
    >> I would like to propose that it be made impossible in the Python
    >> source to import two instances of the same module.

    >
    > Impossible's a pretty strong word.
    >
    > It's a reasonable request, but with Python's importing the way it is
    > it'd be kind of hard to do.  A Python file can be visible in multiple
    > ways.
    >
    > However, anyone who does "import __init__" (or "from . import
    > __init__" with relative import) is asking for trouble, I can't think
    > of any valid reason to do it, and I wouldn't mind seeing that
    > forbidden, but it's simple to avoid.  Someone probably did that
    > because they didn't know how to import a containing package from one
    > of its modules, failing to realize that it created a new module.
    >
    >
    > Carl Banks
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >


    There are corner cases. The corner case that I ran into was that there
    were two ways to find the module on PATH because one value of PATH was
    over another. Since then this problem has been removed and it wasn't
    too much trouble to work around - but finding the problem was a real
    pain.

    I am not intimately familiar with the import code and trust your
    judgment that it is difficult. If people are in agreement that this
    should be changed though it could be put in a list somewhere waiting
    for some ambitious person to figure out the implementation, no?

    Personally I think it would be worthwhile.

    --
    Zachary Burns
    (407)590-4814
    Aim - Zac256FL
    Production Engineer (Digital Overlord)
    Zindagi Games
     
    Zac Burns, Sep 24, 2009
    #3
  4. Zac Burns

    Terry Reedy Guest

    Zac Burns wrote:
    > On Thu, Sep 24, 2009 at 1:38 PM, Carl Banks <> wrote:
    >> On Sep 24, 10:26 am, Zac Burns <> wrote:
    >>> Currently it is possible to import a file of one path to more than one
    >>> 'instance' of a module. One notable example is "import __init__" from
    >>> a package. Seehttp://stackoverflow.com/questions/436497/python-import-the-containin...
    >>>
    >>> This recently caused a devastating bug in some of my code. What I have
    >>> is support for the Perforce global options as a context for a perforce
    >>> module.http://www.perforce.com/perforce/doc.072/manuals/cmdref/o.gopts.html#...
    >>> This way one can call functions that call many perforce command and
    >>> have them execute on a different client for example.
    >>>
    >>> So, in module A and module B both imported the Perforce module, but
    >>> they turned out not to be the same module. Module A did "with
    >>> Perforce.GlobalOptions(client=client): B.function()"
    >>>
    >>> B.function did not receive the new GlobalOptions because of this
    >>> problem. As a result important files on the original client were
    >>> overwritten (OUCH).
    >>>
    >>> I would like to propose that it be made impossible in the Python
    >>> source to import two instances of the same module.

    >> Impossible's a pretty strong word.
    >>
    >> It's a reasonable request, but with Python's importing the way it is
    >> it'd be kind of hard to do. A Python file can be visible in multiple
    >> ways.
    >>
    >> However, anyone who does "import __init__" (or "from . import
    >> __init__" with relative import) is asking for trouble, I can't think
    >> of any valid reason to do it, and I wouldn't mind seeing that
    >> forbidden, but it's simple to avoid.


    /__init__.py is basically an implementation hack to make a directory
    also 'be' a file. Use at one own risk, I say.


    > There are corner cases. The corner case that I ran into was that there
    > were two ways to find the module on PATH because one value of PATH was
    > over another. Since then this problem has been removed and it wasn't
    > too much trouble to work around - but finding the problem was a real
    > pain.
    >
    > I am not intimately familiar with the import code and trust your
    > judgment that it is difficult. If people are in agreement that this
    > should be changed though it could be put in a list somewhere waiting
    > for some ambitious person to figure out the implementation, no?


    1. It would slow down all imports, at least a bit.

    2. It would kill code that intentionally makes use of duplicate modules
    (but this could be considered exploitation of a bug, perhaps). It would
    also make forced module reloads harder, it not impossible. Currently,
    just delete the entry in sys.modules.

    3. The language itself does not specify how and where from an
    implementation 'initializes' a module on first import. Indeed, CPython
    has at least three options (.py, .zip, and .dll or .pyd (Windows)), with
    hooks for more. Lets a take the request as specifically preventing the
    creation of duplicate module objects from a particular .py file.

    One implementatin *might* be add a set to sys, say sys.mod_files for
    x.py or x.pyc files already used to initialize a module. The .py or .pyc
    or .pyo would be stripped but the name otherwise should be the absolute
    path. (Including drive letter, on Windows).

    This would not cover the case when files are symlinked (or copied). For
    *nix, a set of inode numbers could be used, but not for Windows. I
    suspect there might be other system-specific problems I have not thought of.

    Terry Jan Reedy
     
    Terry Reedy, Sep 24, 2009
    #4
  5. Zac Burns

    Francis Carr Guest

    > I would like to propose that it be made impossible in the Python
    > source to import two instances of the same module.


    A fully-automatic solution is more difficult than it might seem at
    first:
    http://www.python.org/dev/peps/pep-0328/
    But there is a simple code-discipline solution: never ever use
    relative imports, even between code in the same package.

    We got "bit" by double-imports (a mix of relative, absolute, and even
    cross-imports A "import B" and B "import A") early on in one of our
    projects. The symptom was that the imported module would be
    initialized *twice*, once for a relative import and once for an
    absolute. This is not a happy situation for pseudo-singletons like
    the "logging" module --- esp. if one is hacking the internals! :) We
    no longer use relative imports *EVER*, even within the same package.

    Perhaps Perforce is doing something tricky with scoping or importing,
    and you've just managed to stumble across this trickiness because of a
    double-import. Good luck, these things are a bugger to debug.
     
    Francis Carr, Sep 27, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. dody suria wijaya

    Get importer module from imported module

    dody suria wijaya, Feb 7, 2005, in forum: Python
    Replies:
    5
    Views:
    392
    dody suria wijaya
    Feb 8, 2005
  2. kj
    Replies:
    9
    Views:
    304
    AK Eric
    Nov 12, 2009
  3. David
    Replies:
    2
    Views:
    546
    David
    Feb 11, 2010
  4. Dun Peal
    Replies:
    10
    Views:
    460
    Chris Rebert
    May 3, 2011
  5. Volker Nicolai
    Replies:
    9
    Views:
    965
    Fabian Pilkowski
    Jul 4, 2005
Loading...

Share This Page