Strange effect with import

Discussion in 'Python' started by Jens Thoms Toerring, Dec 20, 2012.

  1. Hi,

    I hope that this isn't a stupid question, asked already a
    hundred times, but I haven't found anything definitive on
    the problem I got bitten by. I have two Python files like
    this:

    -------- S1.py ------
    import random
    import S2

    class R( object ) :
    r = random.random( )

    if __name__ == "__main__" :
    print R.r
    S2.p( )

    -------- S2.py ------
    import S1

    def p( ) :
    print S1.R.r

    and my expectation was that the static variable 'r' of class
    R would be identical when accessed from S1.py and S2.py.
    Unfortunately, that isn't the case, the output is different
    (and R seems to get instantiated twice).

    But when I define R in S2.py instead

    -------- S1.py ------
    import S2

    print S2.R.r
    S2.p( )

    -------- S2.py ------
    import random

    class R( object ) :
    r = random.random( )

    def p( ) :
    print R.r

    or, alternatively, if I put the defintion of class R into
    a third file which I then import from the other 2 files,
    things suddenly start to work as expected/ Can someone
    explain what's going one here? I found this a bit sur-
    prising.

    This is, of course, not my "real" code - it would be much
    more sensible to pass the number to the function in the
    second file as an argument - but is the smallest possinle
    program I could come up with that demonstrate the prob-
    lem. In my "real" code it's unfortunately not possible
    to pass that number to whatever is going to use it in the
    other file, I have to simulate a kind of global variable
    shared between different files.

    Best regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Dec 20, 2012
    #1
    1. Advertising

  2. Jens Thoms Toerring

    Dave Angel Guest

    On 12/20/2012 03:39 PM, Jens Thoms Toerring wrote:
    > Hi,
    >
    > I hope that this isn't a stupid question, asked already a
    > hundred times, but I haven't found anything definitive on
    > the problem I got bitten by. I have two Python files like
    > this:
    >
    > -------- S1.py ------
    > import random
    > import S2
    >
    > class R( object ) :
    > r = random.random( )
    >
    > if __name__ == "__main__" :
    > print R.r
    > S2.p( )
    >
    > -------- S2.py ------
    > import S1


    You have a big problem right here. You have two modules importing each
    other. Any time you have direct or indirect mutual imports, you have
    the potential for trouble.

    That trouble gets much worse since you are actually running one of these
    as a script. Presumably you're running S1.py as a script. The script's
    module object is NOT the same one as the other module S2 gets by
    importing S1. Don't do that.

    Move the common code into a third module, and import that one from both
    places. Then it'll only exist once.



    > def p( ) :
    > print S1.R.r
    >
    > and my expectation was that the static variable 'r' of class
    > R would be identical when accessed from S1.py and S2.py.
    > Unfortunately, that isn't the case, the output is different
    > (and R seems to get instantiated twice).
    >
    > But when I define R in S2.py instead
    >
    > -------- S1.py ------
    > import S2
    >
    > print S2.R.r
    > S2.p( )
    >
    > -------- S2.py ------
    > import random
    >
    > class R( object ) :
    > r = random.random( )
    >
    > def p( ) :
    > print R.r
    >
    > or, alternatively, if I put the defintion of class R into
    > a third file which I then import from the other 2 files,
    > things suddenly start to work as expected/ Can someone
    > explain what's going one here? I found this a bit sur-
    > prising.
    >
    > This is, of course, not my "real" code - it would be much
    > more sensible to pass the number to the function in the
    > second file as an argument - but is the smallest possinle
    > program I could come up with that demonstrate the prob-
    > lem. In my "real" code it's unfortunately not possible
    > to pass that number to whatever is going to use it in the
    > other file, I have to simulate a kind of global variable
    > shared between different files.
    >
    > Best regards, Jens



    --

    DaveA
    Dave Angel, Dec 20, 2012
    #2
    1. Advertising

  3. Jens Thoms Toerring

    Peter Otten Guest

    Jens Thoms Toerring wrote:

    > Hi,
    >
    > I hope that this isn't a stupid question, asked already a
    > hundred times, but I haven't found anything definitive on
    > the problem I got bitten by. I have two Python files like
    > this:
    >
    > -------- S1.py ------
    > import random
    > import S2
    >
    > class R( object ) :
    > r = random.random( )
    >
    > if __name__ == "__main__" :
    > print R.r
    > S2.p( )
    >
    > -------- S2.py ------
    > import S1
    >
    > def p( ) :
    > print S1.R.r
    >
    > and my expectation was that the static variable 'r' of class
    > R would be identical when accessed from S1.py and S2.py.
    > Unfortunately, that isn't the case, the output is different
    > (and R seems to get instantiated twice).
    >
    > But when I define R in S2.py instead
    >
    > -------- S1.py ------
    > import S2
    >
    > print S2.R.r
    > S2.p( )
    >
    > -------- S2.py ------
    > import random
    >
    > class R( object ) :
    > r = random.random( )
    >
    > def p( ) :
    > print R.r
    >
    > or, alternatively, if I put the defintion of class R into
    > a third file which I then import from the other 2 files,
    > things suddenly start to work as expected/


    That's the correct approach.

    > Can someone
    > explain what's going one here? I found this a bit sur-
    > prising.


    You should never import your program's main module anywhere else in the
    program. When Python imports a module it looks it up by the module's name in
    the sys.modules cache. For the main script that name will be "__main__"
    regardless of the file's actual name, so a subsequent "import S2" will
    result in a cache miss and a new module instance.

    Similar problems occur when there is a PYTHONPATH pointing into a package
    and you have both

    import package.module

    and

    import module

    Again you will end up with two module instances, one called
    "package.module", the other just "module".

    > This is, of course, not my "real" code - it would be much
    > more sensible to pass the number to the function in the
    > second file as an argument - but is the smallest possinle
    > program I could come up with that demonstrate the prob-
    > lem. In my "real" code it's unfortunately not possible
    > to pass that number to whatever is going to use it in the
    > other file, I have to simulate a kind of global variable
    > shared between different files.
    Peter Otten, Dec 20, 2012
    #3
  4. On Thu, 20 Dec 2012 20:39:19 +0000, Jens Thoms Toerring wrote:

    > Hi,
    >
    > I hope that this isn't a stupid question, asked already a
    > hundred times, but I haven't found anything definitive on the problem I
    > got bitten by. I have two Python files like this:
    >
    > -------- S1.py ------
    > import random
    > import S2
    >
    > class R( object ) :
    > r = random.random( )
    >
    > if __name__ == "__main__" :
    > print R.r
    > S2.p( )
    >
    > -------- S2.py ------
    > import S1
    >
    > def p( ) :
    > print S1.R.r
    >
    > and my expectation was that the static variable 'r' of class R


    The terminology we prefer here is "class attribute", not "static
    variable". Attributes are always assigned in dynamic storage, whether
    they are per-instance or on the class.



    > would be
    > identical when accessed from S1.py and S2.py. Unfortunately, that isn't
    > the case, the output is different (and R seems to get instantiated
    > twice).


    You don't instantiate R at all. You only ever refer to the class object,
    you never instantiate it to create an instance. What you are actually
    seeing is a side-effect of the way Python modules are imported:

    - Python modules are instances that are instantiated at import
    time, and then cached by module name;

    - the module name is *usually* the file name (sans .py extension),
    except when you are running it as a script, in which case it
    gets set to the special value "__main__" instead.

    So the end result is that you actually end up with THREE module objects,
    __main__, S2 and S1, even though there are only two module *files*. Both
    __main__ and S1 are instantiated from the same source code and contain
    the same objects: both have a class called R, with fully-qualified names
    __main__.R and S1.R, but they are separate objects.


    [...]
    > or, alternatively, if I put the defintion of class R into a third file
    > which I then import from the other 2 files, things suddenly start to
    > work as expected/ Can someone explain what's going one here? I found
    > this a bit surprising.


    You have a combination of two tricky situations:

    * A circular import: module S1 imports S2, and S2 imports S1.

    * A .py file, S1.py, being used as both an importable module
    and a runnable script.

    Circular imports are usually hard to get rid at the best of time.
    Combined with the second factor, they can lead to perplexing errors, as
    you have just found out.


    > This is, of course, not my "real" code - it would be much more sensible
    > to pass the number to the function in the second file as an argument -
    > but is the smallest possinle program I could come up with that
    > demonstrate the problem.


    And let me say sincerely, thank you for doing so! You would be amazed how
    many people do not make any effort to simplify their problem before
    asking for help.


    > In my "real" code it's unfortunately not
    > possible to pass that number to whatever is going to use it in the
    > other file, I have to simulate a kind of global variable
    > shared between different files.


    Well, I find that hard to believe. "Not convenient"? I could believe
    that. "Difficult"? Maybe. "Tricky"? I could even believe that. But "not
    possible"? No, I don't believe that it is impossible to pass variables
    around as method arguments.



    --
    Steven
    Steven D'Aprano, Dec 20, 2012
    #4
  5. Thanks a lot to all three of you: that helped me understand
    the errors of my ways! You just saved me a few more hours
    of head-scratching;-)

    A few replies to the questions and comments by Steven:

    Steven D'Aprano <> wrote:
    > On Thu, 20 Dec 2012 20:39:19 +0000, Jens Thoms Toerring wrote:
    > > and my expectation was that the static variable 'r' of class R


    > The terminology we prefer here is "class attribute", not "static
    > variable". Attributes are always assigned in dynamic storage, whether
    > they are per-instance or on the class.


    I'm comimg from C/C++ and that's were my terminology is from,
    I know I still have to learn a lot more about Python;-)

    <good advice snipped>

    > > In my "real" code it's unfortunately not
    > > possible to pass that number to whatever is going to use it in the
    > > other file, I have to simulate a kind of global variable
    > > shared between different files.


    > Well, I find that hard to believe. "Not convenient"? I could believe
    > that. "Difficult"? Maybe. "Tricky"? I could even believe that. But "not
    > possible"? No, I don't believe that it is impossible to pass variables
    > around as method arguments.


    You are rather likely right and I probably should have written:
    "I don't see any way to pass that variable to the object that
    is supposed to use it". Perhaps you have an idea how it could
    be done correctly when I explain the complete picture: I'm
    writing a TCP server, based on SocketServer:

    server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)

    where ReqHandler is the name of a class derived from
    SocketServer.BaseRequestHandler

    class ReqHandler(SocketServer.BaseRequestHandler):
    ...

    A new instance of this class is gernerated for each connection
    request to the server. In the call that creates the server I can
    only specify the name of the class but no arguments to be passed
    to it on instantiation - at least I found nothing in the docu-
    mentation. On the other hand I need to get some information into
    this class and thus the only idea I came up with was to use some
    kind of global variable for the purpose. Perhaps there's a much
    better way to do that but I haven't found one yet. Or perhaps it
    is an omission in the design of SocketServer or (more likely) my
    mis-understanding of the documentation (as I wrote I'm relatively
    new to Python).
    Thnak you and best regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Dec 20, 2012
    #5
  6. Jens Thoms Toerring

    Terry Reedy Guest

    On 12/20/2012 5:52 PM, Jens Thoms Toerring wrote:

    > You are rather likely right and I probably should have written:
    > "I don't see any way to pass that variable to the object that
    > is supposed to use it". Perhaps you have an idea how it could
    > be done correctly when I explain the complete picture: I'm
    > writing a TCP server, based on SocketServer:
    >
    > server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)
    >
    > where ReqHandler is the name of a class derived from
    > SocketServer.BaseRequestHandler


    You misunderstood the doc. You pass the class, not the name of the class.
    From 21.19.4.1. socketserver.TCPServer Example
    server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)

    MyTCPHandler is the actual class. What gets 'passed' at the C level in
    CPython is a reference that class that TCPServer can use to call it, but
    conceptually, at the Python level, think of it as the class. In the
    code, you enter the name without quotes and that expression evaluates to
    the (reference to the) class that gets passed.

    If the signature required the name, the example would have had
    'MyTCPHandler', with the quotes, to pass the name as a string.

    Very few builtin functions require names as strings. open('filename'),
    somebytes.encode(encoding='encoding-name', errors =
    'error-handler-name') are two that come to mind. Notice that these are
    situations where requiring a non-string object would be inconvenient at
    best.


    > class ReqHandler(SocketServer.BaseRequestHandler):
    > ...
    >
    > A new instance of this class is gernerated for each connection
    > request to the server. In the call that creates the server I can
    > only specify the name of the class but no arguments to be passed


    Code those arguments directly into the handle method of your version of
    MyTCPhandler. Or if you need to override multiple methods and use the
    same values in multiple methods, override __init__ and add self.x =
    x-value statements.

    --
    Terry Jan Reedy
    Terry Reedy, Dec 21, 2012
    #6
  7. Jens Thoms Toerring

    Hans Mulder Guest

    On 20/12/12 23:52:24, Jens Thoms Toerring wrote:
    > I'm writing a TCP server, based on SocketServer:
    >
    > server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)
    >
    > where ReqHandler is the name of a class derived from
    > SocketServer.BaseRequestHandler
    >
    > class ReqHandler(SocketServer.BaseRequestHandler):
    > ...
    >
    > A new instance of this class is gernerated for each connection
    > request to the server. In the call that creates the server I can
    > only specify the name of the class but no arguments to be passed
    > to it on instantiation - at least I found nothing in the docu-
    > mentation.


    What happens if instead of a class you pass a function that
    takes the same arguments as the SocketServer.BaseRequestHandler
    constructor and returns a new instance of your ReqHandler?

    That's not quite what the documentaion clls for, but I'd hope
    it's close enough.


    Maybe something like this:

    class ReqHandler(SocketServer.BaseRequestHandler):
    def __init__(self, request, client_address, server, ham, spam)
    super(SocketServer, self).__init__(
    self, request, client_address, server)
    self.ham = ham
    self.spam = spam
    ....

    And later:

    import functools

    server = SocketServer.TCPServer((192.168.1.10, 12345),
    functools.partial(ReqHandler, ham="hello", spam=42))

    > On the other hand I need to get some information into
    > this class and thus the only idea I came up with was to use some
    > kind of global variable for the purpose. Perhaps there's a much
    > better way to do that but I haven't found one yet. Or perhaps it
    > is an omission in the design of SocketServer


    I think you could call it a weakness in the design of SocketServer.

    Life would be easier if it took as an optional third argument some
    sequence that it would pass as extra arguments when it instantiates
    the handler instance. Then you wouldn't have to play with functools
    (or closures, or global variables) to solve your problems.

    > or (more likely) my mis-understanding of the documentation
    > (as I wrote I'm relatively new to Python).


    From where I sit, it looks like the authors of the SocketServer
    module didn't expect subclasses of BaseRequestHandler to need
    extra attributes than their base class.

    Or maybe they thought everybody knew functools.partial.


    Hope this helps,

    -- HansM
    Hans Mulder, Dec 21, 2012
    #7
  8. Terry Reedy <> wrote:
    > > server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)
    > >
    > > where ReqHandler is the name of a class derived from
    > > SocketServer.BaseRequestHandler


    > You misunderstood the doc. You pass the class, not the name of the class.
    > From 21.19.4.1. socketserver.TCPServer Example
    > server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)


    Yes, I meant "the class", but I'm a bit weak on nomenclature in
    Python;-)

    > > A new instance of this class is gernerated for each connection
    > > request to the server. In the call that creates the server I can
    > > only specify the name of the class but no arguments to be passed


    > Code those arguments directly into the handle method of your version of
    > MyTCPhandler. Or if you need to override multiple methods and use the
    > same values in multiple methods, override __init__ and add self.x =
    > x-value statements.


    Sorry, you lost me there: what means "code those arguments
    directly into the handle method"? According to the documen-
    tation (or at least to my understanding of it;-) the handle()
    method is suppose to accept just one argument, 'self'. And
    even if I would change the method to accept more arguments
    and that wouldnt blow up into my face, where would they be
    coming from (and from where would I pass them)?

    Best regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Dec 21, 2012
    #8
  9. Hans Mulder <> wrote:
    > What happens if instead of a class you pass a function that
    > takes the same arguments as the SocketServer.BaseRequestHandler
    > constructor and returns a new instance of your ReqHandler?


    > That's not quite what the documentaion clls for, but I'd hope
    > it's close enough.


    Interesting idea - I'm not yet at a level of Python wizardry
    that I would dare to do something that's not explicitely bles-
    sed be the documentation;-)

    > Maybe something like this:


    > class ReqHandler(SocketServer.BaseRequestHandler):
    > def __init__(self, request, client_address, server, ham, spam)
    > super(SocketServer, self).__init__(
    > self, request, client_address, server)
    > self.ham = ham
    > self.spam = spam
    > ....


    > And later:


    > import functools


    > server = SocketServer.TCPServer((192.168.1.10, 12345),
    > functools.partial(ReqHandler, ham="hello", spam=42))


    Ok, that's still way over may head at the moment;-) I will hhave
    to read up on functools tomorrow, it's the first time I heard of
    it but it looks quite interesting at a first glance.

    Thank you for these ideas, I'll need a bit of time to figure out
    these new concepts and I don't think I'm up to it tonight any-
    more;-)
    Best regards. Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Dec 21, 2012
    #9
  10. Hans Mulder <> wrote:
    > Maybe something like this:


    > class ReqHandler(SocketServer.BaseRequestHandler):
    > def __init__(self, request, client_address, server, ham, spam)
    > super(SocketServer, self).__init__(
    > self, request, client_address, server)
    > self.ham = ham
    > self.spam = spam
    > ....


    The only thing I had to change about this was to assign the
    additional class variables before calling super() because in
    the __init__() method of the base class my overloaded handle()
    method is already called which needs those extra variables.

    > And later:


    > import functools


    > server = SocketServer.TCPServer((192.168.1.10, 12345),
    > functools.partial(ReqHandler, ham="hello", spam=42))


    Thanks a lot, that's now all working perfectly well and I got
    rid of those pesky global variables;-) Probably the guys that
    wrote the SocketServer module indeed didn't expect people as
    dense as me to use their module and thus didn't mention that
    passing additional information to a handler object can be done
    this way...
    Best regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Dec 21, 2012
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?R2lhbmNhcmxv?=

    Strange side effect with events

    =?Utf-8?B?R2lhbmNhcmxv?=, Jan 13, 2005, in forum: ASP .Net
    Replies:
    2
    Views:
    394
    =?Utf-8?B?R2lhbmNhcmxv?=
    Jan 13, 2005
  2. Henning Moll
    Replies:
    4
    Views:
    780
    John C. Bollinger
    Nov 3, 2003
  3. asrahma1111
    Replies:
    11
    Views:
    1,355
    Chris Smith
    May 1, 2004
  4. John Salerno

    strange effect with images?

    John Salerno, Feb 9, 2006, in forum: HTML
    Replies:
    6
    Views:
    386
    John Salerno
    Feb 10, 2006
  5. Viabug123
    Replies:
    1
    Views:
    4,269
    Neredbojias
    Apr 21, 2006
Loading...

Share This Page