Strange effect with import

[Jens Thoms Toerring]

Hi,

I hope that this isn't a stupid question, asked already a
hundred times, but I haven't found anything definitive on
the problem I got bitten by. I have two Python files like
this:

-------- S1.py ------
import random
import S2

class R( object ) :
r = random.random( )

if __name__ == "__main__" :
print R.r
S2.p( )

-------- S2.py ------
import S1

def p( ) :
print S1.R.r

and my expectation was that the static variable 'r' of class
R would be identical when accessed from S1.py and S2.py.
Unfortunately, that isn't the case, the output is different
(and R seems to get instantiated twice).

But when I define R in S2.py instead

-------- S1.py ------
import S2

print S2.R.r
S2.p( )

-------- S2.py ------
import random

class R( object ) :
r = random.random( )

def p( ) :
print R.r

or, alternatively, if I put the defintion of class R into
a third file which I then import from the other 2 files,
things suddenly start to work as expected/ Can someone
explain what's going one here? I found this a bit sur-
prising.

This is, of course, not my "real" code - it would be much
more sensible to pass the number to the function in the
second file as an argument - but is the smallest possinle
program I could come up with that demonstrate the prob-
lem. In my "real" code it's unfortunately not possible
to pass that number to whatever is going to use it in the
other file, I have to simulate a kind of global variable
shared between different files.

Best regards, Jens

 

[Dave Angel]

Hi,

I hope that this isn't a stupid question, asked already a
hundred times, but I haven't found anything definitive on
the problem I got bitten by. I have two Python files like
this:

-------- S1.py ------
import random
import S2

class R( object ) :
r = random.random( )

if __name__ == "__main__" :
print R.r
S2.p( )

-------- S2.py ------
import S1

You have a big problem right here. You have two modules importing each
other. Any time you have direct or indirect mutual imports, you have
the potential for trouble.

That trouble gets much worse since you are actually running one of these
as a script. Presumably you're running S1.py as a script. The script's
module object is NOT the same one as the other module S2 gets by
importing S1. Don't do that.

Move the common code into a third module, and import that one from both
places. Then it'll only exist once.

 

[Peter Otten]

Hi,

I hope that this isn't a stupid question, asked already a
hundred times, but I haven't found anything definitive on
the problem I got bitten by. I have two Python files like
this:

-------- S1.py ------
import random
import S2

class R( object ) :
r = random.random( )

if __name__ == "__main__" :
print R.r
S2.p( )

-------- S2.py ------
import S1

def p( ) :
print S1.R.r

and my expectation was that the static variable 'r' of class
R would be identical when accessed from S1.py and S2.py.
Unfortunately, that isn't the case, the output is different
(and R seems to get instantiated twice).

But when I define R in S2.py instead

-------- S1.py ------
import S2

print S2.R.r
S2.p( )

-------- S2.py ------
import random

class R( object ) :
r = random.random( )

def p( ) :
print R.r

or, alternatively, if I put the defintion of class R into
a third file which I then import from the other 2 files,
things suddenly start to work as expected/

That's the correct approach.

Can someone
explain what's going one here? I found this a bit sur-
prising.

You should never import your program's main module anywhere else in the
program. When Python imports a module it looks it up by the module's name in
the sys.modules cache. For the main script that name will be "__main__"
regardless of the file's actual name, so a subsequent "import S2" will
result in a cache miss and a new module instance.

Similar problems occur when there is a PYTHONPATH pointing into a package
and you have both

import package.module

and

import module

Again you will end up with two module instances, one called
"package.module", the other just "module".

 

[Steven D'Aprano]

Hi,

I hope that this isn't a stupid question, asked already a
hundred times, but I haven't found anything definitive on the problem I
got bitten by. I have two Python files like this:

-------- S1.py ------
import random
import S2

class R( object ) :
r = random.random( )

if __name__ == "__main__" :
print R.r
S2.p( )

-------- S2.py ------
import S1

def p( ) :
print S1.R.r

and my expectation was that the static variable 'r' of class R

The terminology we prefer here is "class attribute", not "static
variable". Attributes are always assigned in dynamic storage, whether
they are per-instance or on the class.

would be
identical when accessed from S1.py and S2.py. Unfortunately, that isn't
the case, the output is different (and R seems to get instantiated
twice).

You don't instantiate R at all. You only ever refer to the class object,
you never instantiate it to create an instance. What you are actually
seeing is a side-effect of the way Python modules are imported:

- Python modules are instances that are instantiated at import
time, and then cached by module name;

- the module name is *usually* the file name (sans .py extension),
except when you are running it as a script, in which case it
gets set to the special value "__main__" instead.

So the end result is that you actually end up with THREE module objects,
__main__, S2 and S1, even though there are only two module *files*. Both
__main__ and S1 are instantiated from the same source code and contain
the same objects: both have a class called R, with fully-qualified names
__main__.R and S1.R, but they are separate objects.


[...]

or, alternatively, if I put the defintion of class R into a third file
which I then import from the other 2 files, things suddenly start to
work as expected/ Can someone explain what's going one here? I found
this a bit surprising.

You have a combination of two tricky situations:

* A circular import: module S1 imports S2, and S2 imports S1.

* A .py file, S1.py, being used as both an importable module
and a runnable script.

Circular imports are usually hard to get rid at the best of time.
Combined with the second factor, they can lead to perplexing errors, as
you have just found out.

This is, of course, not my "real" code - it would be much more sensible
to pass the number to the function in the second file as an argument -
but is the smallest possinle program I could come up with that
demonstrate the problem.

And let me say sincerely, thank you for doing so! You would be amazed how
many people do not make any effort to simplify their problem before
asking for help.

In my "real" code it's unfortunately not
possible to pass that number to whatever is going to use it in the
other file, I have to simulate a kind of global variable
shared between different files.

Well, I find that hard to believe. "Not convenient"? I could believe
that. "Difficult"? Maybe. "Tricky"? I could even believe that. But "not
possible"? No, I don't believe that it is impossible to pass variables
around as method arguments.

 

[Jens Thoms Toerring]

Thanks a lot to all three of you: that helped me understand
the errors of my ways! You just saved me a few more hours
of head-scratching;-)

A few replies to the questions and comments by Steven:

The terminology we prefer here is "class attribute", not "static
variable". Attributes are always assigned in dynamic storage, whether
they are per-instance or on the class.

I'm comimg from C/C++ and that's were my terminology is from,
I know I still have to learn a lot more about Python;-)

Well, I find that hard to believe. "Not convenient"? I could believe
that. "Difficult"? Maybe. "Tricky"? I could even believe that. But "not
possible"? No, I don't believe that it is impossible to pass variables
around as method arguments.

You are rather likely right and I probably should have written:
"I don't see any way to pass that variable to the object that
is supposed to use it". Perhaps you have an idea how it could
be done correctly when I explain the complete picture: I'm
writing a TCP server, based on SocketServer:

server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)

where ReqHandler is the name of a class derived from
SocketServer.BaseRequestHandler

class ReqHandler(SocketServer.BaseRequestHandler):
...

A new instance of this class is gernerated for each connection
request to the server. In the call that creates the server I can
only specify the name of the class but no arguments to be passed
to it on instantiation - at least I found nothing in the docu-
mentation. On the other hand I need to get some information into
this class and thus the only idea I came up with was to use some
kind of global variable for the purpose. Perhaps there's a much
better way to do that but I haven't found one yet. Or perhaps it
is an omission in the design of SocketServer or (more likely) my
mis-understanding of the documentation (as I wrote I'm relatively
new to Python).
Thnak you and best regards, Jens

 

[Terry Reedy]

You are rather likely right and I probably should have written:
"I don't see any way to pass that variable to the object that
is supposed to use it". Perhaps you have an idea how it could
be done correctly when I explain the complete picture: I'm
writing a TCP server, based on SocketServer:

server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)

where ReqHandler is the name of a class derived from
SocketServer.BaseRequestHandler

You misunderstood the doc. You pass the class, not the name of the class.
From 21.19.4.1. socketserver.TCPServer Example
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)

MyTCPHandler is the actual class. What gets 'passed' at the C level in
CPython is a reference that class that TCPServer can use to call it, but
conceptually, at the Python level, think of it as the class. In the
code, you enter the name without quotes and that expression evaluates to
the (reference to the) class that gets passed.

If the signature required the name, the example would have had
'MyTCPHandler', with the quotes, to pass the name as a string.

Very few builtin functions require names as strings. open('filename'),
somebytes.encode(encoding='encoding-name', errors =
'error-handler-name') are two that come to mind. Notice that these are
situations where requiring a non-string object would be inconvenient at
best.

class ReqHandler(SocketServer.BaseRequestHandler):
...

A new instance of this class is gernerated for each connection
request to the server. In the call that creates the server I can
only specify the name of the class but no arguments to be passed

Code those arguments directly into the handle method of your version of
MyTCPhandler. Or if you need to override multiple methods and use the
same values in multiple methods, override __init__ and add self.x =
x-value statements.

 

[Hans Mulder]

I'm writing a TCP server, based on SocketServer:

server = SocketServer.TCPServer((192.168.1.10, 12345), ReqHandler)

where ReqHandler is the name of a class derived from
SocketServer.BaseRequestHandler

class ReqHandler(SocketServer.BaseRequestHandler):
...

A new instance of this class is gernerated for each connection
request to the server. In the call that creates the server I can
only specify the name of the class but no arguments to be passed
to it on instantiation - at least I found nothing in the docu-
mentation.

What happens if instead of a class you pass a function that
takes the same arguments as the SocketServer.BaseRequestHandler
constructor and returns a new instance of your ReqHandler?

That's not quite what the documentaion clls for, but I'd hope
it's close enough.


Maybe something like this:

class ReqHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server, ham, spam)
super(SocketServer, self).__init__(
self, request, client_address, server)
self.ham = ham
self.spam = spam
....

And later:

import functools

server = SocketServer.TCPServer((192.168.1.10, 12345),
functools.partial(ReqHandler, ham="hello", spam=42))

On the other hand I need to get some information into
this class and thus the only idea I came up with was to use some
kind of global variable for the purpose. Perhaps there's a much
better way to do that but I haven't found one yet. Or perhaps it
is an omission in the design of SocketServer

I think you could call it a weakness in the design of SocketServer.

Life would be easier if it took as an optional third argument some
sequence that it would pass as extra arguments when it instantiates
the handler instance. Then you wouldn't have to play with functools
(or closures, or global variables) to solve your problems.

or (more likely) my mis-understanding of the documentation
(as I wrote I'm relatively new to Python).

From where I sit, it looks like the authors of the SocketServer
module didn't expect subclasses of BaseRequestHandler to need
extra attributes than their base class.

Or maybe they thought everybody knew functools.partial.


Hope this helps,

-- HansM

 

[Jens Thoms Toerring]

You misunderstood the doc. You pass the class, not the name of the class.
From 21.19.4.1. socketserver.TCPServer Example
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)

Yes, I meant "the class", but I'm a bit weak on nomenclature in
Python;-)

Code those arguments directly into the handle method of your version of
MyTCPhandler. Or if you need to override multiple methods and use the
same values in multiple methods, override __init__ and add self.x =
x-value statements.

Sorry, you lost me there: what means "code those arguments
directly into the handle method"? According to the documen-
tation (or at least to my understanding of it;-) the handle()
method is suppose to accept just one argument, 'self'. And
even if I would change the method to accept more arguments
and that wouldnt blow up into my face, where would they be
coming from (and from where would I pass them)?

Best regards, Jens

 

[Jens Thoms Toerring]

What happens if instead of a class you pass a function that
takes the same arguments as the SocketServer.BaseRequestHandler
constructor and returns a new instance of your ReqHandler?

That's not quite what the documentaion clls for, but I'd hope
it's close enough.

Interesting idea - I'm not yet at a level of Python wizardry
that I would dare to do something that's not explicitely bles-
sed be the documentation;-)

Maybe something like this:

class ReqHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server, ham, spam)
super(SocketServer, self).__init__(
self, request, client_address, server)
self.ham = ham
self.spam = spam
....

And later:

import functools

server = SocketServer.TCPServer((192.168.1.10, 12345),
functools.partial(ReqHandler, ham="hello", spam=42))

Ok, that's still way over may head at the moment;-) I will hhave
to read up on functools tomorrow, it's the first time I heard of
it but it looks quite interesting at a first glance.

Thank you for these ideas, I'll need a bit of time to figure out
these new concepts and I don't think I'm up to it tonight any-
more;-)
Best regards. Jens

 

[Jens Thoms Toerring]

Maybe something like this:

class ReqHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server, ham, spam)
super(SocketServer, self).__init__(
self, request, client_address, server)
self.ham = ham
self.spam = spam
....

The only thing I had to change about this was to assign the
additional class variables before calling super() because in
the __init__() method of the base class my overloaded handle()
method is already called which needs those extra variables.

And later:

import functools

server = SocketServer.TCPServer((192.168.1.10, 12345),
functools.partial(ReqHandler, ham="hello", spam=42))

Thanks a lot, that's now all working perfectly well and I got
rid of those pesky global variables;-) Probably the guys that
wrote the SocketServer module indeed didn't expect people as
dense as me to use their module and thus didn't mention that
passing additional information to a handler object can be done
this way...
Best regards, Jens