A startup puzzle

[Edward K. Ream]

I've just about convinced myself there is no good, clean solution to the
following puzzle. I wonder if you agree.

1. My app has a module called leoGlobals that contains often-used global
functions. All of Leo's source files start with:

from leoGlobals import *

I don't want to discuss whether this is good style: it is simple and it
works well for me

2. All code accesses the singleton application object using the app()
method in leoGlobals.

3. To make the code cleaner looking, I would really like the code to be
able to access an app global instead of the app() function. This has
nothing to do with speed: it's simply that there are lots of references to
app() in the code, and there is soon going to be a lot more references to
app.gui.x() and app.gui.y() etc.

As a workaround, many methods and functions assign a = app(), but I would
like to avoid this step.

4. Alas, it does not seem possible to initialize an app global in
leoGlobals. The reason is simple:

from leoGlobals import *

caches the value of app at the time the import is done. But the startup
code that creates the app "global" (really an attribute of the leoGlobals
module) must do _other_ imports.

For example, doing the following at the top of leoGlobals doesn't work:

import leoApp
app = leoApp.leoApp() # construct the app instance.

Indeed, the leoApp module will be imported before the assignment of app.
Moreover, the leoApp module does other imports, and all the app variables in
those modules will be uninitialized.

It would be horrible style to place detailed constraints on the order in
which modules get imported, and I'm not sure even that would work.

Any ideas for a clean solution? Thanks.

Edward

P.S. The solution must work in 2.1, 2.2 and 2.3.

EKR

 

[Peter Otten]

I've just about convinced myself there is no good, clean solution to the
following puzzle. I wonder if you agree.

If I understand you correctly, you do

leoGlobals.py
_app=None
def app():
global _app
if not _app:
# other code
_app = Application()
return _app

leoMain.py
from leoGlobals import *
app = app()
app.mainLoop()

leoOther.py
from leoGlobals import *

app().doSomething()


Does it happen that your program runs without ever creating the Application
instance? If not, there's no point in lazy creation:

leoGlobals.py
app=None
def createApp():
global app
assert not app
app = Application()

leoMain.py
from leoGlobals import *
createApp()
app.mainLoop()

leoOther.py
from leoGlobals import *
# this works iff you can ensure that leoMain.py is the
# only "entry point" to your application
app.doSomething()


If I missed something, it would probably help if you post similar code
fragments to show where the above scheme would fail to work.

Peter

 

[Stephen Horne]

Any ideas for a clean solution? Thanks.

I am not sure that I understand what you are saying, but I'll make a
quick suggestion...

Perhaps 'app' can be a global variable within 'leoGlobals', but
assigned the value 'None' when the module is imported. Your
initialisation code can then reassign the variable at the appropriate
time.

If you refer to 'app.whatever' while 'app' is still 'None' you should
get an exception.

That's not what most people mean by 'singleton' of course, but if you
can be sure of your initialisation sequence it can probably do the
job.

 

[Michael Hudson]

I am not sure that I understand what you are saying, but I'll make a
quick suggestion...

Perhaps 'app' can be a global variable within 'leoGlobals', but
assigned the value 'None' when the module is imported. Your
initialisation code can then reassign the variable at the appropriate
time.

This wont work.

--- a.py ---
a = None
--- b.py ---
import a
a.a = 1
--- c.py ---
from a import a
import b
print a

running c.py will print "None". Names, objects and bindings, remember


Cheers,
mwh

 

[Stephen Horne]

This wont work.

--- a.py ---
a = None
--- b.py ---
import a
a.a = 1
--- c.py ---
from a import a
import b
print a

running c.py will print "None". Names, objects and bindings, remember

Oops - I didn't realise import worked like that.

Thanks.

 

[Peter Otten]

Peter Otten wrote:

Oops! My consideration does not work with
from leoGlobals import *
as the rebinding is not propagated. The following seems to work although I
do not know if this is approved Python:

leoGlobals.py, second try:
from leoApp import Application

class Dummy(object):
pass

app = Dummy()

def createApp():
app.__class__ = Application
app.__dict__ = Application().__dict__


Peter

 

[Glenn Andreas]

I've just about convinced myself there is no good, clean solution to the
following puzzle. I wonder if you agree. [snip]
For example, doing the following at the top of leoGlobals doesn't work:

import leoApp
app = leoApp.leoApp() # construct the app instance.

Indeed, the leoApp module will be imported before the assignment of app.
Moreover, the leoApp module does other imports, and all the app variables in
those modules will be uninitialized.

It would be horrible style to place detailed constraints on the order in
which modules get imported, and I'm not sure even that would work.

Any ideas for a clean solution? Thanks.

Maybe I'm missing something, but it seems like you want some sort of
lazy initialization thing here. Perhaps using a proxy object sort of
like this might work:


realApp = None
class LeoProxy:
def __getattr__(self,attr):
if not realApp:
import leoApp
realApp = leoApp.leoApp()
return getattr(realApp, attr)

app = LeoProxy()


Then you can import that module as often as you want, but the first time
you do something with leoApp.app (i.e., "leoApp.app.doSomething()"), it
will only then call the "real" function to create it (and then proxy it
over as needed)

 

[Mike C. Fletcher]

I've just about convinced myself there is no good, clean solution to the
following puzzle. I wonder if you agree.

1. My app has a module called leoGlobals that contains often-used global
functions. All of Leo's source files start with:

from leoGlobals import *

I don't want to discuss whether this is good style: it is simple and it
works well for me

Okay, we'll take that as a given then.

2. All code accesses the singleton application object using the app()
method in leoGlobals.

Sure, implements late-binding via indirection.

3. To make the code cleaner looking, I would really like the code to be
able to access an app global instead of the app() function. This has
nothing to do with speed: it's simply that there are lots of references to
app() in the code, and there is soon going to be a lot more references to
app.gui.x() and app.gui.y() etc.

As a workaround, many methods and functions assign a = app(), but I would
like to avoid this step.

This can be done by creating a built-in variable e.g. APPLICATION, which
is an application-wide variable, like so:

def __install( self ):
"""Install self into the builtin module"""
__builtin__.APPLICATION = self

note that this *doesn't* help if there really are hidden dependencies in
your modules on having the APPLICATION object defined, but given your
description of the architecture, that doesn't appear to be the case.

4. Alas, it does not seem possible to initialize an app global in
leoGlobals. The reason is simple:

from leoGlobals import *

caches the value of app at the time the import is done. But the startup
code that creates the app "global" (really an attribute of the leoGlobals
module) must do _other_ imports.

For example, doing the following at the top of leoGlobals doesn't work:

import leoApp
app = leoApp.leoApp() # construct the app instance.

Indeed, the leoApp module will be imported before the assignment of app.
Moreover, the leoApp module does other imports, and all the app variables in
those modules will be uninitialized.

It would be horrible style to place detailed constraints on the order in
which modules get imported, and I'm not sure even that would work.

Such constraints will exist. Any module which relies on the existence
of an initialised APPLICATION variable must be imported after the
creation of that variable. You construct the app so that the
APPLICATION is created ASAP and that becomes a minimal restriction,
since your code is working, it's already passed that level of
restriction .

The alternatives to such restrictions are pretty grotty; defered
execution of suites of code that would likely require Stackless Python
and a decent masters thesis working out the calculus of dependency
resolution (what with needing to defer or speculatively execute each
path in the code dependant on the app object's operation). Luckily
that's all just academic for your project .

Any ideas for a clean solution? Thanks.

Cleanliness is in the eye of the beholder. I don't really like the
APPLICATION-object "pattern", but it is very convenient in many simple
cases. Just about everything else I've seen either requires something
exotic (e.g. proxy objects implementing the late-binding indirection by
pretending to be the app object (raising errors if anything attempts to
use the object before it's bound)), or the solution you started with,
i.e. using function's built-in late-binding to implement it.

HTH,
Mike

_______________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://members.rogers.com/mcfletch/

 

[Bengt Richter]

I've just about convinced myself there is no good, clean solution to the
following puzzle. I wonder if you agree.

1. My app has a module called leoGlobals that contains often-used global
functions. All of Leo's source files start with:

from leoGlobals import *

I don't want to discuss whether this is good style: it is simple and it
works well for me

2. All code accesses the singleton application object using the app()
method in leoGlobals.

3. To make the code cleaner looking, I would really like the code to be
able to access an app global instead of the app() function. This has
nothing to do with speed: it's simply that there are lots of references to
app() in the code, and there is soon going to be a lot more references to
app.gui.x() and app.gui.y() etc.

As a workaround, many methods and functions assign a = app(), but I would
like to avoid this step.

How about an app proxy object that just passes through attribute accesses later?
That way you can immediately in leoGlobal create something that will have the
app name binding that all importers of leoGlobal can copy the binding to, but
which won't/shouldn't get used 'till the app behind it is initialized.

(see code at end)

4. Alas, it does not seem possible to initialize an app global in
leoGlobals. The reason is simple:

from leoGlobals import *

caches the value of app at the time the import is done. But the startup
code that creates the app "global" (really an attribute of the leoGlobals
module) must do _other_ imports.

Ok, see code below

For example, doing the following at the top of leoGlobals doesn't work:

import leoApp
app = leoApp.leoApp() # construct the app instance.

Well, plain vanilla it works, but not if leoApp() wants to import leoGlobals and expect app
to be already bound.

But see code below.

Indeed, the leoApp module will be imported before the assignment of app.

Well, that's the order you wrote it ;-) I mean, no prob so far, right?

Moreover, the leoApp module does other imports, and all the app variables in
those modules will be uninitialized.

Because _they_ do from leoGlobals import * and leoGlobals is not ready
for that yet.

It would be horrible style to place detailed constraints on the order in
which modules get imported, and I'm not sure even that would work.

Any ideas for a clean solution? Thanks.

Well, I'm just exploring your problem in its particular form right now, so
I'm not sure what can be guaranteed, but it looks like when you import leoApp
and it triggers a rash of "from leoGlobals import *", the leoGlobals in effect
will be a snapshot of the leoGlobal module dict at that time, so what has been
defined/bound will be visible, but whatever has not, won't, e.g., the appInvisible
in leoGlobals is visible to my interactive from leoGlobals import *, but not
via the app method that returns the global dict _it_ sees. The latter does have bindings
for the other globals though.

There may be more to it than that, but that's what I get from experimenting as below:

===< leoGlobals.py >======================
class AppWrap(object):
def setapp(self, theRealApp): self.theRealApp = theRealApp
def __getattr__(self, name): return getattr(self.__dict__['theRealApp'], name)
app = AppWrap()
K1k = 1024
whatever = 'whatever value'
app.setapp(__import__('leoApp').leoApp())
appInvisible = 'App import dependents do not see this in their leoGlobals'
==========================================

===< leoApp.py >==========================
from leoGlobals import *
class leoApp(object):
def foometh(self): print 'foometh of %r called.'%self
def getapp(self): return app # from leoGlobals import above
def peek(self): return locals(), globals()
==========================================

Experimental result:

>>> from leoGlobals import *
>>> dir() ['AppWrap', 'K1k', '__builtins__', '__doc__', '__name__', 'app', 'appInvisible', 'whatever']
>>> app

>>> app.foometh()

>>> app.getapp

>>> app.getapp().foometh()

foometh of <leoApp.leoApp object at 0x008F9E70> called.

Now we'll peek at the globals in the leoApp module via the peek method

>>> app.peek()[1]['whatever'] 'whatever value'
>>> app.peek()[1]['K1k']

1024

Those were visible, but let's try that last one ;-)

>>> app.peek()[1]['appInvisible']

Traceback (most recent call last):
File "<stdin>", line 1, in ?
KeyError: 'appInvisible'

But here in the interactive environment, the import * got the final state of leoGlobals, so
'App import dependents do not see this in their leoGlobals'

And the app binding seen in other modules should be ok (though not ok to use until app.setapp
has been called, so you will have to watch dependencies in the way you order things in leoGlobals --
you want app=AppWrap() at the top, and presumably app.setapp(...) at the end, if it's going
to trigger imports that want to see everything preceding).

Here is an access to an app method via the global imported in another (leoApp) module,
though of course this example is otherwise silly ;-)

>>> app.peek()[1]['app']

>>> app.peek()[1]['app'].foometh()

foometh of <leoApp.leoApp object at 0x00902450> called.

I guess you will lose a little speedwise by going through the wrapped app name, but
does that let you spell things the way you wanted?

HTH

Regards,
Bengt Richter

 

[Hans Nowak]

I've just about convinced myself there is no good, clean solution to the
following puzzle. I wonder if you agree.

1. My app has a module called leoGlobals that contains often-used global
functions. All of Leo's source files start with:

from leoGlobals import *

I don't want to discuss whether this is good style: it is simple and it
works well for me

Fair enough, but it's not just a matter of style... using something like

import leoGlobals as g

would probably have avoided this problem, since you can do:

import leoGlobals as g

# later...
g.app = leoApp.leoApp()
# every module that imported leoGlobals now knows g.app as well

Cheers,

 

[Edward K. Ream]

Perhaps using a proxy object...might work:

Great idea! Some complications:

1. Sometimes the test "if not realApp:" causes unbounded recursion. To
avoid any possibility of this happening the startup code just creates the
app object before any code actually uses the app proxies. This is easily
done.

2. A __call__ method in the proxy allows the code to use either app().x or
app.x.

In short, the following pattern appears like it will work in all my modules:

from leoGlobals import *
app = leoProxy() # leoProxy defined in leoGlobals
# Now the code can reference either app.x or app().x.

To do this, I use the following code:

gApp = None # Set before any reference to proxy.
class leoProxy:
def __getattr__(self,attr):
return getattr(gApp,attr)
def __call__(self):
return gApp

Many thanks for this great idea.

Edward

 

[Aahz]

I've just about convinced myself there is no good, clean solution to the
following puzzle. I wonder if you agree.

1. My app has a module called leoGlobals that contains often-used global
functions. All of Leo's source files start with:

from leoGlobals import *

I don't want to discuss whether this is good style: it is simple and it
works well for me

Okay, I won't discuss, I'll just tell you that this is a recipe for
giving yourself a headache. Have fun. ("Doctor, it hurts when I do
this." "Well, stop doing that, then.")

 

[Bengt Richter]

Great idea! Some complications:

1. Sometimes the test "if not realApp:" causes unbounded recursion. To
avoid any possibility of this happening the startup code just creates the
app object before any code actually uses the app proxies. This is easily
done.

1a. I suspect Glenn's code would also need a global realApp in the __getattr__ method,
or it would just keep seeing a global realApp==None and recreating the app instance
to a local realApp that disappears after its desired attribute is returned.
(BTW my version just let an exception happen if use was attempted before setting
the real app reference).

2. A __call__ method in the proxy allows the code to use either app().x or
app.x.

In short, the following pattern appears like it will work in all my modules:

from leoGlobals import *
app = leoProxy() # leoProxy defined in leoGlobals

ISTM it would be safe to put the above line in leoGlobals at the top, after
the class definition, and then app would just come in with the import *,
so in general your modules will only need the one-line from leoGlobals import *.

(see my post for more info. BTW, I didn't see Glenn's post 'til after, honest ;-)

# Now the code can reference either app.x or app().x.

Cool. Gives you some backwards compatibility I guess.

To do this, I use the following code:

Since the code is only executed once on import, I think you could put this
at the top of leoGlobals

gApp = None # Set before any reference to proxy.
class leoProxy:
def __getattr__(self,attr):
return getattr(gApp,attr)
def __call__(self):
return gApp

(Unlike Glenn's code, this doesn't need a global gApp declaration).

and then at the bottom of leoGlobals you could do (untested)
gApp = __import__('leoApp').leoApp()

Or you could (untested) even do it later from somewhere else as
leoGlobals.gApp = __import__('leoApp').leoApp()

Many thanks for this great idea.

I should've just blurted and not fooled with testing, I guess ;-)

Actually, I wonder whether the snap-shot effect you seem to get in
a secondary module's import of the module whose import code you are in
the middle of executing is something one can depend on in detail not to change.
(though I'm not sure what kind of change could be reasonable ;-)

And what happens if there are side effects from the secondary activity
that puts something in the module globals -- will that show up in subsequent
imports immediately? I guess it comes down to whether import locks in a
module instance and ties it to sys.modules immediately, so its state can
be modified and viewed incrementally. It seems so. I guess the dict involved
is the environment that the imported module code gets executed in.

Further .02USD are in my other post ;-)

Regards,
Bengt Richter

 

[Steve Holden]

Great idea! Some complications:

1. Sometimes the test "if not realApp:" causes unbounded recursion. To
avoid any possibility of this happening the startup code just creates the
app object before any code actually uses the app proxies. This is easily
done.

2. A __call__ method in the proxy allows the code to use either app().x or
app.x.

In short, the following pattern appears like it will work in all my modules:

from leoGlobals import *
app = leoProxy() # leoProxy defined in leoGlobals
# Now the code can reference either app.x or app().x.

To do this, I use the following code:

gApp = None # Set before any reference to proxy.
class leoProxy:
def __getattr__(self,attr):
return getattr(gApp,attr)
def __call__(self):
return gApp

Many thanks for this great idea.

I've followed this thread, and I'm having trouble understanding why nobody's
suggested using Alex martelli's "borg" pattern, which he originally called a
"statel;ess proxy", where all objects of a given type share the same state
information. Sounds to me that's what you really want, no?

This would mean that in leo_global you would have something like this:

leo_global.py:
-------------
class Borg(object):
_state = {}
def __new__(cls, *p, **k):
self = object.__new__(cls, *p, **k)
self.__dict__ = cls._state
return self


module1.py
-------------
from leo_global import Borg

myborg = Borg()
myborg.var1 = "This is var 1"

import module2

print "Module 1's borg has var1 =", myborg.var1, id(myborg.var1)
print "Module 1's borg has var2 =", myborg.var2, id(myborg.var2)


module2.py:
-------------
from leo_global import Borg

myborg = Borg()
myborg.var2 = "This is var 2"

print "Module 2's borg has var1 =", myborg.var1, id(myborg.var1)
print "Module 2's borg has var2 =", myborg.var2, id(myborg.var2)

The result of all this appears to be what you want: a simple way of creating
an easily-referenced shared state. Vide:

C:\Steve\Projects\Python>python module1.py
Module 2's borg has var1 = This is var 1 7766584
Module 2's borg has var2 = This is var 2 8008928
Module 1's borg has var1 = This is var 1 7766584
Module 1's borg has var2 = This is var 2 8008928

regards

 

[Edward K. Ream]

Fair enough, but it's not just a matter of style... using something like

import leoGlobals as g

would probably have avoided this problem...

True! However, I must then change every reference x to a function in
leoGlobals to g.x. This is what I want to avoid. Being able to go from
app().x to app.x is a very small improvement: I want to avoid a
corresponding very small step backwards.

Edward

 

[Edward K. Ream]

I don't want to discuss whether this is good style: it is simple and it

Okay, I won't discuss, I'll just tell you that this is a recipe for
giving yourself a headache.

I could agree with you if using "from leoGlobals import *" had ever caused
me the slightest ache. Aside from this startup problem, now completely
solved, it never has.

As suggested in another reply, replacing every reference to x in leoGlobals
by g.x might be considered better style, but this is precisely what I want
to avoid. It is always perfectly obvious in the code that a reference to a
"bare" function is a reference to a function in leoGlobals. What could be
clearer?

Edward

 

[Edward K. Ream]

In short, the following pattern appears like it will work in all my
modules:

from leoGlobals import *
app = leoProxy() # leoProxy defined in leoGlobals
# Now the code can reference either app.x or app().x.

Actually, app = leoProxy() can be done in leoGlobals, so the pattern is
just:

from leoGlobals import *
# Now the code can reference either app.x or app().x.

I have installed the new code in leoGlobals.py and everything "just works".

Edward

 

[Bengt Richter]

I've followed this thread, and I'm having trouble understanding why nobody's
suggested using Alex martelli's "borg" pattern, which he originally called a

I think because (IIUC) the OP wants to import * to avoid naming a container,
and he's willing to live read-only for the resulting bindings (though the
bindings can of course be to mutable things).

"statel;ess proxy", where all objects of a given type share the same state
information. Sounds to me that's what you really want, no?

This would mean that in leo_global you would have something like this:

leo_global.py:
-------------
class Borg(object):
_state = {}
def __new__(cls, *p, **k):
self = object.__new__(cls, *p, **k)
self.__dict__ = cls._state
return self


module1.py
-------------
from leo_global import Borg

myborg = Borg()
myborg.var1 = "This is var 1"

import module2

print "Module 1's borg has var1 =", myborg.var1, id(myborg.var1)
print "Module 1's borg has var2 =", myborg.var2, id(myborg.var2)


module2.py:
-------------
from leo_global import Borg

myborg = Borg()
myborg.var2 = "This is var 2"

print "Module 2's borg has var1 =", myborg.var1, id(myborg.var1)
print "Module 2's borg has var2 =", myborg.var2, id(myborg.var2)

The result of all this appears to be what you want: a simple way of creating
an easily-referenced shared state. Vide:

C:\Steve\Projects\Python>python module1.py
Module 2's borg has var1 = This is var 1 7766584
Module 2's borg has var2 = This is var 2 8008928
Module 1's borg has var1 = This is var 1 7766584
Module 1's borg has var2 = This is var 2 8008928

Yes, but UIAM [1]

import leo_global as myborg # leo_global.py being an empty file for this example

instead of

from leo_global import Borg
myborg = Borg()

would let you do what you are showing in the above ;-)

[1] Got a little nervous about that, so I tried it:

============================================================
[ 6:43] C:\pywk\clp\leo\SteveHolden>dir leo_global.py
Volume in drive C is System
Volume Serial Number is 14CF-C4B9

Directory of C:\pywk\clp\leo\SteveHolden

03-09-30 06:35 0 leo_global.py
1 File(s) 0 bytes
56,197,120 bytes free

[ 6:43] C:\pywk\clp\leo\SteveHolden>type module1.py
#module1.py
#-------------
#from leo_global import Borg
#
#myborg = Borg()
import leo_global as myborg
myborg.var1 = "This is var 1"

import module2

print "Module 1's borg has var1 =", myborg.var1, id(myborg.var1)
print "Module 1's borg has var2 =", myborg.var2, id(myborg.var2)




[ 6:44] C:\pywk\clp\leo\SteveHolden>type module2.py
# module2.py:
#-------------
#from leo_global import Borg
#
#myborg = Borg()
import leo_global as myborg
myborg.var2 = "This is var 2"

print "Module 2's borg has var1 =", myborg.var1, id(myborg.var1)
print "Module 2's borg has var2 =", myborg.var2, id(myborg.var2)


[ 6:44] C:\pywk\clp\leo\SteveHolden>python module1.py
Module 2's borg has var1 = This is var 1 9439256
Module 2's borg has var2 = This is var 2 9439496
Module 1's borg has var1 = This is var 1 9439256
Module 1's borg has var2 = This is var 2 9439496

=================================================================================
Regards,
Bengt Richter

 

[Edward K. Ream]

Okay, I won't discuss, I'll just tell you that this is a recipe for

giving yourself a headache.

The more I think about this remark, the more I tend to agree with you. What
I said about not having trouble with "from leoGlobals import *" is true, and
yet...

The problem isn't so much the present namespace pollution, the problem is
that the technique doesn't extend well. There comes a point at which there
are just too many names to keep track of. So if I were managing a group of
programmers (thank goodness I'm not I would tend to disapprove of what I
have done.

Still, I don't see a perfect alternative. For sure I wouldn't like to put a
g. in front of hundreds or thousands of function calls. I suppose that this
is the kind of thing one would be forced to do in slightly larger
projects...Any other ideas?

Edward

 

[Peter Hansen]

The more I think about this remark, the more I tend to agree with you. What
I said about not having trouble with "from leoGlobals import *" is true, and
yet...

The problem isn't so much the present namespace pollution, the problem is
that the technique doesn't extend well. There comes a point at which there
are just too many names to keep track of. So if I were managing a group of
programmers (thank goodness I'm not I would tend to disapprove of what I
have done.

Still, I don't see a perfect alternative. For sure I wouldn't like to put a
g. in front of hundreds or thousands of function calls. I suppose that this
is the kind of thing one would be forced to do in slightly larger
projects...Any other ideas?

The wxPython project recently switched from using "from wx.wxPython import *"
to a nice simple "import wx" at the top. All calls and constants are now of
course prefixed with "wx." where before they were unadorned.

The change seems identical to what you are proposing, and we can probably
assume it was done for very good reasons. The resulting code, after I've
made changes to my own, already seems more readable. I encourage you to
make the same switch.

-Peter