Using a dict as if it were a module namespace

[Steven D'Aprano]

I have a problem which I think could be solved by using a dict as a
namespace, in a similar way that exec and eval do.

When using the timeit module, it is very inconvenient to have to define
functions as strings. A good alternative is to create the function as
normal, and import it:

def myfunc(x, y):
return x+y

timeit.Timer("myfunc(59, 60)", "from __main__ import myfunc").timeit()


Not only is this an easy idiom to follow, but myfunc can live in another
module: just replace __main__ with the module name.

Now, I'm trying to build a suite of tests to use with timeit. I have a
bunch of tests which I've created as dicts:

test_suite= [dict(x=59, y=60), dict(x=-1, y=-2)]

What I *think* I want to do is use the from ... import idiom to grab
arguments from the dicts as if they were modules, but this doesn't work:

expr = "myfunc(x, y)"
for test in test_suite:
setup = "from __main__ import myfunc; from test import x, y"
t = timeit.Timer(expr, setup).timeit()


Even if the Timer could see test, it is not a module and you can't import
from it. Naturally.


Alternatives that I have found:

(1) Import the test and grab the values needed from it:

setup = """from __main__ import myfunc, test
x, y = test['x'], test['y']"""


I don't like this one. It doesn't seem very elegant to me, and it gets
unwieldy as the complexity increases. Every item I need from test has to
be named twice, violating the principle Don't Repeat Yourself. If the
tests change, the setup string has to be explicitly changed also.


(2) Mess with the global namespace:

globals().update(t)
setup = "from __main__ import myfunc"

I don't like this one. It looks hackish, and I worry about conflicts and
side-effects. If it works (and I haven't tested it) it relies on an
implementation detail of timeit.Timer.__init__, namely the line
"exec code in globals(), ns". Worst of all, it pollutes or even mangles
the global namespace of the calling code, not the code being tested.


(3) Explicitly pass a namespace dict to the Timer class, possibly even
getting rid of setup altogether:

test['myfunc'] = myfunc
t = timeit.Timer(expr, '', ns=test).timeit()

This would be the most elegant solution, but at this time it is
completely hypothetical. Timer does not have that functionality.


(4) Dump the test data straight into the setup string:

setup = "from __main__ import myfunc; x = %(x)s; y = %(y)s" % t

Again, unwieldy and against DRY. The additional disadvantage is that
there are many types of test data that can't be converted to and from
strings like that.


What do others think? Have I missed something? What other alternatives
are there?

 

[thebjorn]

I have a problem which I think could be solved by using a dict as a
namespace, in a similar way that exec and eval do.

When using the timeit module, it is very inconvenient to have to define
functions as strings. A good alternative is to create the function as
normal, and import it:

def myfunc(x, y):
return x+y

timeit.Timer("myfunc(59, 60)", "from __main__ import myfunc").timeit()

Not only is this an easy idiom to follow, but myfunc can live in another
module: just replace __main__ with the module name.

Now, I'm trying to build a suite of tests to use with timeit. I have a
bunch of tests which I've created as dicts:

test_suite= [dict(x=59, y=60), dict(x=-1, y=-2)]

What I *think* I want to do is use the from ... import idiom to grab
arguments from the dicts as if they were modules, but this doesn't work:

expr = "myfunc(x, y)"
for test in test_suite:
setup = "from __main__ import myfunc; from test import x, y"
t = timeit.Timer(expr, setup).timeit()

Even if the Timer could see test, it is not a module and you can't import
from it. Naturally.

Alternatives that I have found:

(1) Import the test and grab the values needed from it:

setup = """from __main__ import myfunc, test
x, y = test['x'], test['y']"""

I don't like this one. It doesn't seem very elegant to me, and it gets
unwieldy as the complexity increases. Every item I need from test has to
be named twice, violating the principle Don't Repeat Yourself. If the
tests change, the setup string has to be explicitly changed also.

(2) Mess with the global namespace:

globals().update(t)
setup = "from __main__ import myfunc"

I don't like this one. It looks hackish, and I worry about conflicts and
side-effects. If it works (and I haven't tested it) it relies on an
implementation detail of timeit.Timer.__init__, namely the line
"exec code in globals(), ns". Worst of all, it pollutes or even mangles
the global namespace of the calling code, not the code being tested.

(3) Explicitly pass a namespace dict to the Timer class, possibly even
getting rid of setup altogether:

test['myfunc'] = myfunc
t = timeit.Timer(expr, '', ns=test).timeit()

This would be the most elegant solution, but at this time it is
completely hypothetical. Timer does not have that functionality.

(4) Dump the test data straight into the setup string:

setup = "from __main__ import myfunc; x = %(x)s; y = %(y)s" % t

Again, unwieldy and against DRY. The additional disadvantage is that
there are many types of test data that can't be converted to and from
strings like that.

What do others think? Have I missed something? What other alternatives
are there?

You might have lost me, but wouldn't it be easier to do some variation
on this

test_suite = [
'(x=59, y=60)', # either have strings here...
'(x=-1, y=-2)',
]

for test in test_suite:
# ... or convert the dicts to appropriate strings here...
expr = 'myfunc' + test
t = timeit.Timer(expr, 'from __main__ import myfunc').timeit()
...

-- bjorn

 

[Ross Ridge]

(1) Import the test and grab the values needed from it:

setup = """from __main__ import myfunc, test
x, y = test['x'], test['y']"""


I don't like this one. It doesn't seem very elegant to me, and it gets
unwieldy as the complexity increases. Every item I need from test has to
be named twice, violating the principle Don't Repeat Yourself. If the
tests change, the setup string has to be explicitly changed also.

I think this is the way to go as it follows the principle of "say what
you mean." You can however simplify it, and repeat yourself less,
by using the extended call syntax:

expr = "myfunc(**test)"
setup = """from __main__ import myfunc, test"""

....

I don't like this one. It looks hackish, and I worry about conflicts and
side-effects. If it works (and I haven't tested it) it relies on an
implementation detail of timeit.Timer.__init__, namely the line
"exec code in globals(), ns". Worst of all, it pollutes or even mangles
the global namespace of the calling code, not the code being tested.

It wouldn't work because the timeit module's "globals" are different
from the __main__ module's globals.

Ross Ridge

 

[Steven Bethard]

I have a problem which I think could be solved by using a dict as a
namespace, in a similar way that exec and eval do.

When using the timeit module, it is very inconvenient to have to define
functions as strings. A good alternative is to create the function as
normal, and import it:

def myfunc(x, y):
return x+y

timeit.Timer("myfunc(59, 60)", "from __main__ import myfunc").timeit()


Not only is this an easy idiom to follow, but myfunc can live in another
module: just replace __main__ with the module name.

Now, I'm trying to build a suite of tests to use with timeit. I have a
bunch of tests which I've created as dicts:

test_suite= [dict(x=59, y=60), dict(x=-1, y=-2)]

What I *think* I want to do is use the from ... import idiom to grab
arguments from the dicts as if they were modules, but this doesn't work:

expr = "myfunc(x, y)"
for test in test_suite:
setup = "from __main__ import myfunc; from test import x, y"
t = timeit.Timer(expr, setup).timeit() [snip]
(2) Mess with the global namespace:

globals().update(t)
setup = "from __main__ import myfunc"

Why not mess with the namespace inside the setup code? E.g.::

>>> test_suite = [dict(x=59, y=60), dict(x=-1, y=-2)]
>>> expr = "myfunc(x, y)"
>>> for test in test_suite:

... setup = textwrap.dedent('''
... from __main__ import myfunc, test
... globals().update(test)''')
... t = timeit.Timer(expr, setup).timeit()
...

That shouldn't pollute your actual namespace::
Traceback (most recent call last):
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
NameError: name 'y' is not defined

STeVe

 

[Steven D'Aprano]

Why not mess with the namespace inside the setup code? E.g.::

[snip]

Ah, thanks, good thinking. It's not perfect, but it should be close
enough for what I'm trying to do.

 

[Steven D'Aprano]

I think this is the way to go as it follows the principle of "say what
you mean." You can however simplify it, and repeat yourself less, by
using the extended call syntax:

expr = "myfunc(**test)"
setup = """from __main__ import myfunc, test"""

Ah, also good thinking! I'd have to modify it, because the tests carry
other information which shouldn't be passed to the test function, but
that's a simple modification.

Thanks to all who replied.