replacing __dict__ with an OrderedDict

[Ulrich Eckhardt]

Hi!

The topic explains pretty much what I'm trying to do under Python
2.7[1]. The reason for this is that I want dir(SomeType) to show the
attributes in the order of their declaration. This in turn should
hopefully make unittest execute my tests in the order of their
declaration[2], so that the output becomes more readable and structured,
just as my test code (hopefully) is.

I've been toying around with metaclasses, trying to replace __dict__
directly, or using type() to construct the class, but either I hit
read-only attributes or the changes seem to be ignored.

As additional thought (but I'm not there yet), I guess that if I want
dir(some_object) to be ordered similarly, I will also have to perform
some steps on instance creation, not only on class creation, right?

Thanks for any suggestions!

Uli


[1] I'm stuck with 2.x for now, from what I found it would be easier
using 3.x.

[2] No, don't tell me that the tests should run in any order. It's not
the case that my tests depend on each other, but if one basic test for
the presence of an API fails, the elaborate tests using that API will
obviously fail, too, and I just want to take the first test that fails
and analyse that instead of guessing the point to start debugging from
the N failed tests.

 

[Lie Ryan]

Hi!

The topic explains pretty much what I'm trying to do under Python
2.7[1]. The reason for this is that I want dir(SomeType) to show the
attributes in the order of their declaration. This in turn should
hopefully make unittest execute my tests in the order of their
declaration[2], so that the output becomes more readable and structured,
just as my test code (hopefully) is.

IMO that's a futile effort, first, because as you already know, the test
should not rely on the order. If you want the result to be printed in a
certain order, that's a display issue, not testing order issue. I guess
you would have better luck doing what you want by customizing the
TestResult or TestRunner.

 

[Peter Otten]

The topic explains pretty much what I'm trying to do under Python
2.7[1]. The reason for this is that I want dir(SomeType) to show the
attributes in the order of their declaration. This in turn should
hopefully make unittest execute my tests in the order of their
declaration[2], so that the output becomes more readable and structured,
just as my test code (hopefully) is.

I've been toying around with metaclasses, trying to replace __dict__
directly, or using type() to construct the class, but either I hit
read-only attributes or the changes seem to be ignored.

Alternatively you can write your own test loader:

$ cat ordered_unittest2.py
import unittest

class Test(unittest.TestCase):
def test_gamma(self):
pass
def test_beta(self):
pass
def test_alpha(self):
pass

class Loader(unittest.TestLoader):
def getTestCaseNames(self, testCaseClass):
"""Return a sequence of method names found within testCaseClass
sorted by co_firstlineno.
"""
def first_lineno(name):
method = getattr(testCaseClass, name)
return method.im_func.__code__.co_firstlineno

function_names = super(Loader, self).getTestCaseNames(testCaseClass)
function_names.sort(key=first_lineno)
return function_names


if __name__ == "__main__":
unittest.main(testLoader=Loader())

$ python2.7 ordered_unittest2.py -v
test_gamma (__main__.Test) ... ok
test_beta (__main__.Test) ... ok
test_alpha (__main__.Test) ... ok

 

[Ulrich Eckhardt]

Am 06.01.2012 12:43, schrieb Lie Ryan:

Hi!

The topic explains pretty much what I'm trying to do under Python
2.7[1]. The reason for this is that I want dir(SomeType) to show the
attributes in the order of their declaration. This in turn should
hopefully make unittest execute my tests in the order of their
declaration[2], so that the output becomes more readable and structured,
just as my test code (hopefully) is.

IMO that's a futile effort, first, because as you already know, the test
should not rely on the order. If you want the result to be printed in a
certain order, that's a display issue, not testing order issue.

I'm not sure if you just -ahem- enjoy Usenet discussions, but please
read the footnote that explains that I don't want to discuss this part
and why you are actually wrong with your partial understanding of the
issue. ;^)

I guess you would have better luck doing what you want by customizing
the TestResult or TestRunner.

True, perhaps, but doing it this way would be more fun and easier
reusable in other cases where the default order is not desirable. I can
also go and name the test functions test_000 to test_009 to get results
quickly, if that was the only goal.

Cheers!

Uli

 

[Ulrich Eckhardt]

Am 06.01.2012 12:44, schrieb Peter Otten:

Alternatively you can write your own test loader:

[...CODE...]

Well, actually you just did that for me and it works!


Nonetheless, I'm still wondering if I could somehow replace the dict
with an OrderedDict.

Thank you!

Uli

 

[Ethan Furman]

Hi!

The topic explains pretty much what I'm trying to do under Python
2.7[1]. The reason for this is that I want dir(SomeType) to show the
attributes in the order of their declaration. This in turn should
hopefully make unittest execute my tests in the order of their
declaration[2], so that the output becomes more readable and structured,
just as my test code (hopefully) is.

I believe unittest executes tests in alphabetical order, but it does not
display them in the same order when it prints error/fail results.

~Ethan~

 

[Ian Kelly]

Nonetheless, I'm still wondering if I could somehow replace the dict with an
OrderedDict.

In Python 3, yes. This is pretty much the entire use case for the new
__prepare__ method of metaclasses. See the "OrderedClass" example at:
http://docs.python.org/py3k/reference/datamodel.html#special-method-names

Cheers,
Ian

 

[Ian Kelly]

In Python 3, yes.  This is pretty much the entire use case for the new
__prepare__ method of metaclasses.  See the "OrderedClass" example at:
http://docs.python.org/py3k/reference/datamodel.html#special-method-names

That link should be:
http://docs.python.org/py3k/reference/datamodel.html#customizing-class-creation

 

[Lie Ryan]

Am 06.01.2012 12:43, schrieb Lie Ryan:

Hi!

The topic explains pretty much what I'm trying to do under Python
2.7[1]. The reason for this is that I want dir(SomeType) to show the
attributes in the order of their declaration. This in turn should
hopefully make unittest execute my tests in the order of their
declaration[2], so that the output becomes more readable and structured,
just as my test code (hopefully) is.

IMO that's a futile effort, first, because as you already know, the test
should not rely on the order. If you want the result to be printed in a
certain order, that's a display issue, not testing order issue.

I'm not sure if you just -ahem- enjoy Usenet discussions, but please
read the footnote that explains that I don't want to discuss this part
and why you are actually wrong with your partial understanding of the
issue. ;^)

I fully understand your issue, and I stand by my opinion. I believe
you're misunderstanding the nature of your problem, your issue is not
that you want a customized test order execution, but you want a
customized view of the test result.

That unittest executes its tests in alphabetical order is implementation
detail for a very good reason, and good unittest practice dictates that
execution order should never be defined (some even argued that the
execution order should be randomized). If the test runner turns out to
execute tests concurrently, that should not cause problems for a
well-designed test. Displaying the test results in a more convenient
order for viewing is what you really wanted in 99.99% of the cases.

True, perhaps, but doing it this way would be more fun and easier
reusable in other cases where the default order is not desirable. I can
also go and name the test functions test_000 to test_009 to get results
quickly, if that was the only goal.

Fun and easier, perhaps. Except that it solves the wrong problem.

 

[Ian Kelly]

That unittest executes its tests in alphabetical order is implementation
detail for a very good reason, and good unittest practice dictates that
execution order should never be defined (some even argued that the execution
order should be randomized). If the test runner turns out to execute tests
concurrently, that should not cause problems for a well-designed test.
Displaying the test results in a more convenient order for viewing is what
you really wanted in 99.99% of the cases.

Randomizing the order is not a bad idea, but you also need to be able
to run the tests in a consistent order, from a specific random seed.
In the real world, test conflicts and dependencies do happen, and if
we observe a failure, make a change, rerun the tests and observe
success, we need to be able to be sure that we actually fixed the bug,
and that it didn't pass only because it was run in a different order.

Concurrent testing is a bad idea for this reason -- it's not
repeatable (testing concurrency, OTOH, is a perfectly fine thing to be
thinking about).

Cheers,
Ian

 

[Lie Ryan]

Randomizing the order is not a bad idea, but you also need to be able
to run the tests in a consistent order, from a specific random seed.
In the real world, test conflicts and dependencies do happen, and if
we observe a failure, make a change, rerun the tests and observe
success, we need to be able to be sure that we actually fixed the bug,
and that it didn't pass only because it was run in a different order.

Concurrent testing is a bad idea for this reason -- it's not
repeatable (testing concurrency, OTOH, is a perfectly fine thing to be
thinking about).

Concurrent testing is perfectly fine strategy in the case where you have
thousands of tests and running them synchronously will just take too
long. Certainly it makes it harder to repeat the test if there is any
sort of dependency in the tests, but when you have the large number of
tests, the benefit may exceeds the drawbacks.

 

[Arnaud Delobelle]

class Loader(unittest.TestLoader):
   def getTestCaseNames(self, testCaseClass):
       """Return a sequence of method names found within testCaseClass
       sorted by co_firstlineno.
       """

That's a clever trick!

 

[Arnaud Delobelle]

Am 06.01.2012 12:44, schrieb Peter Otten:

Alternatively you can write your own test loader:

[...CODE...]

Well, actually you just did that for me and it works!


Nonetheless, I'm still wondering if I could somehow replace the dict with an
OrderedDict.

No, you can't.

 

[Steven D'Aprano]

Fun and easier, perhaps. Except that it solves the wrong problem.

Fun and easier, and a terrible thing to do. Contrast:

"test_binsearch_tuple is failing. What does that mean?"
"Oh, that tests that binsearch works on a tuple. You need to look at the
BinSearch.search_tuple method and see what it's doing."

with

"test_047 is failing. What does that mean?"
"How the hell should I know?"


Debugging is hard enough without obfuscating the test names. You wouldn't
write a function called "func_009", why write one called "test_009"?

 

[Steven D'Aprano]

Randomizing the order is not a bad idea, but you also need to be able to
run the tests in a consistent order, from a specific random seed. In the
real world, test conflicts and dependencies do happen,

In the real world, test conflicts and dependencies are bugs in your test
suite that should be fixed, like any other bug in code. The fact that it
is test code that is failing is irrelevant.

Also in the real world, sometimes it is too hard to fix a bug and you
just live with it.

and if we observe
a failure, make a change, rerun the tests and observe success, we need
to be able to be sure that we actually fixed the bug, and that it didn't
pass only because it was run in a different order.

Every test should stand alone. You should be able to isolate each and
every test and run it without the others. If you can't, your test suite
is buggy. (Chances are good that most test suites are buggy. After all,
who writes tests for their tests? That's just the start of an infinite
regress with rapidly diminishing benefit.)

You may not be able to run tests *simultaneously*, due to clashes
involving external resources, but you should be able to run them in
random order.

 

[Lie Ryan]

You may not be able to run tests*simultaneously*, due to clashes
involving external resources, but you should be able to run them in
random order.

tests that involves external resources should be mocked, although there
are always a few external resources that cannot be mocked, those are the
exceptions not the rule; a concurrent test runner might have mechanisms
to mark them to be run synchronously.

 

[Ian Kelly]

In the real world, test conflicts and dependencies are bugs in your test
suite that should be fixed, like any other bug in code. The fact that it
is test code that is failing is irrelevant.

I agree 100%, but none of that changes my point, which is that when
that this sort of problem arises, you need to be able to test
consistently to know that the bug is actually fixed.

Every test should stand alone. You should be able to isolate each and
every test and run it without the others.

Ideally, yes. I'm talking about *unintentional* conflicts and dependencies.

 

[Eelco]

i havnt read every post in great detail, but it doesnt seem like your
actual question has been answered, so ill give it a try.

AFAIK, changing __dict__ to be an ordereddict is fundamentally
impossible in python 2. __dict__ is a builtin language construct
hardcoded into the C API. There is no way to mess with it.

Apparently this is different in python 3, but I dont know much about
that.

 

[alex23]

Nonetheless, I'm still wondering if I could somehow replace the dict with an
OrderedDict.

In Python 3, yes.  This is pretty much the entire use case for the new
__prepare__ method of metaclasses.  See the "OrderedClass" example[...]

This isn't accurate. The OrderedClass example uses an OrderedDict to
remember the method creation order:

def __new__(cls, name, bases, classdict):
result = type.__new__(cls, name, bases, dict(classdict))
result.members = tuple(classdict)
return result

The instantiated objects __dict__ will still be a regularly
dictionary, while the assignment order is stored in the class
attribute .members.

 

[Lie Ryan]

i havnt read every post in great detail, but it doesnt seem like your
actual question has been answered, so ill give it a try.

AFAIK, changing __dict__ to be an ordereddict is fundamentally
impossible in python 2. __dict__ is a builtin language construct
hardcoded into the C API. There is no way to mess with it.

Apparently this is different in python 3, but I dont know much about
that.

Actually the primary question has been answered by Ian Kelly which
suggested __prepare__ for Python 3, and Peter Otten posted a code for a
custom TestLoader that will essentially do what the OP wanted.

I was just suggesting that what the OP thinks he wants is quite likely
not what he actually wants.