Debugging parallel nose tests?

R

Roy Smith

I've got a suite of about 150 tests written using unittest. It takes
5-10 minutes to run, so I'd really like to use nose to run things in
parallel. The problem is, when I do that, I get lots of test failures.
Obviously, we've got some kind of inter-test dependency that I haven't
been able to locate.

Is there some way to make nose print a report of how it partitioned the
tests across the various processes? If I could see that, it might help
us reproduce the failures.

We're using:

nosetests --process-timeout=60 --processes=40 test_api.py

and

_multiprocess_can_split_ = True
 
D

Dave Angel

<SNIP>

nosetests --process-timeout=60 --processes=40 test_api.py

Do you have a 40-processor system? And do you have enough RAM to run
all of those processes?
 
R

Roy Smith

Dave Angel said:
Do you have a 40-processor system?

No, but many of the tests are I/O bound.
And do you have enough RAM to run all of those processes?

Yes. The box I'm on now has 8 gig. I'd doing a 40 process run right
now, and I've still got 1/2 gig free.
 
R

Roy Smith

Roy Smith said:
Is there some way to make nose print a report of how it partitioned the
tests across the various processes?

I never found such a feature, but I did figure out a way to do what I
needed. We use a system of unique id's to track HTTP requests through
our system. All you need do is add a X-Unique-Request-ID header and any
of our server processes will log that. So, I just had my test suite add:

X-Unique-Request-ID: name-of-test.process-id

to each HTTP request. That got me what I was looking for (to know what
tests were grouped into the same process).
We're using:

nosetests --process-timeout=60 --processes=40 test_api.py

It turned out, the answer was purely a timing issue. We had some tests
that called out to facebook. Those were (intermittently) so slow they
caused the 60 second timeout to be exceeded. The solution there was to
pull the facebook tests out to their own suite (which we'll run in
single-process mode).
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top