Parallel Processing

Y

Yigit Turgut

Hi all,

I am trying to run two functions at the same time with Parallel
Processing (pp) as following ;

import pygame
import sys
import time
import math
import pp

screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
timer = pygame.time.Clock()
white = True
start = time.time()
end = time.time() - start

def test1():
global end
while(end<5):
end = time.time() - start
timer.tick(4) #FPS
screen.fill((255,255,255) if white else (0, 0, 0))
white = not white
pygame.display.update()

def test2():
global end
while(end2<5):
end2 = time.time() - start
print end

ppservers = ()

if len(sys.argv) > 1:
ncpus = int(sys.argv[1])
# Creates jobserver with ncpus workers
job_server = pp.Server(ncpus, ppservers=ppservers)
else:
# Creates jobserver with automatically detected number of workers
job_server = pp.Server(ppservers=ppservers)
print "Starting PP with", job_server.get_ncpus(), "workers"

job1 = job_server.submit(test1,test2)
result = job1()

And the output is ;

Starting PP with 2 workers
Traceback (most recent call last):
File "fl.py", line 46, in <module>
job1 = job_server.submit(test1,test2)
File "/usr/lib/python2.6/site-packages/pp.py", line 402, in submit
raise TypeError("args argument must be a tuple")
TypeError: args argument must be a tuple


When I change job1 to just to see if it will run one function only;
job1 = job_server.submit(test1)

I get an output of;

NameError: global name 'end' is not defined

end variable is defined as global but I get a NameError. Anyone has an
idea what's going on ?
 
D

Dave Angel

Hi all,

I am trying to run two functions at the same time with Parallel
Processing (pp) as following ;

import pygame
import sys
import time
import math
import pp

screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
timer = pygame.time.Clock()
white = True
start = time.time()
end = time.time() - start

def test1():
global end
while(end<5):
end = time.time() - start
timer.tick(4) #FPS
screen.fill((255,255,255) if white else (0, 0, 0))
white = not white
pygame.display.update()

def test2():
global end
while(end2<5):
end2 = time.time() - start
print end

ppservers = ()

if len(sys.argv)> 1:
ncpus = int(sys.argv[1])
# Creates jobserver with ncpus workers
job_server = pp.Server(ncpus, ppservers=ppservers)
else:
# Creates jobserver with automatically detected number of workers
job_server = pp.Server(ppservers=ppservers)
print "Starting PP with", job_server.get_ncpus(), "workers"

job1 = job_server.submit(test1,test2)
result = job1()

And the output is ;

Starting PP with 2 workers
Traceback (most recent call last):
File "fl.py", line 46, in<module>
job1 = job_server.submit(test1,test2)
File "/usr/lib/python2.6/site-packages/pp.py", line 402, in submit
raise TypeError("args argument must be a tuple")
TypeError: args argument must be a tuple


When I change job1 to just to see if it will run one function only;
job1 = job_server.submit(test1)

I get an output of;

NameError: global name 'end' is not defined

end variable is defined as global but I get a NameError. Anyone has an
idea what's going on ?
First, please tell us about any external dependencies (eg. imports).
Pygame may not matter, since there are many people here using it, but
each of us has to hunt down something that does parallel processing with
an interface similar to what you're using. For the next person, the
most likely candidate out of the dozens available is:
http://www.parallelpython.com/

Your problem, as stated in the raise statement, is that the submit
method is expecting a tuple for its "args argument". Now I had to go to
the website to find an example, but it appears that your second argument
should have been a tuple of the arguments for your function. See
* submit*(self, func, args=(), depfuncs=(), modules=(),
callback=None, callbackargs=(), group='default', globals=None)

on page: http://www.parallelpython.com/content/view/15/30/#API

So you're passing it two function objects, and the second one is in the
place where it's expecting a tuple of arguments. Apparently you should
be calling submit multiple times, once for each function.

As for the error you get when you do that, please post the full
traceback. I suspect that the message you did post has a typo in it, or
that the error occurred when your source code was not as you show it
here. You have two variables end and end2, and only the first one is
ever declared global. func2() should get a different error when you run
it; since it tries to use a local end2 before defining it.

I recommend always testing your code with a single thread before trying
to launch more complex multithread stuff. Just call func() and func2(),
and see if they complete, and get reasonable answers.
 
Y

Yigit Turgut

I am trying to run two functions at the same time with Parallel
Processing (pp) as following ;
import pygame
import sys
import time
import math
import pp
screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
timer = pygame.time.Clock()
white = True
start = time.time()
end = time.time() - start
def test1():
   global end
   while(end<5):
     end = time.time() - start
     timer.tick(4) #FPS
     screen.fill((255,255,255) if white else (0, 0, 0))
     white = not white
     pygame.display.update()
def test2():
   global end
   while(end2<5):
     end2 = time.time() - start
     print end
ppservers = ()
if len(sys.argv)>  1:
     ncpus = int(sys.argv[1])
     # Creates jobserver with ncpus workers
     job_server = pp.Server(ncpus, ppservers=ppservers)
else:
     # Creates jobserver with automatically detected number of workers
     job_server = pp.Server(ppservers=ppservers)
print "Starting PP with", job_server.get_ncpus(), "workers"
job1 = job_server.submit(test1,test2)
result = job1()
And the output is ;
Starting PP with 2 workers
Traceback (most recent call last):
   File "fl.py", line 46, in<module>
     job1 = job_server.submit(test1,test2)
   File "/usr/lib/python2.6/site-packages/pp.py", line 402, in submit
     raise TypeError("args argument must be a tuple")
TypeError: args argument must be a tuple
When I change job1 to just to see if it will run one function only;
job1 = job_server.submit(test1)
I get an output of;
NameError: global name 'end' is not defined
end variable is defined as global but I get a NameError. Anyone has an
idea what's going on ?

First, please tell us about any external dependencies (eg. imports).
Pygame may not matter, since there are many people here using it, but
each of us has to hunt down something that does parallel processing with
an interface similar to what you're using.  For the next person, the
most likely candidate out of the dozens available is:http://www.parallelpython.com/

Your problem, as stated in the raise statement, is that the submit
method is expecting a tuple for its "args argument".  Now I had to go to
the website to find an example, but it appears that your second argument
should have been a tuple of the arguments for your function.  See
*      submit*(self, func, args=(), depfuncs=(), modules=(),
callback=None, callbackargs=(), group='default', globals=None)

on page:http://www.parallelpython.com/content/view/15/30/#API

So you're passing it two function objects, and the second one is in the
place where it's expecting a tuple of arguments.  Apparently you should
be calling submit multiple times, once for each function.

As for the error you get when you do that, please post the full
traceback. I suspect that the message you did post has a typo in it, or
that the error occurred when your source code was not as you show it
here.  You have two variables end and end2, and only the first one is
ever declared global.  func2() should get a different error when you run
it; since it tries to use a local end2 before defining it.

I recommend always testing your code with a single thread before trying
to launch more complex multithread stuff.  Just call func() and func2(),
and see if they complete, and get reasonable answers.

There are no imports other than defined on the script, which are;

import pygame
import sys
import time
import math
import pp

You are correct about trying to pass two functions and second one is
in place where a tuple of arguments supposed to be. But what if these
functions don't have any arguments ? I tested functions test1() and
test2() seperately ; they work. Once I figure out how to run these
functions simultaneously, I will add an argument to test2 and try then
on. My main goal is to simultaneously run two functions, one of them
has one argument the other doesn't. To get familiar with parallel
processing I am experimenting now without arguments and then I will
embed the code to my application. I am experimenting with the
following ;

import pygame
import sys
import time
import math
import pp

screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
timer = pygame.time.Clock()
white = True
start = time.time()
end = time.time() - start

def test1():
global end
global white
while(end<5):
end = time.time() - start
timer.tick(4) #FPS
screen.fill((255,255,255) if white else (0, 0, 0))
white = not white
pygame.display.update()

def test2():
global end
while(end<5):
end = time.time() - start
print end

ppservers = ()
job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"

job1 = job_server.submit(test1())
job2 = job_server.submit(test2())
result = job1()
result2 = job2()

print "Counting...", result2

job_server.print_stats()

test1() works as expected (job1) but test2() doesn't work and I get
the following traceback ;

Traceback (most recent call last):
File "fl.py", line 33, in <module>
job1 = job_server.submit(test1())
File "/usr/lib/python2.6/site-packages/pp.py", line 458, in submit
sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
File "/usr/lib/python2.6/site-packages/pp.py", line 629, in
__dumpsfunc
sources = [self.__get_source(func) for func in funcs]
File "/usr/lib/python2.6/site-packages/pp.py", line 696, in
__get_source
sourcelines = inspect.getsourcelines(func)[0]
File "/usr/lib/python2.6/inspect.py", line 678, in getsourcelines
lines, lnum = findsource(object)
File "/usr/lib/python2.6/inspect.py", line 519, in findsource
file = getsourcefile(object) or getfile(object)
File "/usr/lib/python2.6/inspect.py", line 441, in getsourcefile
filename = getfile(object)
File "/usr/lib/python2.6/inspect.py", line 418, in getfile
raise TypeError('arg is not a module, class, method, '
TypeError: arg is not a module, class, method, function, traceback,
frame, or code object

Error is related to test1 not having an argument. When I leave it
empty as following ;

job1 = job_server.submit(test1,())

test1 doesn't run. When I do ;

job1 = job_server.submit(test1())

Display works but I get;

TypeError: arg is not a module, class, method, function, traceback,
frame, or code object (complete traceback same as above).

And test2 doesn't work also. But when I do;

job1 = job_server.submit(test1,())
job2 = job_server.submit(test2())

I get test2 working but test1 not working. Obviously related to
argument arrangement in submit.
 
C

Chris Angelico

job1 = job_server.submit(test1,())
job2 = job_server.submit(test2())

The first of these passes test1 and an empty tuple as arguments to
submit(). The second calls test2 with no arguments, then passes its
return value to submit(), which is not what you want to do.

Chris Angelico
 
Y

Yigit Turgut

The first of these passes test1 and an empty tuple as arguments to
submit(). The second calls test2 with no arguments, then passes its
return value to submit(), which is not what you want to do.

Chris Angelico

Yes that's correct but (test1,()) doesn't do any good since it doesn't
execute the loop.

screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
timer = pygame.time.Clock()
white = True
start = time.time()
end = time.time() - start
end2= time.time() - start

def test1():
global end
global white
while(end<5):
end = time.time() - start
timer.tick(4) #FPS
screen.fill((255,255,255) if white else (0, 0, 0))
white = not white
pygame.display.update()

def test2():
global end2
while(end2<5):
end2 = time.time() - start
print end2

ppservers = ()
job_server = pp.Server(ppservers=ppservers)

job1 = job_server.submit(test1, (), globals=globals())
job2 = job_server.submit(test2, (), globals=globals())
result = job1()
result2 = job2()

print result2

job_server.print_stats()

This *supposed to* print values of 'end' and simultaneously execute
test1. Eventhough I set globals parameter and nothing seems to be
wrong this code generates the following traceback ;

Starting pp with 2 workers
An error has occured during the function execution
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ppworker.py", line 90, in run
__result = __f(*__args)
File "<string>", line 4, in test1
NameError: global name 'end' is not defined
An error has occured during the function execution
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ppworker.py", line 90, in run
__result = __f(*__args)
File "<string>", line 3, in test2
NameError: global name 'end2' is not defined

How can this be, what am I missing ?
 
D

Dave Angel

screen = pygame.display.set_mode((0, 0), pygame.FULLSCREEN)
timer = pygame.time.Clock()
white = True
start = time.time()
end = time.time() - start
end2= time.time() - start

def test1():
global end
global white
while(end<5):
end = time.time() - start
timer.tick(4) #FPS
screen.fill((255,255,255) if white else (0, 0, 0))
white = not white
pygame.display.update()

def test2():
global end2
while(end2<5):
end2 = time.time() - start
print end2

ppservers = ()
job_server = pp.Server(ppservers=ppservers)

job1 = job_server.submit(test1, (), globals=globals())
job2 = job_server.submit(test2, (), globals=globals())
result = job1()
result2 = job2()

print result2

job_server.print_stats()

This *supposed to* print values of 'end' and simultaneously execute
test1. Eventhough I set globals parameter and nothing seems to be
wrong this code generates the following traceback ;

Starting pp with 2 workers
An error has occured during the function execution
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ppworker.py", line 90, in run
__result = __f(*__args)
File "<string>", line 4, in test1
NameError: global name 'end' is not defined
An error has occured during the function execution
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ppworker.py", line 90, in run
__result = __f(*__args)
File "<string>", line 3, in test2
NameError: global name 'end2' is not defined

How can this be, what am I missing ?
I don't see anything on the http://www.parallelpython.com
<http://www.parallelpython.com/> website that indicates how it handles
globals. Remember this is creating a separate process, so it can't
literally share the globals you have. i would have expected it to
pickle them when you say globals=globals(), but I dunno. In any case, I
can't see any value in making end global with the "global" statement.
I'd move the end= line inside the function, and forget about making it
global.

The other thing you don't supply is a list of functions that might be
called by your function. See the depfuncs argument. It probably
handles all the system libraries, but I can't see how it'd be expected
to handle pygame.

With the limited information supplied by the website, I'd experiment
first with simpler things. Make two functions that are self-contained,
and try them first. No global statements, and no calls to pygame.
After that much worked, then I'd try adding arguments, and then return
values.

Then i'd try calling separate functions (declaring them in depfuncs).
And finally I'd try some 3rd party library.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,900
Latest member
Nell636132

Latest Threads

Top