Python 3.2 | WIndows 7 -- Multiprocessing and files not closing

[Isaac Gerg]

I have a function that looks like the following:

#---------------------------------
filename = 'c:\testfile.h5'
f = open(filename,'r')
data = f.read()

q = multiprocessing.Queue()
p = multiprocess.Process(target=myFunction,args=(data,q))
p.start()
result = q.get()
p.join()
q.close()

f.close()

os.remove(filename)
#---------------------------------

When I run this code, I get an error on the last line when I try to remove the file. It tells me that someone has access to the file. When I remove the queue and multiprocessing stuff, the function works fine.

What is going on here?

Thanks in advance,
Isaac

 

[Piet van Oostrum]

I have a function that looks like the following:

That doesn't look like a function

#---------------------------------
filename = 'c:\testfile.h5'

Your filename is most probably wrong. It should be something like:

filename = 'c:/testfile.h5'
filename = 'c:\\testfile.h5'
filename = r'c:\testfile.h5'

 

[Isaac Gerg]

Sorry, I am just providing pseudo code since I the code i have is quite large.

As I mentioned, the code works fine when I remove the multirpcessing stuff so the filename is not the issue (though you are right in your correction).

Someone with the same problem posted a smaller, more complete example here:

http://stackoverflow.com/questions/948119/preventing-file-handle-inheritance-in-multiprocessing-lib

None of the solutions posted work.

I have a function that looks like the following:

That doesn't look like a function

filename = 'c:\testfile.h5'

Your filename is most probably wrong. It should be something like:



filename = 'c:/testfile.h5'

filename = 'c:\\testfile.h5'

filename = r'c:\testfile.h5'

--

Piet van Oostrum <[email protected]>

WWW: http://pietvanoostrum.com/

PGP key: [8DAE142BE17999C4]

 

[Ned Batchelder]

Sorry, I am just providing pseudo code since I the code i have is quite large.

As I mentioned, the code works fine when I remove the multirpcessing stuff so the filename is not the issue (though you are right in your correction).

Someone with the same problem posted a smaller, more complete example here:

http://stackoverflow.com/questions/948119/preventing-file-handle-inheritance-in-multiprocessing-lib

None of the solutions posted work.

(BTW: it's better form to reply beneath the original text, not above it.)

None of the solutions try the obvious thing of closing the file before
spawning more processes. Would that work for you? A "with" statement
is a convenient way to do this:

with open(filename,'r') as f:
data = f.read()

The file is closed automatically when the with statement ends.

--Ned.

I have a function that looks like the following:

That doesn't look like a function

#---------------------------------
filename = 'c:\testfile.h5'

Your filename is most probably wrong. It should be something like:



filename = 'c:/testfile.h5'

filename = 'c:\\testfile.h5'

filename = r'c:\testfile.h5'

--

Piet van Oostrum <[email protected]>

WWW: http://pietvanoostrum.com/

PGP key: [8DAE142BE17999C4]

 

[Piet van Oostrum]

Sorry, I am just providing pseudo code since I the code i have is quite large.

As I mentioned, the code works fine when I remove the multirpcessing stuff so the filename is not the issue (though you are right in your correction).

Someone with the same problem posted a smaller, more complete example here:

Then you should give us real code (a minimal example) that we can try. You use myFunction in your example that isn't defined in your code. And by the way, why don't you close f just after reading? Or even better, use a with statement.

 

[Isaac Gerg]

Hi Piet,

Here is a real code example: http://stackoverflow.com/questions/948119/preventing-file-handle-inheritance-in-multiprocessing-lib

As I said before, I had provide pseudocode.

I cannot close the file after reading because it is part of a class and other threads may be calling member functions which read from the file.

Isaac

 

[Terry Reedy]

Additionally, is there a place on the web to view this conversation and
reply? Currently, I am only able to access this list through email.

news.gmane.org newsgroup gmane.lang.python.general
Look at the headers for this message.
The site also has a searchable archive.