A design problem

[Dan Upton]

Or: How to write Python like a Python programmer, not a Java
programmer. This will be a little long-winded...

So I just recently started picking up Python, mostly learning the new
bits I need via Google and otherwise cobbling together the functions
I've already written. It occurred to me though that one of my
programs was still probably written very much like I would in Java
(part of the reason I'm picking up Python is I'm tired of my coworkers
making fun of me for writing parsing/reformatting programs in Java).
Anyway, basically here's the problem I have:

-Fork off n copies of a program, where n is a command line parameter,
and save their PIDs. The way I've been accomplishing this is
basically:

processes=[]
for i in range(numProcs):
pid=os.fork()
if pid == 0:
# do the forking
else:
processes.append(pid)

-Every so much time (say, every second), I want to ask the OS
something about that process from under /proc/pid (this is on Linux),
including what core it's on.
while 1:
for i in processes:
file = open("/proc/"+str(i)+"/stat")

From that, one of the pieces of data I'll get is which core it's

running on, which then will prompt me to open another file.
Ultimately, I want to have n files, that are a bunch of lines:
corenum data1 data2 ...
corenum data1 data2 ...
....

and so on. The way I was going to approach it was to every time
through the loop, read the data for one of the processes, open its
file, write out to it, and close it, then do the same for the next
process, and so on. Really though I don't need to be able to look at
the data until the processes are finished, and it would be less I/O,
at the expense of memory, to just save all of the lists of data as I
go along and then dump them out to disk at the end of the Python
program's execution. I feel like Python's lists or dictionaries
should be useful here, but I'm not really sure how to apply them,
particularly in a "Python-like" way.

For anybody who made it all the way through that description any suggestions?

Thanks,
-dan

 

[Paddy]

Or: How to write Python like a Python programmer, not a Java
programmer. This will be a little long-winded...

...

and so on. The way I was going to approach it was to every time
through the loop, read the data for one of the processes, open its
file, write out to it, and close it, then do the same for the next
process, and so on. Really though I don't need to be able to look at
the data until the processes are finished, and it would be less I/O,
at the expense of memory, to just save all of the lists of data as I
go along and then dump them out to disk at the end of the Python
program's execution. I feel like Python's lists or dictionaries
should be useful here, but I'm not really sure how to apply them,
particularly in a "Python-like" way.
Thanks,
-dan

For 'reasonable' n you might want to go with something like your
original idea except don't close the files until the end:
* Open a file for each process
* loop:
* get data for all processes
* Write one record to each file
* Optionally flush the file for data integrity
* sleep for some time
* close all files.
If you flush then you get some data back if the program fails part
way. If you don't then you may get some data. If you kept all the data
in memory then on a crash, it would all be gone.
The above is close to what you were originally thinking so should be
easier for you to code and see if its performance is adequate - You
automatically take advantage of the OS and disk caches this way too.

- Paddy.