Is it possible to save a running program and reload next time ?

[fdu.xiaojf]

Hi,

I have a program which will continue to run for several days. When it is
running, I can't do anything except waiting because it takes over most
of the CUP time.

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

Regards,

xiaojf

 

[Dennis Lee Bieber]

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

You'd have to write the code to save state, which means you'd have
to be at a known resumable point... I suspect that, if you're nested
into a recursive function, that will not be possible as you'd have to
recreate the calling stack along with populating all know data.

It should be pointed out that "chaos theory" sort of got its start
from an attempt at doing something similar -- saving a massive data
array for later reloading. It turns out the saved data didn't have the
full resolution of the computations, so the restart from the saved data
diverged rapidly from where the prior run had been heading. (As I
recall, it was a weather simulation in the late 60s early 70s, and the
data was saved in single-precision text format -- meaning round-off
errors on output/input; these days it would be the equivalent of the
losses inherent in doing a long computation in 80-bit IEEE, but saving
intermediates [for reload] in 64-bit IEEE).

The other option may not be available under most operating system...
Namely a snapshot of the entire "core" of memory, so you can restore the
machine /exactly/ to the state it had been at the time of the snapshot
(this would probably affect everything on the machine -- all other
programs, etc.)


Though if someone else have more information contradicting me, I'd
be happy to hear it... <G>
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Iain King]

Hi,

I have a program which will continue to run for several days. When it is
running, I can't do anything except waiting because it takes over most
of the CUP time.

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

Regards,

xiaojf

Can't you just use time.sleep to put your program to, uh, sleep? For
example, time.sleep(10) will effectively pause your program for ten
seconds, making it use very few CPU cycles. I don't know what
interface you have, but assuming you can pick up a button or a key
press, you can ask for a length of time, or just put your code into a
holding pattern:

def pause():
paused = True
while paused:
time.sleep(1)
if wake_up_key_pressed:
paused = False

or with a gui:

paused = False

def pause():
global paused
paused = True
while paused:
time.sleep(1)

def onWakeUpButton(): #bind this to button
global paused
paused = False


Iain

 

[MonkeeSage]

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

This isn't what you asked for (I have no idea how to do that), but
given your description, perhaps a different solution would work.

If you're using a *nix type OS (or possibly Cygwin, never tried) with a
bash-style shell, then you probably already have job control built in.
For example, in bash you can type <CTRL>Z to stop the current process,
and you can see all jobs and their states and job numbers with the
'job' command, and you can resume a stopped job with '%[job_number]'.
So for example, if only one job is in he queue, just stop it and then
when you're ready do %1.

Another *nix option would be to use the nice command to set the
schedular priority of the process so that when something with a higher
priority needs the CPU then it gets it first rather than your nice'd
process (something like 'nice -n 15 python your_program.py' should do
well -- very low priority). I'm pretty sure windows has some kind of
priority option for tasks as well, in fact, IIRC you open the task
manager and right click on a task and can set the priority lower.

Regards,
Jordan

 

[Jeremy Sanders]

I have a program which will continue to run for several days. When it is
running, I can't do anything except waiting because it takes over most
of the CUP time.

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

For Linux (and other Unix like OSs), there are several "checkpointing"
libraries available which allow programs to be saved to disk, and restarted
later.

If the other suggestions people have given to you (STOPping and CONTinuing
processes), or sleep statements, or saving state, don't work investigate
these.

e.g. http://www.cs.wisc.edu/~zandy/ckpt/

These programs have limitations on what can be restored (e.g. threads,
shared memory, network connections...). I don't know which ones work with
python.

 

[Roland.Csaszar]

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

Well, on Irix you could use cpr
<http://techpubs.sgi.com/library/tpl...=man&fname=/usr/share/catman/u_man/cat1/cpr.z>
but i don't know if your OS supports a similar feature

 

[Duncan Booth]

For Linux (and other Unix like OSs), there are several "checkpointing"
libraries available which allow programs to be saved to disk, and
restarted later.

Another option which will work on just about anything would be to run the
program in a vmware virtual machine. Then you can suspend or resume the
virtual machine whenever you wish.

 

[fdu.xiaojf]

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

You'd have to write the code to save state, which means you'd have
to be at a known resumable point... I suspect that, if you're nested
into a recursive function, that will not be possible as you'd have to
recreate the calling stack along with populating all know data.

It should be pointed out that "chaos theory" sort of got its start
from an attempt at doing something similar -- saving a massive data
array for later reloading. It turns out the saved data didn't have the
full resolution of the computations, so the restart from the saved data
diverged rapidly from where the prior run had been heading. (As I
recall, it was a weather simulation in the late 60s early 70s, and the
data was saved in single-precision text format -- meaning round-off
errors on output/input; these days it would be the equivalent of the
losses inherent in doing a long computation in 80-bit IEEE, but saving
intermediates [for reload] in 64-bit IEEE).

The other option may not be available under most operating system...
Namely a snapshot of the entire "core" of memory, so you can restore the
machine /exactly/ to the state it had been at the time of the snapshot
(this would probably affect everything on the machine -- all other
programs, etc.)


Though if someone else have more information contradicting me, I'd
be happy to hear it... <G>

Thank you all!

Can objects be saved and reloaded by "Pickle" ? I have tried but no
success.

One of my friends tell me that VMware can save the current state of a
virtual machine by suspending it, and can restore the identical state
later. When a virtual machine is suspended , a file with a .vmss
extension is created(about 20M), which contains the entire state of the
virtual machine. When you resume the virtual machine, its state is
restored from the .vmss file.

So can this method be implemented with python ?

I have another idea. When python is running, it uses about 10M memory.
So can I just save a copy of the memory used by python and restore it
later ?

Regards,

xiaojf

 

[Larry Bates]

Hi,

I have a program which will continue to run for several days. When it is
running, I can't do anything except waiting because it takes over most
of the CUP time.

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

Regards,

xiaojf

You can save the state of Python objects and reload them at a later time/date by
using Zope's ZODB.

http://www.zope.org/Wikis/ZODB/FrontPage

-Larry Bates

 

[Larry Bates]

Hi,

I have a program which will continue to run for several days. When it is
running, I can't do anything except waiting because it takes over most
of the CUP time.

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

Regards,

xiaojf

You can save the state of Python objects and reload them at a later time/date by
using Zope's ZODB.

http://www.zope.org/Wikis/ZODB/FrontPage

-Larry Bates

 

[Hans Georg Krauthaeuser]

Can objects be saved and reloaded by "Pickle" ? I have tried but no
success.

Yes, that's the intended use of pickle/cPickle. There are examples in
the docs:

http://docs.python.org/lib/module-pickle.html

What have you tried and what didn't work?

Hans Georg

 

[Richard Tew]

I have a program which will continue to run for several days. When it is
running, I can't do anything except waiting because it takes over most
of the CUP time.

Is it possible that the program can save all running data to a file when
I want it to stop, and can reload the data and continue to run from
where it stops when the computer is free ?

Yes. Stackless Python can allow you to pickle running code and
to save it to disk and load it and continue running it from where it
left off. It won't of course work if there is external state because
this will not be there, or necessarily still be relevant when the code
resumes. So as long as you know the limitations and they suit
your uses, it should be usable for this.

http://www.stackless.com

Here's an example:

import stackless
import cPickle
import time

def f():
n = 0
while True:
n += 1
stackless.schedule()

t = stackless.tasklet(f)()

while True:
# Run the tasklet until it yields or dies.
t.run()
# Yield to the operating system.
try:
time.sleep(0.1)
except KeyboardInterrupt:
if not t.alive:
print "Tasklet unexpectedly dead"
break

# Serialise the running tasklet.
v = cPickle.dumps(t)
# Kill the old version.
t.kill()

# y / enter = continue
# anything else = exit
# otherwise stalled
s = raw_input("Continue? [y]:").lower()
if s not in ("y", ""):
break

t = cPickle.loads(v)

Unfortunately KeyboardInterrupt or probably any other
exception as a way of interrupting the running code is
problematic as it will kill the running tasklet if it happens
there instead of during the sleep.

Hope this helps,
Richard.

 

[Dennis Lee Bieber]

Can objects be saved and reloaded by "Pickle" ? I have tried but no
success.

I've never used Pickle -- none of my uses have needed it.

One of my friends tell me that VMware can save the current state of a
virtual machine by suspending it, and can restore the identical state
later. When a virtual machine is suspended , a file with a .vmss
extension is created(about 20M), which contains the entire state of the
virtual machine. When you resume the virtual machine, its state is
restored from the .vmss file.

So can this method be implemented with python ?

That VMSS file IS a core image as I originally mentioned. VMware is
an OS/processor emulation system, so /IT/ can do a full (emulated)
memory dump and restore. But even if you could create a process-specific
image dump/restore in Python (and don't forget you need to capture
anything that might have been put into swap file) you can't capture the
Python program doing the capture -- for the simple reason that a fill
image save/restore would restore the program with a program counter in
the middle of the save image code...

I have another idea. When python is running, it uses about 10M memory.
So can I just save a copy of the memory used by python and restore it
later ?

It would have to be done from outside of the Python program (see
previous paragraph) AND would have to be able to restore to the SAME
memory addresses. VMware is making a snapshot of the entire emulated
machine -- that means the OS and ALL running/suspended are saved as one
chunk.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[fdu.xiaojf]

Yes, that's the intended use of pickle/cPickle. There are examples in
the docs:

http://docs.python.org/lib/module-pickle.html

What have you tried and what didn't work?

Hans Georg

My program is a genetic algorithm based program, so I thought I just
need to save a generation object of the population and reload it later
then my program could continue. But after I saved a generation object
from within python using cPickle and reload it from another program, it
said that some class object couldn't be found. Now I realized that a
program cannot be suspended and reloaded by just simply dumping and
loading using cPickle/Pickle.

I will try stackless python later.

Thanks a lot .

Regards,

xiaojf

 

[Hans Georg Krauthaeuser]

My program is a genetic algorithm based program, so I thought I just
need to save a generation object of the population and reload it later
then my program could continue. But after I saved a generation object
from within python using cPickle and reload it from another program, it
said that some class object couldn't be found. Now I realized that a
program cannot be suspended and reloaded by just simply dumping and
loading using cPickle/Pickle.

I will try stackless python later.

Thanks a lot .

Regards,

xiaojf

I use pickle to periodically save measured data for measurements that
take several days to finish. Actually, I save a class instance with the
data, the measurement methods and additional informations what method
was called and with what parameters. If the measurement crashes due to
some reasons, I recreate the class instance from the pickle file, call
the appropriate method with the saved parameters. Then, the method has
to look what data is already there and will continue to measure the rest.

I have no idea whether that will help you in your situation.

Anyway, I wish you all the best.

Hans Georg

 

[fdu.xiaojf]

I use pickle to periodically save measured data for measurements that
take several days to finish. Actually, I save a class instance with the
data, the measurement methods and additional informations what method
was called and with what parameters. If the measurement crashes due to
some reasons, I recreate the class instance from the pickle file, call
the appropriate method with the saved parameters. Then, the method has
to look what data is already there and will continue to measure the rest.

I have no idea whether that will help you in your situation.

Anyway, I wish you all the best.

Hans Georg

I just realized another serious problem.

Random values are extensively used in my program, so the program may
probably not continue to run from an absolutely identical state where it
stopped.

Can the state of the random value generator be saved ?

Regards,

xiaojf

 

[Peter Otten]

Can the state of the random value generator be saved ?

Yes. You can pickle random._inst or your own random.Random instance.

Peter