Is it possible to save a running program and reload next time ?

Discussion in 'Python' started by fdu.xiaojf@gmail.com, Sep 21, 2006.

  1. Guest

    Hi,

    I have a program which will continue to run for several days. When it is
    running, I can't do anything except waiting because it takes over most
    of the CUP time.

    Is it possible that the program can save all running data to a file when
    I want it to stop, and can reload the data and continue to run from
    where it stops when the computer is free ?

    Regards,

    xiaojf
     
    , Sep 21, 2006
    #1
    1. Advertising

  2. On Thu, 21 Sep 2006 15:34:21 +0800, ""
    <> declaimed the following in comp.lang.python:

    >
    > Is it possible that the program can save all running data to a file when
    > I want it to stop, and can reload the data and continue to run from
    > where it stops when the computer is free ?
    >

    You'd have to write the code to save state, which means you'd have
    to be at a known resumable point... I suspect that, if you're nested
    into a recursive function, that will not be possible as you'd have to
    recreate the calling stack along with populating all know data.

    It should be pointed out that "chaos theory" sort of got its start
    from an attempt at doing something similar -- saving a massive data
    array for later reloading. It turns out the saved data didn't have the
    full resolution of the computations, so the restart from the saved data
    diverged rapidly from where the prior run had been heading. (As I
    recall, it was a weather simulation in the late 60s early 70s, and the
    data was saved in single-precision text format -- meaning round-off
    errors on output/input; these days it would be the equivalent of the
    losses inherent in doing a long computation in 80-bit IEEE, but saving
    intermediates [for reload] in 64-bit IEEE).

    The other option may not be available under most operating system...
    Namely a snapshot of the entire "core" of memory, so you can restore the
    machine /exactly/ to the state it had been at the time of the snapshot
    (this would probably affect everything on the machine -- all other
    programs, etc.)


    Though if someone else have more information contradicting me, I'd
    be happy to hear it... <G>
    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
     
    Dennis Lee Bieber, Sep 21, 2006
    #2
    1. Advertising

  3. Iain King Guest

    wrote:
    > Hi,
    >
    > I have a program which will continue to run for several days. When it is
    > running, I can't do anything except waiting because it takes over most
    > of the CUP time.
    >
    > Is it possible that the program can save all running data to a file when
    > I want it to stop, and can reload the data and continue to run from
    > where it stops when the computer is free ?
    >
    > Regards,
    >
    > xiaojf


    Can't you just use time.sleep to put your program to, uh, sleep? For
    example, time.sleep(10) will effectively pause your program for ten
    seconds, making it use very few CPU cycles. I don't know what
    interface you have, but assuming you can pick up a button or a key
    press, you can ask for a length of time, or just put your code into a
    holding pattern:

    def pause():
    paused = True
    while paused:
    time.sleep(1)
    if wake_up_key_pressed:
    paused = False

    or with a gui:

    paused = False

    def pause():
    global paused
    paused = True
    while paused:
    time.sleep(1)

    def onWakeUpButton(): #bind this to button
    global paused
    paused = False


    Iain
     
    Iain King, Sep 21, 2006
    #3
  4. MonkeeSage Guest

    wrote:
    > Is it possible that the program can save all running data to a file when
    > I want it to stop, and can reload the data and continue to run from
    > where it stops when the computer is free ?


    This isn't what you asked for (I have no idea how to do that), but
    given your description, perhaps a different solution would work.

    If you're using a *nix type OS (or possibly Cygwin, never tried) with a
    bash-style shell, then you probably already have job control built in.
    For example, in bash you can type <CTRL>Z to stop the current process,
    and you can see all jobs and their states and job numbers with the
    'job' command, and you can resume a stopped job with '%[job_number]'.
    So for example, if only one job is in he queue, just stop it and then
    when you're ready do %1.

    Another *nix option would be to use the nice command to set the
    schedular priority of the process so that when something with a higher
    priority needs the CPU then it gets it first rather than your nice'd
    process (something like 'nice -n 15 python your_program.py' should do
    well -- very low priority). I'm pretty sure windows has some kind of
    priority option for tasks as well, in fact, IIRC you open the task
    manager and right click on a task and can set the priority lower.

    Regards,
    Jordan
     
    MonkeeSage, Sep 21, 2006
    #4
  5. wrote:

    > I have a program which will continue to run for several days. When it is
    > running, I can't do anything except waiting because it takes over most
    > of the CUP time.
    >
    > Is it possible that the program can save all running data to a file when
    > I want it to stop, and can reload the data and continue to run from
    > where it stops when the computer is free ?


    For Linux (and other Unix like OSs), there are several "checkpointing"
    libraries available which allow programs to be saved to disk, and restarted
    later.

    If the other suggestions people have given to you (STOPping and CONTinuing
    processes), or sleep statements, or saving state, don't work investigate
    these.

    e.g. http://www.cs.wisc.edu/~zandy/ckpt/

    These programs have limitations on what can be restored (e.g. threads,
    shared memory, network connections...). I don't know which ones work with
    python.

    --
    Jeremy Sanders
    http://www.jeremysanders.net/
     
    Jeremy Sanders, Sep 21, 2006
    #5
  6. On 2006-09-21, <> wrote:
    > Is it possible that the program can save all running data to a file when
    > I want it to stop, and can reload the data and continue to run from
    > where it stops when the computer is free ?


    Well, on Irix you could use cpr
    <http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/u_man/cat1/cpr.z>
    but i don't know if your OS supports a similar feature

    --
    Roland Csaszar ----------- \\\ /// -------------- +43 316 495 2129
    Software Development ------ \\\ /// ----------- http://www.knapp.com
    KNAPP Logistics Automation - \\V// - mailto:
     
    Roland.Csaszar, Sep 21, 2006
    #6
  7. Duncan Booth Guest

    Jeremy Sanders <> wrote:

    > For Linux (and other Unix like OSs), there are several "checkpointing"
    > libraries available which allow programs to be saved to disk, and
    > restarted later.
    >

    Another option which will work on just about anything would be to run the
    program in a vmware virtual machine. Then you can suspend or resume the
    virtual machine whenever you wish.
     
    Duncan Booth, Sep 21, 2006
    #7
  8. Guest

    Re: Is it possible to save a running program and reload next time?

    Dennis Lee Bieber wrote:
    > On Thu, 21 Sep 2006 15:34:21 +0800, ""
    > <> declaimed the following in comp.lang.python:
    >
    >
    >> Is it possible that the program can save all running data to a file when
    >> I want it to stop, and can reload the data and continue to run from
    >> where it stops when the computer is free ?
    >>
    >>

    > You'd have to write the code to save state, which means you'd have
    > to be at a known resumable point... I suspect that, if you're nested
    > into a recursive function, that will not be possible as you'd have to
    > recreate the calling stack along with populating all know data.
    >
    > It should be pointed out that "chaos theory" sort of got its start
    > from an attempt at doing something similar -- saving a massive data
    > array for later reloading. It turns out the saved data didn't have the
    > full resolution of the computations, so the restart from the saved data
    > diverged rapidly from where the prior run had been heading. (As I
    > recall, it was a weather simulation in the late 60s early 70s, and the
    > data was saved in single-precision text format -- meaning round-off
    > errors on output/input; these days it would be the equivalent of the
    > losses inherent in doing a long computation in 80-bit IEEE, but saving
    > intermediates [for reload] in 64-bit IEEE).
    >
    > The other option may not be available under most operating system...
    > Namely a snapshot of the entire "core" of memory, so you can restore the
    > machine /exactly/ to the state it had been at the time of the snapshot
    > (this would probably affect everything on the machine -- all other
    > programs, etc.)
    >
    >
    > Though if someone else have more information contradicting me, I'd
    > be happy to hear it... <G>
    >

    Thank you all!

    Can objects be saved and reloaded by "Pickle" ? I have tried but no
    success.

    One of my friends tell me that VMware can save the current state of a
    virtual machine by suspending it, and can restore the identical state
    later. When a virtual machine is suspended , a file with a .vmss
    extension is created(about 20M), which contains the entire state of the
    virtual machine. When you resume the virtual machine, its state is
    restored from the .vmss file.

    So can this method be implemented with python ?

    I have another idea. When python is running, it uses about 10M memory.
    So can I just save a copy of the memory used by python and restore it
    later ?

    Regards,

    xiaojf
     
    , Sep 21, 2006
    #8
  9. Larry Bates Guest

    Re: Is it possible to save a running program and reload next time?

    wrote:
    > Hi,
    >
    > I have a program which will continue to run for several days. When it is
    > running, I can't do anything except waiting because it takes over most
    > of the CUP time.
    >
    > Is it possible that the program can save all running data to a file when
    > I want it to stop, and can reload the data and continue to run from
    > where it stops when the computer is free ?
    >
    > Regards,
    >
    > xiaojf
    >

    You can save the state of Python objects and reload them at a later time/date by
    using Zope's ZODB.

    http://www.zope.org/Wikis/ZODB/FrontPage

    -Larry Bates
     
    Larry Bates, Sep 21, 2006
    #9
  10. Larry Bates Guest

    Re: Is it possible to save a running program and reload next time?

    wrote:
    > Hi,
    >
    > I have a program which will continue to run for several days. When it is
    > running, I can't do anything except waiting because it takes over most
    > of the CUP time.
    >
    > Is it possible that the program can save all running data to a file when
    > I want it to stop, and can reload the data and continue to run from
    > where it stops when the computer is free ?
    >
    > Regards,
    >
    > xiaojf
    >

    You can save the state of Python objects and reload them at a later time/date by
    using Zope's ZODB.

    http://www.zope.org/Wikis/ZODB/FrontPage

    -Larry Bates
     
    Larry Bates, Sep 21, 2006
    #10
  11. Re: Is it possible to save a running program and reload next time?

    wrote:
    >
    > Can objects be saved and reloaded by "Pickle" ? I have tried but no
    > success.
    >

    Yes, that's the intended use of pickle/cPickle. There are examples in
    the docs:

    http://docs.python.org/lib/module-pickle.html

    What have you tried and what didn't work?

    Hans Georg
     
    Hans Georg Krauthaeuser, Sep 21, 2006
    #11
  12. Richard Tew Guest

    wrote:
    > I have a program which will continue to run for several days. When it is
    > running, I can't do anything except waiting because it takes over most
    > of the CUP time.
    >
    > Is it possible that the program can save all running data to a file when
    > I want it to stop, and can reload the data and continue to run from
    > where it stops when the computer is free ?


    Yes. Stackless Python can allow you to pickle running code and
    to save it to disk and load it and continue running it from where it
    left off. It won't of course work if there is external state because
    this will not be there, or necessarily still be relevant when the code
    resumes. So as long as you know the limitations and they suit
    your uses, it should be usable for this.

    http://www.stackless.com

    Here's an example:

    import stackless
    import cPickle
    import time

    def f():
    n = 0
    while True:
    n += 1
    stackless.schedule()

    t = stackless.tasklet(f)()

    while True:
    # Run the tasklet until it yields or dies.
    t.run()
    # Yield to the operating system.
    try:
    time.sleep(0.1)
    except KeyboardInterrupt:
    if not t.alive:
    print "Tasklet unexpectedly dead"
    break

    # Serialise the running tasklet.
    v = cPickle.dumps(t)
    # Kill the old version.
    t.kill()

    # y / enter = continue
    # anything else = exit
    # otherwise stalled
    s = raw_input("Continue? [y]:").lower()
    if s not in ("y", ""):
    break

    t = cPickle.loads(v)

    Unfortunately KeyboardInterrupt or probably any other
    exception as a way of interrupting the running code is
    problematic as it will kill the running tasklet if it happens
    there instead of during the sleep.

    Hope this helps,
    Richard.
     
    Richard Tew, Sep 21, 2006
    #12
  13. On Thu, 21 Sep 2006 21:25:03 +0800, ""
    <> declaimed the following in comp.lang.python:

    > Can objects be saved and reloaded by "Pickle" ? I have tried but no
    > success.
    >

    I've never used Pickle -- none of my uses have needed it.

    > One of my friends tell me that VMware can save the current state of a
    > virtual machine by suspending it, and can restore the identical state
    > later. When a virtual machine is suspended , a file with a .vmss
    > extension is created(about 20M), which contains the entire state of the
    > virtual machine. When you resume the virtual machine, its state is
    > restored from the .vmss file.
    >
    > So can this method be implemented with python ?
    >

    That VMSS file IS a core image as I originally mentioned. VMware is
    an OS/processor emulation system, so /IT/ can do a full (emulated)
    memory dump and restore. But even if you could create a process-specific
    image dump/restore in Python (and don't forget you need to capture
    anything that might have been put into swap file) you can't capture the
    Python program doing the capture -- for the simple reason that a fill
    image save/restore would restore the program with a program counter in
    the middle of the save image code...

    > I have another idea. When python is running, it uses about 10M memory.
    > So can I just save a copy of the memory used by python and restore it
    > later ?
    >

    It would have to be done from outside of the Python program (see
    previous paragraph) AND would have to be able to restore to the SAME
    memory addresses. VMware is making a snapshot of the entire emulated
    machine -- that means the OS and ALL running/suspended are saved as one
    chunk.
    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
     
    Dennis Lee Bieber, Sep 22, 2006
    #13
  14. Guest

    Re: Is it possible to save a running program and reload next time?

    Hans Georg Krauthaeuser wrote:
    > wrote:
    >
    >> Can objects be saved and reloaded by "Pickle" ? I have tried but no
    >> success.
    >>
    >>

    > Yes, that's the intended use of pickle/cPickle. There are examples in
    > the docs:
    >
    > http://docs.python.org/lib/module-pickle.html
    >
    > What have you tried and what didn't work?
    >
    > Hans Georg
    >

    My program is a genetic algorithm based program, so I thought I just
    need to save a generation object of the population and reload it later
    then my program could continue. But after I saved a generation object
    from within python using cPickle and reload it from another program, it
    said that some class object couldn't be found. Now I realized that a
    program cannot be suspended and reloaded by just simply dumping and
    loading using cPickle/Pickle.

    I will try stackless python later.

    Thanks a lot .

    Regards,

    xiaojf
     
    , Sep 22, 2006
    #14
  15. Re: Is it possible to save a running program and reload next time?

    wrote:
    > Hans Georg Krauthaeuser wrote:
    >> wrote:
    >>
    >>> Can objects be saved and reloaded by "Pickle" ? I have tried but no
    >>> success.
    >>>
    >>>

    >> Yes, that's the intended use of pickle/cPickle. There are examples in
    >> the docs:
    >>
    >> http://docs.python.org/lib/module-pickle.html
    >>
    >> What have you tried and what didn't work?
    >>
    >> Hans Georg
    >>

    > My program is a genetic algorithm based program, so I thought I just
    > need to save a generation object of the population and reload it later
    > then my program could continue. But after I saved a generation object
    > from within python using cPickle and reload it from another program, it
    > said that some class object couldn't be found. Now I realized that a
    > program cannot be suspended and reloaded by just simply dumping and
    > loading using cPickle/Pickle.
    >
    > I will try stackless python later.
    >
    > Thanks a lot .
    >
    > Regards,
    >
    > xiaojf
    >

    I use pickle to periodically save measured data for measurements that
    take several days to finish. Actually, I save a class instance with the
    data, the measurement methods and additional informations what method
    was called and with what parameters. If the measurement crashes due to
    some reasons, I recreate the class instance from the pickle file, call
    the appropriate method with the saved parameters. Then, the method has
    to look what data is already there and will continue to measure the rest.

    I have no idea whether that will help you in your situation.

    Anyway, I wish you all the best.

    Hans Georg
     
    Hans Georg Krauthaeuser, Sep 22, 2006
    #15
  16. Guest

    Re: Is it possible to save a running program and reload next time?

    Hans Georg Krauthaeuser wrote:
    > wrote:
    >
    >> Hans Georg Krauthaeuser wrote:
    >>
    >>> wrote:
    >>>
    >>>
    >>>> Can objects be saved and reloaded by "Pickle" ? I have tried but no
    >>>> success.
    >>>>
    >>>>
    >>>>
    >>> Yes, that's the intended use of pickle/cPickle. There are examples in
    >>> the docs:
    >>>
    >>> http://docs.python.org/lib/module-pickle.html
    >>>
    >>> What have you tried and what didn't work?
    >>>
    >>> Hans Georg
    >>>
    >>>

    >> My program is a genetic algorithm based program, so I thought I just
    >> need to save a generation object of the population and reload it later
    >> then my program could continue. But after I saved a generation object
    >> from within python using cPickle and reload it from another program, it
    >> said that some class object couldn't be found. Now I realized that a
    >> program cannot be suspended and reloaded by just simply dumping and
    >> loading using cPickle/Pickle.
    >>
    >> I will try stackless python later.
    >>
    >> Thanks a lot .
    >>
    >> Regards,
    >>
    >> xiaojf
    >>
    >>

    > I use pickle to periodically save measured data for measurements that
    > take several days to finish. Actually, I save a class instance with the
    > data, the measurement methods and additional informations what method
    > was called and with what parameters. If the measurement crashes due to
    > some reasons, I recreate the class instance from the pickle file, call
    > the appropriate method with the saved parameters. Then, the method has
    > to look what data is already there and will continue to measure the rest.
    >
    > I have no idea whether that will help you in your situation.
    >
    > Anyway, I wish you all the best.
    >
    > Hans Georg
    >

    I just realized another serious problem.

    Random values are extensively used in my program, so the program may
    probably not continue to run from an absolutely identical state where it
    stopped.

    Can the state of the random value generator be saved ?

    Regards,

    xiaojf
     
    , Sep 22, 2006
    #16
  17. Peter Otten Guest

    wrote:

    > Can the state of the random value generator be saved ?


    Yes. You can pickle random._inst or your own random.Random instance.

    Peter
     
    Peter Otten, Sep 22, 2006
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Joshua Beall
    Replies:
    26
    Views:
    221,710
    AndreiKaa
    Jun 29, 2007
  2. gen_tricomi
    Replies:
    2
    Views:
    767
    gen_tricomi
    May 23, 2006
  3. Deniz Bahar
    Replies:
    2
    Views:
    492
    Andrey Tarasevich
    Mar 9, 2005
  4. Replies:
    1
    Views:
    485
    Walter Roberson
    Aug 12, 2005
  5. Stefan Mueller
    Replies:
    4
    Views:
    309
    Stefan Mueller
    Nov 5, 2005
Loading...

Share This Page