Re: subprocess call is not waiting.

Discussion in 'Python' started by MRAB, Sep 13, 2012.

  1. MRAB

    MRAB Guest

    On 2012-09-13 16:34, Jean-Michel Pichavant wrote:
    > ----- Original Message -----
    >> I have a subprocess.call which tries to download a data from a remote
    >> server using HTAR. I put the call in a while loop, which tests to
    >> see if the download was successful, and if not, loops back around up
    >> to five times, just in case my internet connection has a hiccup.
    >>
    >> Subprocess.call is supposed to wait.
    >>
    >> But it doesn't work as intended. The loop quickly runs 5 times,
    >> starting a new htar command each time. After five times around, my
    >> program tells me my download failed, because the target file doesn't
    >> yet exist. But it turns out that the download is still
    >> happening---five times.
    >>
    >> When I run htar from the shell, I don't get a shell prompt again
    >> until after the download is complete. How come control is returned
    >> to python before the htar command is through?
    >>
    >> I've tried using Popen with wait and/or communicate, but no waiting
    >> ever happens. This is troublesome not only because I don't get to
    >> post process my data, but because when I run this script for
    >> multiple datasets (checking to see whether I have local copies), I
    >> quickly get a "Too many open files" error. (I began working on that
    >> by trying to use Popopen with fds_close, etc.)
    >>
    >> Should I just go back to os.system?
    >> --
    >> http://mail.python.org/mailman/listinfo/python-list
    >>

    >
    > A related subset of code would be useful.
    >
    > You can use subprocess.PIPE to redirect stdout & stderr et get them with communicate, something like:
    >
    > proc = subprocess.Popen(['whatever'], stdout=subprocess.PIPE, stdout=subprocess.PIPE)
    > stdout, stderr = proc.communicate()
    > print stdout
    > print stderr
    >
    > Just by looking at stdout and stderr, you should be able to see why htar is returning so fast.
    >
    > JM
    >
    > PS : if you see nothing wrong, is it possible that htar runs asynchronously ?
    >

    The OP says that it waits when run from the shell.
    MRAB, Sep 13, 2012
    #1
    1. Advertising

  2. MRAB

    Guest

    Thanks, guys.
    MRAB-RedHat 6 64-bit, Python 2.6.5
    JM-Here's the relevant stuff from my last try. I've also tried with subprocess.call. Just now I tried shell=True, but it made no difference.

    sticking a print(out) in there just prints a blank line in between each iteration. It's not until the 5 trials are finished that I am told: download failed, etc.

    from os.path import exists
    from subprocess import call
    from subprocess import Popen
    from shlex import split
    from time import sleep

    while (exists(file)==0) and (nTries < 5):
    a = Popen(split('htar -xvf ' + htarArgs), stdout=PIPE, stderr=PIPE)
    (out,err) = a.communicate()
    if exists(file)==0:
    nTries += 1
    sleep(0.5)

    if exists(file)==0: # now that the file should be moved
    print('download failed: ' + file)
    return 1

    I've also tried using shell=True with popopen.
    , Sep 13, 2012
    #2
    1. Advertising

  3. MRAB

    Guest

    Thanks, guys.
    MRAB-RedHat 6 64-bit, Python 2.6.5
    JM-Here's the relevant stuff from my last try. I've also tried with subprocess.call. Just now I tried shell=True, but it made no difference.

    sticking a print(out) in there just prints a blank line in between each iteration. It's not until the 5 trials are finished that I am told: download failed, etc.

    from os.path import exists
    from subprocess import call
    from subprocess import Popen
    from shlex import split
    from time import sleep

    while (exists(file)==0) and (nTries < 5):
    a = Popen(split('htar -xvf ' + htarArgs), stdout=PIPE, stderr=PIPE)
    (out,err) = a.communicate()
    if exists(file)==0:
    nTries += 1
    sleep(0.5)

    if exists(file)==0: # now that the file should be moved
    print('download failed: ' + file)
    return 1

    I've also tried using shell=True with popopen.
    , Sep 13, 2012
    #3
  4. MRAB

    Chris Rebert Guest

    On Thu, Sep 13, 2012 at 11:36 AM, <> wrote:
    > Thanks, guys.
    > MRAB-RedHat 6 64-bit, Python 2.6.5


    In your Unix shell, what does the command:
    type htar
    output?

    > JM-Here's the relevant stuff from my last try.


    If you could give a complete, self-contained example, it would assist
    us in troubleshooting your problem.

    > I've also tried with subprocess.call. Just now I tried shell=True, but it made no difference.


    It's possible that htar uses some trickery to determine whether it's
    being invoked from a terminal or by another program, and changes its
    behavior accordingly, although I could not find any evidence of that
    based on scanning its manpage.

    > sticking a print(out) in there just prints a blank line in between each iteration. It's not until the 5 trials are finished that I am told: download failed, etc.
    >
    > from os.path import exists
    > from subprocess import call
    > from subprocess import Popen
    > from shlex import split
    > from time import sleep
    >
    > while (exists(file)==0) and (nTries < 5):


    `file` is the name of a built-in type in Python; it should therefore
    not be used as a variable name.
    Also, one would normally write that as:
    while not exists(file) and nTries < 5:

    > a = Popen(split('htar -xvf ' + htarArgs), stdout=PIPE, stderr=PIPE)


    What's the value of `htarArgs`? (with any sensitive parts anonymized)

    Also, you really shouldn't use shlex.split() at run-time like that.
    Unless `htarArgs` is already quoted/escaped, you'll get bad results
    for many inputs. Use shlex.split() once at the interactive interpreter
    to figure out the general form of the tokenization, then use the
    static result in your program as a template.

    > (out,err) = a.communicate()
    > if exists(file)==0:
    > nTries += 1
    > sleep(0.5)
    >
    > if exists(file)==0: # now that the file should be moved
    > print('download failed: ' + file)
    > return 1
    >
    > I've also tried using shell=True with popopen.


    I presume you meant Popen.

    Cheers,
    Chris
    Chris Rebert, Sep 14, 2012
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. erikcw
    Replies:
    6
    Views:
    452
    Michael Palmer
    Sep 19, 2008
  2. Miles Kaufmann

    Re: Waiting for a subprocess to exit

    Miles Kaufmann, Aug 21, 2009, in forum: Python
    Replies:
    6
    Views:
    301
  3. Replies:
    17
    Views:
    388
    Terry Reedy
    Sep 19, 2013
  4. Jean-Michel Pichavant

    Re: subprocess call is not waiting.

    Jean-Michel Pichavant, Sep 13, 2012, in forum: Python
    Replies:
    0
    Views:
    175
    Jean-Michel Pichavant
    Sep 13, 2012
  5. loial

    subprocess question re waiting

    loial, Apr 8, 2013, in forum: Python
    Replies:
    4
    Views:
    97
    Dave Angel
    Apr 8, 2013
Loading...

Share This Page