Trying to chain processes together on a pipeline

Discussion in 'Python' started by Andrew Berg, Jun 28, 2011.

  1. Andrew Berg

    Andrew Berg Guest

    I'm working on an audio/video converter script (moving from bash to
    Python for some extra functionality), and part of it is chaining the
    audio decoder (FFmpeg) either into SoX to change the volume and then to
    the Nero AAC encoder or directly into the Nero encoder. This is the
    chunk of code from my working bash script to give an idea of what I'm
    trying to accomplish (it's indented because it's nested inside a while
    loop and an if statement):
    > if [ "$process_audio" = "true" ]
    > then
    > if [ $vol == 1.0 ]
    > then
    > ffmpeg -i "${ifile_a}" -f wav - 2>$nul | neroaacenc
    > -ignorelength -q 0.4 -if - -of ${prefix}${zero}${ep}.m4a
    > else
    > # the pipeline-as-file option of sox fails on Windows
    > 7, so I use the safe method since there's only one pipeline going into sox
    > ffmpeg -i "${ifile_a}" -f sox - 2>$nul | sox -t sox -
    > -t wav - vol $vol 2>$nul | neroaacenc -ignorelength -q 0.4 -if - -of
    > ${prefix}${zero}${ep}.m4a
    > fi
    > else
    > echo "Audio skipped."
    > fi

    This is pretty easy and straightforward in bash, but not so in Python.
    This is what I have in Python (queue[position] points to an object I
    create earlier that holds a bunch of info on what needs to be encoded -
    input and output file names, command line options for the various
    encoders used, and so forth), but clearly it has some problems:
    > try:
    > ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
    > stdout=subprocess.PIPE, stderr=os.devnull)
    > except WindowsError:
    > error_info = str(sys.exc_info()[1])
    > last_win_error_num = find_win_error_no(error_msg=error_info)
    > if last_win_error_num == '2': # Error 2 = 'The system cannot
    > find the file specified'
    > logger.critical('Could not execute ' +
    > queue[position].ffmpeg_exe + ': File not found.')
    > elif last_win_error_num == '193': # Error 193 = '%1 is not a
    > valid Win32 application'
    > logger.critical('Could not execute ' +
    > queue[position].ffmpeg_exe + ': It\'s not a valid Win32 application.')
    > break
    > if queue[position].vol != 1:
    > try:
    > sox_proc = subprocess.Popen(queue[position].sox_cmd,
    > stdin=ffmpeg_proc.stdout, stdout=subprocess.PIPE, stderr=os.devnull)
    > except WindowsError:
    > error_info = str(sys.exc_info()[1])
    > last_win_error_num = find_win_error_no(error_msg=error_info)
    > if last_win_error_num == '2': # Error 2 = 'The system
    > cannot find the file specified'
    > logger.critical('Could not execute ' +
    > queue[position].sox_exe + ': File not found.')
    > elif last_win_error_num == '193': # Error 193 = '%1 is not
    > a valid Win32 application'
    > logger.critical('Could not execute ' +
    > queue[position].sox_exe + ': It\'s not a valid Win32 application.')
    > break
    > wav_pipe = sox_proc.stdout
    > else:
    > wav_pipe = ffmpeg_proc.stdout
    > try:
    > nero_aac_proc = subprocess.Popen(queue[position].nero_aac_cmd,
    > stdin=wav_pipe)
    > except WindowsError:
    > error_info = str(sys.exc_info()[1])
    > last_win_error_num = find_win_error_no(error_msg=error_info)
    > if last_win_error_num == '2': # Error 2 = 'The system cannot
    > find the file specified'
    > logger.critical('Could not execute ' +
    > queue[position].sox_exe + ': File not found.')
    > elif last_win_error_num == '193': # Error 193 = '%1 is not a
    > valid Win32 application'
    > logger.critical('Could not execute ' +
    > queue[position].sox_exe + ': It\'s not a valid Win32 application.')
    > break
    >
    > ffmpeg_proc.wait()
    > if queue[position].vol != 1:
    > sox_proc.wait()
    > nero_aac_proc.wait()
    > break

    Note: those break statements are there to break out of the while loop
    this is in.
    Firstly, that first assignment to ffmpeg_proc raises an exception:
    > Traceback (most recent call last):
    > File "C:\Users\Bahamut\workspace\Disillusion\disillusion.py", line
    > 288, in <module>
    > ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
    > stdout=subprocess.PIPE, stderr=os.devnull)
    > File "C:\Python32\lib\subprocess.py", line 700, in __init__
    > errread, errwrite) = self._get_handles(stdin, stdout, stderr)
    > File "C:\Python32\lib\subprocess.py", line 861, in _get_handles
    > errwrite = msvcrt.get_osfhandle(stderr.fileno())
    > AttributeError: 'str' object has no attribute 'fileno'

    I'm not really sure what it's complaining about since the exception
    propagates from the msvcrt module through the subprocess module into my
    program. I'm thinking it has to do my stderr assignment, but if that's
    not right, I don't know what is.
    Secondly, there are no Popen.stdout.close() calls because I'm not sure
    where to put them.
    Thirdly, I have nearly identical except WindowsError: blocks repeated -
    I'm sure I can avoid this with decorators as suggested in a recent
    thread, but I haven't learned decorators yet. This code isn't very
    complete; I'm trying to get it to work in ideal conditions first, then
    worry about how to handle failure.

    Using Python 3 on Windows 7.
     
    Andrew Berg, Jun 28, 2011
    #1
    1. Advertising

  2. Am 28.06.2011 07:57 schrieb Andrew Berg:
    > I'm working on an audio/video converter script (moving from bash to
    > Python for some extra functionality), and part of it is chaining the
    > audio decoder (FFmpeg) either into SoX to change the volume and then to
    > the Nero AAC encoder or directly into the Nero encoder. This is the
    > chunk of code from my working bash script to give an idea of what I'm
    > trying to accomplish (it's indented because it's nested inside a while
    > loop and an if statement):
    >> if [ "$process_audio" = "true" ]
    >> then
    >> if [ $vol == 1.0 ]
    >> then
    >> ffmpeg -i "${ifile_a}" -f wav - 2>$nul | neroaacenc
    >> -ignorelength -q 0.4 -if - -of ${prefix}${zero}${ep}.m4a
    >> else
    >> # the pipeline-as-file option of sox fails on Windows
    >> 7, so I use the safe method since there's only one pipeline going into sox
    >> ffmpeg -i "${ifile_a}" -f sox - 2>$nul | sox -t sox -
    >> -t wav - vol $vol 2>$nul | neroaacenc -ignorelength -q 0.4 -if - -of
    >> ${prefix}${zero}${ep}.m4a
    >> fi
    >> else
    >> echo "Audio skipped."
    >> fi

    > This is pretty easy and straightforward in bash, but not so in Python.
    > This is what I have in Python (queue[position] points to an object I
    > create earlier that holds a bunch of info on what needs to be encoded -
    > input and output file names, command line options for the various
    > encoders used, and so forth), but clearly it has some problems:
    >> try:
    >> ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
    >> stdout=subprocess.PIPE, stderr=os.devnull)
    >> except WindowsError:
    >> error_info = str(sys.exc_info()[1])
    >> last_win_error_num = find_win_error_no(error_msg=error_info)
    >> if last_win_error_num == '2': # Error 2 = 'The system cannot
    >> find the file specified'
    >> logger.critical('Could not execute ' +
    >> queue[position].ffmpeg_exe + ': File not found.')
    >> elif last_win_error_num == '193': # Error 193 = '%1 is not a
    >> valid Win32 application'
    >> logger.critical('Could not execute ' +
    >> queue[position].ffmpeg_exe + ': It\'s not a valid Win32 application.')
    >> break
    >> if queue[position].vol != 1:
    >> try:
    >> sox_proc = subprocess.Popen(queue[position].sox_cmd,
    >> stdin=ffmpeg_proc.stdout, stdout=subprocess.PIPE, stderr=os.devnull)
    >> except WindowsError:
    >> error_info = str(sys.exc_info()[1])
    >> last_win_error_num = find_win_error_no(error_msg=error_info)
    >> if last_win_error_num == '2': # Error 2 = 'The system
    >> cannot find the file specified'
    >> logger.critical('Could not execute ' +
    >> queue[position].sox_exe + ': File not found.')
    >> elif last_win_error_num == '193': # Error 193 = '%1 is not
    >> a valid Win32 application'
    >> logger.critical('Could not execute ' +
    >> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
    >> break
    >> wav_pipe = sox_proc.stdout
    >> else:
    >> wav_pipe = ffmpeg_proc.stdout
    >> try:
    >> nero_aac_proc = subprocess.Popen(queue[position].nero_aac_cmd,
    >> stdin=wav_pipe)
    >> except WindowsError:
    >> error_info = str(sys.exc_info()[1])
    >> last_win_error_num = find_win_error_no(error_msg=error_info)
    >> if last_win_error_num == '2': # Error 2 = 'The system cannot
    >> find the file specified'
    >> logger.critical('Could not execute ' +
    >> queue[position].sox_exe + ': File not found.')
    >> elif last_win_error_num == '193': # Error 193 = '%1 is not a
    >> valid Win32 application'
    >> logger.critical('Could not execute ' +
    >> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
    >> break
    >>
    >> ffmpeg_proc.wait()
    >> if queue[position].vol != 1:
    >> sox_proc.wait()
    >> nero_aac_proc.wait()
    >> break

    > Note: those break statements are there to break out of the while loop
    > this is in.
    > Firstly, that first assignment to ffmpeg_proc raises an exception:
    >> Traceback (most recent call last):
    >> File "C:\Users\Bahamut\workspace\Disillusion\disillusion.py", line
    >> 288, in<module>
    >> ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
    >> stdout=subprocess.PIPE, stderr=os.devnull)
    >> File "C:\Python32\lib\subprocess.py", line 700, in __init__
    >> errread, errwrite) = self._get_handles(stdin, stdout, stderr)
    >> File "C:\Python32\lib\subprocess.py", line 861, in _get_handles
    >> errwrite = msvcrt.get_osfhandle(stderr.fileno())
    >> AttributeError: 'str' object has no attribute 'fileno'

    > I'm not really sure what it's complaining about since the exception
    > propagates from the msvcrt module through the subprocess module into my
    > program. I'm thinking it has to do my stderr assignment, but if that's
    > not right, I don't know what is.
    > Secondly, there are no Popen.stdout.close() calls because I'm not sure
    > where to put them.


    > Thirdly, I have nearly identical except WindowsError: blocks repeated -
    > I'm sure I can avoid this with decorators as suggested in a recent
    > thread, but I haven't learned decorators yet.


    There is a variety of possibilities. As you are a starter, a function is
    prefecty fine, as Peter already pointed out. Then you have something like

    def handle_winerr(exc):
    error_info = str(exc)
    last_win_error_num = find_win_error_no(error_msg=error_info)
    if last_win_error_num == '2': # Error 2 = 'The system cannot find
    the file specified'
    logger.critical('Could not execute ' +
    queue[position].sox_exe + ': File not found.')
    elif last_win_error_num == '193': # Error 193 = '%1 is not a valid
    Win32 application'
    logger.critical('Could not execute ' +
    queue[position].sox_exe + ': It\'s not a valid Win32
    application.')

    try:
    <do_stuff>
    except WindowsError:
    exc = sys.exc_info()[1]
    handle_winerr(exc)
    break


    On the next step, you could write like this:

    try:
    <do_stuff>
    except WindowsError, exc:
    handle_winerr(exc)
    break


    The way you work with the exception is not the very best - instead of
    parsing the stringified exception, you better would trigger on
    exc.winerror (it is an integer with the error number).

    Or, even better, just pas the error information contained in the exception:

    def handle_winerr(exc):
    logger.critical('Could not execute %s: %s' %
    (queue[position].sox_exe, exc.strerror))


    Next step: If you always break out of the loop when there is an error,
    you could as well do the error handling outside the loop:


    try:
    while <whatever>:
    <do_stuff>
    except WindowsError, exc:
    handle_winerr(exc)



    In the following, I'm pointing out some more or less advanced techniques
    which you also could use.

    If you want to, you can create a context manager dealing with the stuff:

    import contextlib

    @contextlib.contextmanager
    def winerr_handling():
    try:
    yield None
    except WindowsError, exc:
    handle_winerr(exc) # <-- the function from above

    and then write

    with winerr_handling():
    while <whatever>:
    <do_stuff>

    And, if you absolutely want a decorator, you can do so:

    def make_exc_ctx(hdlr):
    from functools import wraps
    @wraps(hdlr)
    def wrapper():
    try:
    yield None
    except WindowsError, exc:
    hdlr(exc) # <-- the function from above


    and then

    @make_exc_ctx
    def winerr_handling(exc):
    logger.critical('Could not execute %s: %s' %
    (queue[position].sox_exe, exc.strerror))

    with winerr_handling():
    while <whatever>:
    <do_stuff>


    HTH,

    Thomas
     
    Thomas Rachel, Jun 28, 2011
    #2
    1. Advertising

  3. Andrew Berg

    Andrew Berg Guest

    On 2011.06.28 02:44 AM, Thomas Rachel wrote:
    > The way you work with the exception is not the very best - instead of
    > parsing the stringified exception, you better would trigger on
    > exc.winerror (it is an integer with the error number).
    >
    > Or, even better, just pas the error information contained in the exception:
    >
    > def handle_winerr(exc):
    > logger.critical('Could not execute %s: %s' %
    > (queue[position].sox_exe, exc.strerror))

    I didn't see winerror and strerror in the docs before, but another look
    and they are indeed documented. I brought up Windows error codes before,
    and I'm surprised no one pointed this out. Thanks for that.
     
    Andrew Berg, Jun 28, 2011
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    1
    Views:
    728
    Wendy S
    Jan 7, 2005
  2. Jeff Rodriguez
    Replies:
    23
    Views:
    1,182
    David Schwartz
    Dec 9, 2003
  3. Cramer
    Replies:
    2
    Views:
    378
    Cramer
    May 8, 2008
  4. Andrew Berg
    Replies:
    3
    Views:
    257
    Chris Rebert
    Jul 1, 2011
  5. Marc Heiler
    Replies:
    1
    Views:
    187
    Robert Klemme
    May 24, 2009
Loading...

Share This Page