Trying to chain processes together on a pipeline


A

Andrew Berg

I'm working on an audio/video converter script (moving from bash to
Python for some extra functionality), and part of it is chaining the
audio decoder (FFmpeg) either into SoX to change the volume and then to
the Nero AAC encoder or directly into the Nero encoder. This is the
chunk of code from my working bash script to give an idea of what I'm
trying to accomplish (it's indented because it's nested inside a while
loop and an if statement):
if [ "$process_audio" = "true" ]
then
if [ $vol == 1.0 ]
then
ffmpeg -i "${ifile_a}" -f wav - 2>$nul | neroaacenc
-ignorelength -q 0.4 -if - -of ${prefix}${zero}${ep}.m4a
else
# the pipeline-as-file option of sox fails on Windows
7, so I use the safe method since there's only one pipeline going into sox
ffmpeg -i "${ifile_a}" -f sox - 2>$nul | sox -t sox -
-t wav - vol $vol 2>$nul | neroaacenc -ignorelength -q 0.4 -if - -of
${prefix}${zero}${ep}.m4a
fi
else
echo "Audio skipped."
fi
This is pretty easy and straightforward in bash, but not so in Python.
This is what I have in Python (queue[position] points to an object I
create earlier that holds a bunch of info on what needs to be encoded -
input and output file names, command line options for the various
encoders used, and so forth), but clearly it has some problems:
try:
ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
stdout=subprocess.PIPE, stderr=os.devnull)
except WindowsError:
error_info = str(sys.exc_info()[1])
last_win_error_num = find_win_error_no(error_msg=error_info)
if last_win_error_num == '2': # Error 2 = 'The system cannot
find the file specified'
logger.critical('Could not execute ' +
queue[position].ffmpeg_exe + ': File not found.')
elif last_win_error_num == '193': # Error 193 = '%1 is not a
valid Win32 application'
logger.critical('Could not execute ' +
queue[position].ffmpeg_exe + ': It\'s not a valid Win32 application.')
break
if queue[position].vol != 1:
try:
sox_proc = subprocess.Popen(queue[position].sox_cmd,
stdin=ffmpeg_proc.stdout, stdout=subprocess.PIPE, stderr=os.devnull)
except WindowsError:
error_info = str(sys.exc_info()[1])
last_win_error_num = find_win_error_no(error_msg=error_info)
if last_win_error_num == '2': # Error 2 = 'The system
cannot find the file specified'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': File not found.')
elif last_win_error_num == '193': # Error 193 = '%1 is not
a valid Win32 application'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': It\'s not a valid Win32 application.')
break
wav_pipe = sox_proc.stdout
else:
wav_pipe = ffmpeg_proc.stdout
try:
nero_aac_proc = subprocess.Popen(queue[position].nero_aac_cmd,
stdin=wav_pipe)
except WindowsError:
error_info = str(sys.exc_info()[1])
last_win_error_num = find_win_error_no(error_msg=error_info)
if last_win_error_num == '2': # Error 2 = 'The system cannot
find the file specified'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': File not found.')
elif last_win_error_num == '193': # Error 193 = '%1 is not a
valid Win32 application'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': It\'s not a valid Win32 application.')
break

ffmpeg_proc.wait()
if queue[position].vol != 1:
sox_proc.wait()
nero_aac_proc.wait()
break
Note: those break statements are there to break out of the while loop
this is in.
Firstly, that first assignment to ffmpeg_proc raises an exception:
Traceback (most recent call last):
File "C:\Users\Bahamut\workspace\Disillusion\disillusion.py", line
288, in <module>
ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
stdout=subprocess.PIPE, stderr=os.devnull)
File "C:\Python32\lib\subprocess.py", line 700, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\Python32\lib\subprocess.py", line 861, in _get_handles
errwrite = msvcrt.get_osfhandle(stderr.fileno())
AttributeError: 'str' object has no attribute 'fileno'
I'm not really sure what it's complaining about since the exception
propagates from the msvcrt module through the subprocess module into my
program. I'm thinking it has to do my stderr assignment, but if that's
not right, I don't know what is.
Secondly, there are no Popen.stdout.close() calls because I'm not sure
where to put them.
Thirdly, I have nearly identical except WindowsError: blocks repeated -
I'm sure I can avoid this with decorators as suggested in a recent
thread, but I haven't learned decorators yet. This code isn't very
complete; I'm trying to get it to work in ideal conditions first, then
worry about how to handle failure.

Using Python 3 on Windows 7.
 
Ad

Advertisements

T

Thomas Rachel

Am 28.06.2011 07:57 schrieb Andrew Berg:
I'm working on an audio/video converter script (moving from bash to
Python for some extra functionality), and part of it is chaining the
audio decoder (FFmpeg) either into SoX to change the volume and then to
the Nero AAC encoder or directly into the Nero encoder. This is the
chunk of code from my working bash script to give an idea of what I'm
trying to accomplish (it's indented because it's nested inside a while
loop and an if statement):
if [ "$process_audio" = "true" ]
then
if [ $vol == 1.0 ]
then
ffmpeg -i "${ifile_a}" -f wav - 2>$nul | neroaacenc
-ignorelength -q 0.4 -if - -of ${prefix}${zero}${ep}.m4a
else
# the pipeline-as-file option of sox fails on Windows
7, so I use the safe method since there's only one pipeline going into sox
ffmpeg -i "${ifile_a}" -f sox - 2>$nul | sox -t sox -
-t wav - vol $vol 2>$nul | neroaacenc -ignorelength -q 0.4 -if - -of
${prefix}${zero}${ep}.m4a
fi
else
echo "Audio skipped."
fi
This is pretty easy and straightforward in bash, but not so in Python.
This is what I have in Python (queue[position] points to an object I
create earlier that holds a bunch of info on what needs to be encoded -
input and output file names, command line options for the various
encoders used, and so forth), but clearly it has some problems:
try:
ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
stdout=subprocess.PIPE, stderr=os.devnull)
except WindowsError:
error_info = str(sys.exc_info()[1])
last_win_error_num = find_win_error_no(error_msg=error_info)
if last_win_error_num == '2': # Error 2 = 'The system cannot
find the file specified'
logger.critical('Could not execute ' +
queue[position].ffmpeg_exe + ': File not found.')
elif last_win_error_num == '193': # Error 193 = '%1 is not a
valid Win32 application'
logger.critical('Could not execute ' +
queue[position].ffmpeg_exe + ': It\'s not a valid Win32 application.')
break
if queue[position].vol != 1:
try:
sox_proc = subprocess.Popen(queue[position].sox_cmd,
stdin=ffmpeg_proc.stdout, stdout=subprocess.PIPE, stderr=os.devnull)
except WindowsError:
error_info = str(sys.exc_info()[1])
last_win_error_num = find_win_error_no(error_msg=error_info)
if last_win_error_num == '2': # Error 2 = 'The system
cannot find the file specified'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': File not found.')
elif last_win_error_num == '193': # Error 193 = '%1 is not
a valid Win32 application'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': It\'s not a valid Win32 application.')
break
wav_pipe = sox_proc.stdout
else:
wav_pipe = ffmpeg_proc.stdout
try:
nero_aac_proc = subprocess.Popen(queue[position].nero_aac_cmd,
stdin=wav_pipe)
except WindowsError:
error_info = str(sys.exc_info()[1])
last_win_error_num = find_win_error_no(error_msg=error_info)
if last_win_error_num == '2': # Error 2 = 'The system cannot
find the file specified'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': File not found.')
elif last_win_error_num == '193': # Error 193 = '%1 is not a
valid Win32 application'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': It\'s not a valid Win32 application.')
break

ffmpeg_proc.wait()
if queue[position].vol != 1:
sox_proc.wait()
nero_aac_proc.wait()
break
Note: those break statements are there to break out of the while loop
this is in.
Firstly, that first assignment to ffmpeg_proc raises an exception:
Traceback (most recent call last):
File "C:\Users\Bahamut\workspace\Disillusion\disillusion.py", line
288, in<module>
ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
stdout=subprocess.PIPE, stderr=os.devnull)
File "C:\Python32\lib\subprocess.py", line 700, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\Python32\lib\subprocess.py", line 861, in _get_handles
errwrite = msvcrt.get_osfhandle(stderr.fileno())
AttributeError: 'str' object has no attribute 'fileno'
I'm not really sure what it's complaining about since the exception
propagates from the msvcrt module through the subprocess module into my
program. I'm thinking it has to do my stderr assignment, but if that's
not right, I don't know what is.
Secondly, there are no Popen.stdout.close() calls because I'm not sure
where to put them.
Thirdly, I have nearly identical except WindowsError: blocks repeated -
I'm sure I can avoid this with decorators as suggested in a recent
thread, but I haven't learned decorators yet.

There is a variety of possibilities. As you are a starter, a function is
prefecty fine, as Peter already pointed out. Then you have something like

def handle_winerr(exc):
error_info = str(exc)
last_win_error_num = find_win_error_no(error_msg=error_info)
if last_win_error_num == '2': # Error 2 = 'The system cannot find
the file specified'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': File not found.')
elif last_win_error_num == '193': # Error 193 = '%1 is not a valid
Win32 application'
logger.critical('Could not execute ' +
queue[position].sox_exe + ': It\'s not a valid Win32
application.')

try:
<do_stuff>
except WindowsError:
exc = sys.exc_info()[1]
handle_winerr(exc)
break


On the next step, you could write like this:

try:
<do_stuff>
except WindowsError, exc:
handle_winerr(exc)
break


The way you work with the exception is not the very best - instead of
parsing the stringified exception, you better would trigger on
exc.winerror (it is an integer with the error number).

Or, even better, just pas the error information contained in the exception:

def handle_winerr(exc):
logger.critical('Could not execute %s: %s' %
(queue[position].sox_exe, exc.strerror))


Next step: If you always break out of the loop when there is an error,
you could as well do the error handling outside the loop:


try:
while <whatever>:
<do_stuff>
except WindowsError, exc:
handle_winerr(exc)



In the following, I'm pointing out some more or less advanced techniques
which you also could use.

If you want to, you can create a context manager dealing with the stuff:

import contextlib

@contextlib.contextmanager
def winerr_handling():
try:
yield None
except WindowsError, exc:
handle_winerr(exc) # <-- the function from above

and then write

with winerr_handling():
while <whatever>:
<do_stuff>

And, if you absolutely want a decorator, you can do so:

def make_exc_ctx(hdlr):
from functools import wraps
@wraps(hdlr)
def wrapper():
try:
yield None
except WindowsError, exc:
hdlr(exc) # <-- the function from above


and then

@make_exc_ctx
def winerr_handling(exc):
logger.critical('Could not execute %s: %s' %
(queue[position].sox_exe, exc.strerror))

with winerr_handling():
while <whatever>:
<do_stuff>


HTH,

Thomas
 
Ad

Advertisements

A

Andrew Berg

The way you work with the exception is not the very best - instead of
parsing the stringified exception, you better would trigger on
exc.winerror (it is an integer with the error number).

Or, even better, just pas the error information contained in the exception:

def handle_winerr(exc):
logger.critical('Could not execute %s: %s' %
(queue[position].sox_exe, exc.strerror))
I didn't see winerror and strerror in the docs before, but another look
and they are indeed documented. I brought up Windows error codes before,
and I'm surprised no one pointed this out. Thanks for that.
 

Top