Shell like syntax for subprocess.Popen # overloading >, <, |

[jelle]

Hi,

I use python quite a bit to couple different programs together.
Doing so has been a _lot_ easier since subprocess came around, but
would really like to be able to use the succinct shell syntax; >, <, |

That really shouldn't be too hard to wrap in a class, but so far I
didn't succeed to do so this well, since I'm facing some trouble with
operator precedence that I do not know how to overcome.

Consider the following:

A = 'inputString'
B = Process('process.exe')
C = cStringIO.StringIO() # output bucket

A > B > C

A is being piped to B and processed, but the output of B is not being
piped to C
executing A > B; B > C works as expected however.
Which is disappointing, since what I'm trying to achieve is a sugar
syntax for Popen processes, where directly sees the chain of
commands...

Any suggestions to overcome this issue are greatly appreciated!

cheers,
-jelle

-------------------------------------------------------------------------
class Process(Popen, object):
def __init__(self, commandString, wait=False):
assert isinstance(commandString, str)
cmd = commandString.split()

self.cmd = commandString
self.exe = cmd.pop(0)
## self.args = cmd

self.process = Popen(self.cmd, shell=True, stdin=PIPE,
stdout=PIPE, close_fds=False)

self.stdout = self.process.stdout
self.stderr = self.process.stderr

def __repr__(self):
return 'Process instance ( %s ) ' % (self.exe)

def __or__(self, other): # PIPE
'''
returns the output of Process A -> Process B
takes a Process instance as argument
'''
assert isinstance(other, Process), '%s\n is not a Process
instance' % (other)
print 'PIPE'
self > other


def __lt__(self, other): # STDIN
'''
takes a StringIO, file or string objectas argument
'''
print '>'
print 'STDIN'
if isinstance(other, str):
self.process.communicate(other)
else:
self.stdout, self.stderr =
self.process.communicate(other.read())
self.process.wait()

def __gt__(self, other): # STDOUT
'''
takes a StringIO, file or string object as argument
returns the result of an external process
'''
print '<'
print 'STDOUT'
assert hasattr(other, 'write') or isinstance(other, str)
if isinstance(other, str):
other += self.stdout
else:
other.write(self.stdout)

-------------------------------------------------------------------------

 

[Christos Georgiou]

Hi,

I use python quite a bit to couple different programs together.
Doing so has been a _lot_ easier since subprocess came around, but
would really like to be able to use the succinct shell syntax; >, <, |

That really shouldn't be too hard to wrap in a class, but so far I
didn't succeed to do so this well, since I'm facing some trouble with
operator precedence that I do not know how to overcome.

[snip]

Overload the __or__ special function (ie the 'pipe' operator) instead of the
__gt__ operator.

I remember I have seen such a proposition (mentioning pump, filters and
sinks) but I couldn't find it in google.groups.com --I think Aahz had
something to do with it, but ICBW.

Ah, I found it:

http://mail.python.org/pipermail/python-dev/2004-April/044205.html

I don't know why I remembered Aahz about it

Check this too:

http://groups.google.gr/group/comp....oad+pipe+syntax&rnum=1&hl=en#777efd4d3aa490ed

 

[jelle]

Hi Christos,

Thanks for your pointers there, impressive to see
-that a 12 year old thread still can make an interesting read
-you being able to remember & trace it... impressive...

Thanks for your pointers.
I think the
input > process > output
Syntax is more powerful , since it would let you build chaining
commmands in a more readable fashion.

The goal of this class would be to construct command chains such as:

input > processA | processB > ouput

Something which wouldn't be possible if only one operator is
overloaded.
I'm curious to see if its doable via overloading, since no __rgt__
methods exists...

 

[Christos Georgiou]

Hi Christos,

Thanks for your pointers there, impressive to see
-that a 12 year old thread still can make an interesting read
-you being able to remember & trace it... impressive...

Thanks for your pointers.
I think the
input > process > output
Syntax is more powerful , since it would let you build chaining
commmands in a more readable fashion.

The goal of this class would be to construct command chains such as:

input > processA | processB > ouput

Something which wouldn't be possible if only one operator is
overloaded.
I'm curious to see if its doable via overloading, since no __rgt__
methods exists...

The problem with the operators chaining is that ">" is treated differently
than "|". Check the following disassembly:
0 0 LOAD_NAME 0 (a)
3 LOAD_NAME 1 (b)
6 DUP_TOP
7 ROT_THREE
8 COMPARE_OP 0 (<)
11 JUMP_IF_FALSE 10 (to 24)
14 POP_TOP
15 LOAD_NAME 2 (c)
18 COMPARE_OP 0 (<)
21 JUMP_FORWARD 2 (to 26) 0 0 LOAD_NAME 0 (a)
3 LOAD_NAME 1 (b)
6 BINARY_OR
7 LOAD_NAME 2 (c)
10 BINARY_OR
11 RETURN_VALUE


The comparison operators include some logic in order to "do what I mean" (so
that 4<x<10 works like 4<x and x<10, but x<10 will never be evaluated if 4<x
is False), and that is why I suggested you use the "|" operator instead.

Of course, you can use the ">" operator, just don't chain it, in order to
avoid such unexpected behaviour.

 

[Serge Orlov]

Hi,

I use python quite a bit to couple different programs together.
Doing so has been a _lot_ easier since subprocess came around, but
would really like to be able to use the succinct shell syntax; >, <, |

That really shouldn't be too hard to wrap in a class, but so far I
didn't succeed to do so this well, since I'm facing some trouble with
operator precedence that I do not know how to overcome.

Consider the following:

A = 'inputString'
B = Process('process.exe')
C = cStringIO.StringIO() # output bucket

A > B > C

A is being piped to B and processed, but the output of B is not being
piped to C
executing A > B; B > C works as expected however.
Which is disappointing, since what I'm trying to achieve is a sugar
syntax for Popen processes, where directly sees the chain of
commands...

Any suggestions to overcome this issue are greatly appreciated!

How about good old function call?

A = Lines('inputString')
B = Process('process.exe')
C = Process('proc2.exe')
result = pipeline(A, B, C, Lines()) # Text result
result = pipeline(A, B, C, Bytes()) # Binary result
pipeline(A, B, C, file("somefile","wb")) # dump to binary file