One line command line filter

[Jon Redgrave]

It seems unreasonably hard to write simple one-line unix command line
filters in python:

eg: ls | python -c "<something> print x.upper()"

to get at sys.stdin or similar needs an import, which makes a
subsequent for-loop illegal.
python -c "import sys; for x in sys.stdin(): print x" <<- SyntaxError

Am I missing something obvious?

The best I've come up with is to use sitecustomize.py to add to
__builtin__
def stdin():
import sys
for x in sys.stdin():
if not x: return
yield x.strip()
import __builtin__
__builtin__.stdin = stdin

This allows
ls | python -c "for x in stdin(): print x.upper()"

Is there a better solution - if not is this worth a PEP?

 

[Thomas Jollans]

It seems unreasonably hard to write simple one-line unix command line
filters in python:

eg: ls | python -c "<something> print x.upper()"

to get at sys.stdin or similar needs an import, which makes a
subsequent for-loop illegal.
python -c "import sys; for x in sys.stdin(): print x" <<- SyntaxError

Am I missing something obvious?

ls | python -c "for line in __import__('sys').stdin: print (line.upper())"

 

[Jon Redgrave]

Am I missing something obvious?

ls | python -c "for line in __import__('sys').stdin: print (line.upper())"

Ah, so I am missing something - it is possible - but 'obvious'?
Do people think it should be more accessible

 

[Terry Reedy]

It seems unreasonably hard to write simple one-line unix command line
filters in python:

eg: ls | python -c "<something> print x.upper()"

to get at sys.stdin or similar needs an import, which makes a
subsequent for-loop illegal.
python -c "import sys; for x in sys.stdin(): print x"<<- SyntaxError

Am I missing something obvious?

The doc says "-c <command>
Execute the Python code in command. command can be one or more
statements separated by newlines,"

However, I have no idea how to put newlines into a command-line string.
Changing '; ' to '\n' does not work, at least not in Windows. The '\n'
is passed on to Python as 2 chars, and the '\' is seen as the
line-continuation char and the 'n' as illegal following it. Will a *nix
shell 'cook' the string and convert '\n' to a literal newline before
passing it to Python?

For *nix, I would expect the <<EOF mechanism to work. Have you tried that?

That said, Python is designed for named multiple-statement programs,
just as it is designed for named multiple-statement functions. Or it is
designed for interactive work.

The following works:
dir | python -c "while True: print(input())"

except that it finishes with an error traceback:
Traceback (most recent call last):
File "<string>", line 1, in <module>
EOFError: EOF when reading a line

Perhaps tolerable if Python is at the end of the pipe, not otherwise.

Is there a better solution - if not is this worth a PEP?

PEPs have to propose a concrete solution, preferably with some previous
discussion. I take is that your proposal would be to add another builtin
means to access stdin.

 

[Terry Reedy]

Ah, so I am missing something - it is possible - but 'obvious'?
Do people think it should be more accessible

__import__ is well-documented and is listed in the index of the builtin
functions chapter. "Direct use of __import__() is rare, except in cases
where you want to import a module whose name is only known at runtime."
could be explanded to include "or where you want to import as part of an
expression"

Every Python programmer should peruse that chapter to learn what is
available for possible future use.

 

[Steven D'Aprano]

The doc says "-c <command>
Execute the Python code in command. command can be one or more
statements separated by newlines,"

However, I have no idea how to put newlines into a command-line string.

I imagine that it depends on the shell you are using, but bash on Linux
makes it simple: double quotes "..." are like Python's triple-quoted
strings in that they can include newlines.

[steve@sylar python]$ ls f*.py | python -c "import sys

print sys.stdin.read()"

factorial.py
fetchqm.py
fib.py
fileutils.py
findsingle.py
fixascii.py
fix.py
frange.py
frequencies.py


Other shells may be different.

 

[Steven D'Aprano]

It seems unreasonably hard to write simple one-line unix command line
filters in python:

eg: ls | python -c "<something> print x.upper()"

Python is neither bash nor Perl. It is not intended to compete in the "quick
and dirty one-liner commands" stakes.

However, you might like to consider ipython, which is intended as a complete
Python shell, not just an interactive interpreter.

http://ipython.org/


[...]

The best I've come up with is to use sitecustomize.py to add to
__builtin__

If you're a system administrator who wants to replace bash one-liners at the
command line with Python one-liners, sure, why not?

If you build up a useful collection of sys admin tools or recipes, you might
like to consider sharing them. Perhaps if Python was used more by sys
admins, there might be less resistance to making it easier to compete with
bash one-liners.

 

[Terry Reedy]

The doc says "-c<command>
Execute the Python code in command. command can be one or more
statements separated by newlines,"

However, I have no idea how to put newlines into a command-line string.

I imagine that it depends on the shell you are using, but bash on Linux
makes it simple: double quotes "..." are like Python's triple-quoted
strings in that they can include newlines.

[steve@sylar python]$ ls f*.py | python -c "import sys

print sys.stdin.read()"

factorial.py
fetchqm.py

I was guessing that whoever wrote the doc could do something like that.
As far as I know, there is no way to escape a newline with Windows
cmd.exe. (Someone please tell me if I am wrong!) An "unmatched" quote is
either ignored or matched by the newline!

C:\Programs\Python32> python -c "print('haha')
haha

 

[Peter Otten]

It seems unreasonably hard to write simple one-line unix command line
filters in python:

eg: ls | python -c "<something> print x.upper()"

to get at sys.stdin or similar needs an import, which makes a
subsequent for-loop illegal.
python -c "import sys; for x in sys.stdin(): print x" <<- SyntaxError

Am I missing something obvious?

The best I've come up with is to use sitecustomize.py to add to
__builtin__
def stdin():
import sys
for x in sys.stdin():
if not x: return
yield x.strip()
import __builtin__
__builtin__.stdin = stdin

This allows
ls | python -c "for x in stdin(): print x.upper()"

Is there a better solution - if not is this worth a PEP?

$ touch alpha beta gamma omega
$ ls | python -c 'import sys; sys.stdout.writelines(s.upper() for s in
sys.stdin if s.startswith(("a", "o")))'
ALPHA
OMEGA

If you are doing this a lot, why don't you write a helper script and invoke
that?

$ ls | pyfilter.py -f '"m" in line' -s 'lines = (line + line[::-1] for line
in map(str.strip, lines))' -s'import re' -p 're.compile(r"(([a-
z])\2)").sub(lambda m: m.group(1).upper(), line)'
alphAAhpla
betAAteb
gaMMAAMMag
omegAAgemo

This relies on the convention that a single line of input is accessible as
"line" and the complete input is called "lines". Of course the same can be
done with python -c ..., -- and it is even more readable:

$ ls | python -c 'import re, sys
for line in sys.stdin:

line = line.strip()
line = line + line[::-1]
print re.compile(r"(([a-z])\2)").sub(lambda m: m.group(1).upper(), line)

'

alphAAhpla
betAAteb
gaMMAAMMag
omegAAgemo

Is there a better solution - if not is this worth a PEP?

The python interpreter could accept multiple -c arguments, but to see how
this will be received a post on python-ideas should be sufficient.

For the sake of completeness here's the script I used to produce the example
above:

$ cat pyfilter.py
#!/usr/bin/env python
import sys

def main():
import argparse
parser = argparse.ArgumentParser()
parser.add_argument(
"-s", "--setup",
action="append", default=[],
dest="setups", metavar="SETUP")
parser.add_argument(
"-f", "--filter",
action="append", default=[],
dest="filters", metavar="FILTER")
parser.add_argument("-p", "--print", dest="format")

args = parser.parse_args()
lines = sys.stdin
for setup in args.setups:
exec setup
for line in lines:
line = line.rstrip("\n")
for filter in args.filters:
if not eval(filter):
continue
if args.format:
line = eval(args.format)
try:
print line
except IOError as e:
if e.errno == 32: # broken pipe
break
raise
if __name__ == "__main__":
main()

 

[Hans Mulder]

I imagine that it depends on the shell you are using, but bash on Linux
makes it simple: double quotes "..." are like Python's triple-quoted
strings in that they can include newlines.

Single quotes in bash are more similar to triple quotes in Python, in
that some shell syntax is recognized within double quoted string, but
not within single quoted strings:

$ python -c 'import sys
print `sys.version`'
'2.7.1 (r271:86882M, Nov 30 2010, 10:35:34) \n[GCC 4.2.1 (Apple Inc.
build 5664)]'

$ python -c "import sys

print `sys.copyright`"

-bash: sys.copyright: command not found

When using single quotes, the `` are passed to Python, which interprets
them as a call to repr(). With double quotes, the shell interprets ``
as a sub-command.

Other shells may be different.

Most *nix shells are similar to bash. The ones that are different
are csh and tcsh: they require a backslash before each newline:

$ tcsh
% ls f*.py | python -c 'import sys \
? print sys.stdin.read()'
filecmp.py
fileinput.py
fnmatch.py
formatter.py
fpformat.py
fractions.py
ftplib.py
functools.py
%

Sometimes you need two backslashes, one for csh and one for Python:

% python -c 'print "spam spam spam spam spam spam spam spam " \\
? "spam spam spam spam spam"'
spam spam spam spam spam spam spam spam spam spam spam spam spam
%

Hope this helps,

-- HansM