piping into a python script

[Donn Ingle]

Hi,
(Gnu/Linux - Python 2.4/5)
Given these two examples:
1.
../fui.py *.py
2.
ls *.py | ./fui.py

How can I capture a list of the arguments?
I need to get all the strings (file or dir names) passed via the normal
command line and any that may come from a pipe.

There is a third case:
3.
ls *.jpg | ./fui.py *.png
Where I would be gathering strings from two places.

I am trying to write a command-line friendly tool that can be used in
traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.

I have tried:
1. pipedIn = sys.stdin.readlines()
Works fine for example 2, but example 1 goes into a 'wait for input' mode
and that's no good. Is there a way to tell when no input is coming from a
pipe at all?

2. import fileinput
for line in fileinput.input():
print (line)
But this opens each file and I don't want that.


I have seen a lot of search results that don't quite answer this angle of
the question, so I'm trying on the list.

\d

 

[Marc 'BlackJack' Rintsch]

Given these two examples:
1.
./fui.py *.py
2.
ls *.py | ./fui.py

How can I capture a list of the arguments?
I need to get all the strings (file or dir names) passed via the normal
command line and any that may come from a pipe.

There is a third case:
3.
ls *.jpg | ./fui.py *.png
Where I would be gathering strings from two places.

I am trying to write a command-line friendly tool that can be used in
traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.

I have tried:
1. pipedIn = sys.stdin.readlines()
Works fine for example 2, but example 1 goes into a 'wait for input' mode
and that's no good. Is there a way to tell when no input is coming from a
pipe at all?

Usually Linux tools that can get the data from command line or files treat
a single - as file name special with the meaning of: read from stdin.

So the interface if `fui.py` would be:

1. ./fui.py *.a
2. ls *.a | ./fui.py -
3. ls *.a | ./fui.py *.b -

Ciao,
Marc 'BlackJack' Rintsch

 

[Paddy]

Hi,
(Gnu/Linux - Python 2.4/5)
Given these two examples:
1.
./fui.py *.py
2.
ls *.py | ./fui.py

How can I capture a list of the arguments?
I need to get all the strings (file or dir names) passed via the normal
command line and any that may come from a pipe.

There is a third case:
3.
ls *.jpg | ./fui.py *.png
Where I would be gathering strings from two places.

I am trying to write a command-line friendly tool that can be used in
traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.

I have tried:
1. pipedIn = sys.stdin.readlines()
Works fine for example 2, but example 1 goes into a 'wait for input' mode
and that's no good. Is there a way to tell when no input is coming from a
pipe at all?

2. import fileinput
for line in fileinput.input():
print (line)
But this opens each file and I don't want that.

I have seen a lot of search results that don't quite answer this angle of
the question, so I'm trying on the list.

\d

Try the fileinput module.
What you describe above is pretty close to the unix 'standard' but not
quite.
if we substitute the lp command instead of ./fui, the command normally
takes a list of files to act on as its arguments, and anything piped
in goes to its stdin where it is processed if it has an argument of -
fileinput works that way but you may have problems with your:
ls *.jpg | ./fui.py *.png
Which might better be expressed as:
./fui.py `ls *.jpg` *.png
which would work for ls and a python program using the fileinput
module.

- Paddy.

 

[Paddy]

Usually Linux tools that can get the data from command line or files treat
a single - as file name special with the meaning of: read from stdin.

So the interface if `fui.py` would be:

1. ./fui.py *.a
2. ls *.a | ./fui.py -
3. ls *.a | ./fui.py *.b -

Ciao,
Marc 'BlackJack' Rintsch

If X.a X.b Y.a Y.b are all files whose contents are to be processed
then
To process all files:
./fui.py *.a *.b
Or:
./fui.py `ls *.a *.b`
To process one file from a pipe unix usually does:
cat X.a | ./fui.py -
To get the filenames from stdin would usually need a command line
switch telling fui.py to read a file *list* from stdin. For verilog
simulators for example you have the -f switch that says insert further
command line arguments from the file name in the next argument, so you
could do:
ls *.a | ./fui.py -f - *.b
For equivalent functionality to my first example.

- Paddy.

 

[Donn Ingle]

Try the fileinput module.
I did give the fileinput module a go, but I can't find much info on it and
the help is ... well, it's python help

in goes to its stdin where it is processed if it has an argument of -
fileinput works that way

Okay, I did think of the dash, but did not know how to handle it. Is it a
bash thing or will that dash get passed into the args? (I am using getopt
to parse the options and args)

which would work for ls and a python program using the fileinput
module.

Any examples of fileinput (that do not open each file) would be great!
(I'll go searching now anyway)

Thanks,
\d

 

[Donn Ingle]

ls *.a | ./fui.py -f - *.b

To be sure I grok this: I am seeing the single dash as a placeholder for
where all the piped filenames will go, so *.b happens after *.a has been
expanded and they all get fed to -f, right?

I'm also guessing you mean that I should detect the single dash and then go
look for stdin at that point. How do I detect a lack of stdin?

Thanks,
\d

 

[Paddy]

I did give the fileinput module a go, but I can't find much info on it and
the help is ... well, it's python help

Try http://effbot.org/librarybook/fileinput.htm

Okay, I did think of the dash, but did not know how to handle it. Is it a
bash thing or will that dash get passed into the args? (I am using getopt
to parse the options and args)

- gets passed in and fileinput handles it.

Any examples of fileinput (that do not open each file) would be great!
(I'll go searching now anyway)

fileinput is set to process each file a line at a time unfortunately.

Thanks,

Your welcome

- Paddy.

 

[Donn Ingle]

fileinput is set to process each file a line at a time unfortunately.

Wow. So there seems to be no solution to my OP. I'm amazed, I would have
thought a simple list of strings, one from stdin and one from the args,
would be easy to get.

I *really* don't want to open each file, that would be insane.

Perhaps I shall have to forgo the stdin stuff then, after all.

\d

 

[Donn]

wget -i -

it doesn't do anything, just waits for your input. Your applications
probably should behave the same.

Okay, that works for me.

It doesn't seem to me that -f parameter is necessary for your
application.

Yes and no, I have another option that needs to take a variable number of
args.

It should treat all the arguments as the filenames,
shouldn't it? And when one of the filenames is -, just try to read
stdin.

I have tested getopt and it strips the lone '-' out. I can get it from
sys.argv, but then I am really doing more parsing than I want to. It's a
tricky job this. I think I will look in sys.argv, if I find a single dash the
I will replace that element in the list with whatever comes from stdin. Then
I'll pass all of it to getopt.

Thanks for the help.
\d


--
When you allow legends to rule your life, your world is based on fiction
-- Segio Aragones (Groo the Wanderer Number 99)

Fonty Python and other dev news at:
http://otherwiseingle.blogspot.com/

 

[Hexamorph]

Wow. So there seems to be no solution to my OP. I'm amazed, I would have
thought a simple list of strings, one from stdin and one from the args,
would be easy to get.

I *really* don't want to open each file, that would be insane.

Perhaps I shall have to forgo the stdin stuff then, after all.

Hi!

I'm not sure if I completely get what you want, but what's about this:

#!/usr/bin/python

import sys

filelist = []
with_stdin=0

if len(sys.argv) > 1:
for file in sys.argv[1:]:
if file == "-":
with_stdin=1
continue
filelist.append(file)
else:
with_stdin=1

if with_stdin:
for file in sys.stdin:
filelist.append(file)


for file in filelist:
print "Processing file: %s" % file



It's a bit clumsy, but seems to do what I guess you want.


HTH

 

[Reedick, Andrew]

-----Original Message-----

From: [email protected] [mailtoython-
[email protected]] On Behalf Of Donn
Sent: Thursday, January 24, 2008 12:03 PM
To: Micha³ Bentkowski
Cc: (e-mail address removed)
Subject: Re: piping into a python script

I have tested getopt and it strips the lone '-' out. I can get it from

Try 'foo.py -- -'. The '--' normally tells the parser to stop parsing args. Ex: date > -foo.txt; rm -foo.txt; rm -- -foo.txt


I think this will tell you if stdin is being piped in or not:
import sys
import os
print os.isatty(sys.stdin.fileno())

D:\>type a.txt | python a.py
False

D:\>python a.py
True


Also if you're lazy, look at the StringIO class:

if options.filelist is None and len(args) < 1: # read from stdin
f = sys.stdin
elif options.filelist is not None and len(args) < 1: # read filenames from file
f = open(options.filelist, 'r')
elif options.filelist is None and len(args) > 0: # filenames on command line
f = StringIO.StringIO('\n'.join(args))
else: ## Thanks for playing.
parser.print_help()
exit(1)

if f:
for filename in f:

-- Segio Aragones (Groo the Wanderer Number 99)

Ah yes, Groo. Ever wonder who would win if Groo and Forrest Gump fought each other?

 

[Donn]

Thanks for the tips, I'll decode and try 'em all out.

Ah yes, Groo. Ever wonder who would win if Groo and Forrest Gump fought
each other?

Heh I reckon they'd both die laughing. Be fun to watch -- if anyone else
survived!

\d

--
"A computer without Windows is like chocolate cake without mustard."
-- Anonymous Coward /.

Fonty Python and other dev news at:
http://otherwiseingle.blogspot.com/

 

[Nick Craig-Wood]

Usually Linux tools that can get the data from command line or files treat
a single - as file name special with the meaning of: read from stdin.

So the interface if `fui.py` would be:

1. ./fui.py *.a
2. ls *.a | ./fui.py -
3. ls *.a | ./fui.py *.b -

Did anyone mention the (standard library) fileinput module? (I missed
the start of this thread.)

http://docs.python.org/lib/module-fileinput.html

11.2 fileinput -- Iterate over lines from multiple input streams

This module implements a helper class and functions to quickly write a
loop over standard input or a list of files.

The typical use is:

import fileinput
for line in fileinput.input():
process(line)

This iterates over the lines of all files listed in sys.argv[1:],
defaulting to sys.stdin if the list is empty. If a filename is '-', it
is also replaced by sys.stdin. To specify an alternative list of
filenames, pass it as the first argument to input(). A single file
name is also allowed.

 

[Donn]

Andrew,
Thanks for your tips. I managed to get a working script going. I am sure there
will be stdin 'issues' to come, but I hope not.

If anyone wants to have a look, it's on the cheese shop at:
http://pypi.python.org/pypi/fui

\d
--
"You know, I've gone to a lot of psychics, and they've told me a lot of
different things, but not one of them has ever told me 'You are an undercover
policewoman here to arrest me.'"
-- New York City undercover policewoman

Fonty Python and other dev news at:
http://otherwiseingle.blogspot.com/

 

[Donn Ingle]

It's a bit clumsy, but seems to do what I guess you want.

Hey, thanks for that! I will have a go.

\d

 

[Donn Ingle]

This iterates over the lines of all files listed in sys.argv[1:],
defaulting to sys.stdin if the list is empty. If a filename is '-', it
is also replaced by sys.stdin. To specify an alternative list of
filenames, pass it as the first argument to input(). A single file
name is also allowed.

Yeah it has been discussed. It seems the one problem with it is that it
opens each file. I only want the filenames.

Anyway, this has more-or-less been solved now.

Thanks,
\d

 

[anonymous]

I'm not sure I understane the question but my contribution is :

import sys
names = sys.argv[1:]

line = 'x'
while line:
line = sys.stdin.readline().strip()
if line: names.append (line)

print "names=", names

Called using:
ls | stdtest.py arg1 arg2 arg3

Does this help?

Andy