piping into a python script

Discussion in 'Python' started by Donn Ingle, Jan 24, 2008.

  1. Donn Ingle

    Donn Ingle Guest

    Hi,
    (Gnu/Linux - Python 2.4/5)
    Given these two examples:
    1.
    ../fui.py *.py
    2.
    ls *.py | ./fui.py

    How can I capture a list of the arguments?
    I need to get all the strings (file or dir names) passed via the normal
    command line and any that may come from a pipe.

    There is a third case:
    3.
    ls *.jpg | ./fui.py *.png
    Where I would be gathering strings from two places.

    I am trying to write a command-line friendly tool that can be used in
    traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.

    I have tried:
    1. pipedIn = sys.stdin.readlines()
    Works fine for example 2, but example 1 goes into a 'wait for input' mode
    and that's no good. Is there a way to tell when no input is coming from a
    pipe at all?

    2. import fileinput
    for line in fileinput.input():
    print (line)
    But this opens each file and I don't want that.


    I have seen a lot of search results that don't quite answer this angle of
    the question, so I'm trying on the list.

    \d
    Donn Ingle, Jan 24, 2008
    #1
    1. Advertising

  2. On Thu, 24 Jan 2008 17:17:25 +0200, Donn Ingle wrote:

    > Given these two examples:
    > 1.
    > ./fui.py *.py
    > 2.
    > ls *.py | ./fui.py
    >
    > How can I capture a list of the arguments?
    > I need to get all the strings (file or dir names) passed via the normal
    > command line and any that may come from a pipe.
    >
    > There is a third case:
    > 3.
    > ls *.jpg | ./fui.py *.png
    > Where I would be gathering strings from two places.
    >
    > I am trying to write a command-line friendly tool that can be used in
    > traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.
    >
    > I have tried:
    > 1. pipedIn = sys.stdin.readlines()
    > Works fine for example 2, but example 1 goes into a 'wait for input' mode
    > and that's no good. Is there a way to tell when no input is coming from a
    > pipe at all?


    Usually Linux tools that can get the data from command line or files treat
    a single - as file name special with the meaning of: read from stdin.

    So the interface if `fui.py` would be:

    1. ./fui.py *.a
    2. ls *.a | ./fui.py -
    3. ls *.a | ./fui.py *.b -

    Ciao,
    Marc 'BlackJack' Rintsch
    Marc 'BlackJack' Rintsch, Jan 24, 2008
    #2
    1. Advertising

  3. Donn Ingle

    Paddy Guest

    On Jan 24, 3:17 pm, Donn Ingle <> wrote:
    > Hi,
    > (Gnu/Linux - Python 2.4/5)
    > Given these two examples:
    > 1.
    > ./fui.py *.py
    > 2.
    > ls *.py | ./fui.py
    >
    > How can I capture a list of the arguments?
    > I need to get all the strings (file or dir names) passed via the normal
    > command line and any that may come from a pipe.
    >
    > There is a third case:
    > 3.
    > ls *.jpg | ./fui.py *.png
    > Where I would be gathering strings from two places.
    >
    > I am trying to write a command-line friendly tool that can be used in
    > traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.
    >
    > I have tried:
    > 1. pipedIn = sys.stdin.readlines()
    > Works fine for example 2, but example 1 goes into a 'wait for input' mode
    > and that's no good. Is there a way to tell when no input is coming from a
    > pipe at all?
    >
    > 2. import fileinput
    > for line in fileinput.input():
    > print (line)
    > But this opens each file and I don't want that.
    >
    > I have seen a lot of search results that don't quite answer this angle of
    > the question, so I'm trying on the list.
    >
    > \d


    Try the fileinput module.
    What you describe above is pretty close to the unix 'standard' but not
    quite.
    if we substitute the lp command instead of ./fui, the command normally
    takes a list of files to act on as its arguments, and anything piped
    in goes to its stdin where it is processed if it has an argument of -
    fileinput works that way but you may have problems with your:
    ls *.jpg | ./fui.py *.png
    Which might better be expressed as:
    ./fui.py `ls *.jpg` *.png
    which would work for ls and a python program using the fileinput
    module.

    - Paddy.
    Paddy, Jan 24, 2008
    #3
  4. Donn Ingle

    Paddy Guest

    On Jan 24, 3:25 pm, Marc 'BlackJack' Rintsch <> wrote:
    > On Thu, 24 Jan 2008 17:17:25 +0200, Donn Ingle wrote:
    > > Given these two examples:
    > > 1.
    > > ./fui.py *.py
    > > 2.
    > > ls *.py | ./fui.py

    >
    > > How can I capture a list of the arguments?
    > > I need to get all the strings (file or dir names) passed via the normal
    > > command line and any that may come from a pipe.

    >
    > > There is a third case:
    > > 3.
    > > ls *.jpg | ./fui.py *.png
    > > Where I would be gathering strings from two places.

    >
    > > I am trying to write a command-line friendly tool that can be used in
    > > traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.

    >
    > > I have tried:
    > > 1. pipedIn = sys.stdin.readlines()
    > > Works fine for example 2, but example 1 goes into a 'wait for input' mode
    > > and that's no good. Is there a way to tell when no input is coming from a
    > > pipe at all?

    >
    > Usually Linux tools that can get the data from command line or files treat
    > a single - as file name special with the meaning of: read from stdin.
    >
    > So the interface if `fui.py` would be:
    >
    > 1. ./fui.py *.a
    > 2. ls *.a | ./fui.py -
    > 3. ls *.a | ./fui.py *.b -
    >
    > Ciao,
    > Marc 'BlackJack' Rintsch


    If X.a X.b Y.a Y.b are all files whose contents are to be processed
    then
    To process all files:
    ./fui.py *.a *.b
    Or:
    ./fui.py `ls *.a *.b`
    To process one file from a pipe unix usually does:
    cat X.a | ./fui.py -
    To get the filenames from stdin would usually need a command line
    switch telling fui.py to read a file *list* from stdin. For verilog
    simulators for example you have the -f switch that says insert further
    command line arguments from the file name in the next argument, so you
    could do:
    ls *.a | ./fui.py -f - *.b
    For equivalent functionality to my first example.

    - Paddy.
    Paddy, Jan 24, 2008
    #4
  5. Donn Ingle

    Donn Ingle Guest

    > Try the fileinput module.
    I did give the fileinput module a go, but I can't find much info on it and
    the help is ... well, it's python help ;)

    > in goes to its stdin where it is processed if it has an argument of -
    > fileinput works that way

    Okay, I did think of the dash, but did not know how to handle it. Is it a
    bash thing or will that dash get passed into the args? (I am using getopt
    to parse the options and args)

    > which would work for ls and a python program using the fileinput
    > module.

    Any examples of fileinput (that do not open each file) would be great!
    (I'll go searching now anyway)

    Thanks,
    \d
    Donn Ingle, Jan 24, 2008
    #5
  6. Donn Ingle

    Donn Ingle Guest

    Paddy wrote:
    > ls *.a | ./fui.py -f - *.b

    To be sure I grok this: I am seeing the single dash as a placeholder for
    where all the piped filenames will go, so *.b happens after *.a has been
    expanded and they all get fed to -f, right?

    I'm also guessing you mean that I should detect the single dash and then go
    look for stdin at that point. How do I detect a lack of stdin?

    Thanks,
    \d
    Donn Ingle, Jan 24, 2008
    #6
  7. Donn Ingle

    Paddy Guest

    On Jan 24, 4:02 pm, Donn Ingle <> wrote:
    > > Try the fileinput module.

    >
    > I did give the fileinput module a go, but I can't find much info on it and
    > the help is ... well, it's python help ;)


    Try http://effbot.org/librarybook/fileinput.htm

    >
    > > in goes to its stdin where it is processed if it has an argument of -
    > > fileinput works that way

    >
    > Okay, I did think of the dash, but did not know how to handle it. Is it a
    > bash thing or will that dash get passed into the args? (I am using getopt
    > to parse the options and args)


    - gets passed in and fileinput handles it.

    >
    > > which would work for ls and a python program using the fileinput
    > > module.

    >
    > Any examples of fileinput (that do not open each file) would be great!
    > (I'll go searching now anyway)

    fileinput is set to process each file a line at a time unfortunately.

    >
    > Thanks,


    Your welcome :)

    - Paddy.
    Paddy, Jan 24, 2008
    #7
  8. Donn Ingle

    Donn Ingle Guest

    Paddy wrote:
    > fileinput is set to process each file a line at a time unfortunately.

    Wow. So there seems to be no solution to my OP. I'm amazed, I would have
    thought a simple list of strings, one from stdin and one from the args,
    would be easy to get.

    I *really* don't want to open each file, that would be insane.

    Perhaps I shall have to forgo the stdin stuff then, after all.

    \d
    Donn Ingle, Jan 24, 2008
    #8
  9. Donn Ingle

    Donn Guest

    > wget -i -
    > it doesn't do anything, just waits for your input. Your applications
    > probably should behave the same.

    Okay, that works for me.

    > Paddy wrote:
    > > ls *.a | ./fui.py -f - *.b

    > It doesn't seem to me that -f parameter is necessary for your
    > application.

    Yes and no, I have another option that needs to take a variable number of
    args.

    > It should treat all the arguments as the filenames,
    > shouldn't it? And when one of the filenames is -, just try to read
    > stdin.

    I have tested getopt and it strips the lone '-' out. I can get it from
    sys.argv, but then I am really doing more parsing than I want to. It's a
    tricky job this. I think I will look in sys.argv, if I find a single dash the
    I will replace that element in the list with whatever comes from stdin. Then
    I'll pass all of it to getopt.

    Thanks for the help.
    \d


    --
    When you allow legends to rule your life, your world is based on fiction
    -- Segio Aragones (Groo the Wanderer Number 99)

    Fonty Python and other dev news at:
    http://otherwiseingle.blogspot.com/
    Donn, Jan 24, 2008
    #9
  10. Donn Ingle

    Hexamorph Guest

    Donn Ingle wrote:
    > Paddy wrote:
    >> fileinput is set to process each file a line at a time unfortunately.

    > Wow. So there seems to be no solution to my OP. I'm amazed, I would have
    > thought a simple list of strings, one from stdin and one from the args,
    > would be easy to get.
    >
    > I *really* don't want to open each file, that would be insane.
    >
    > Perhaps I shall have to forgo the stdin stuff then, after all.
    >


    Hi!

    I'm not sure if I completely get what you want, but what's about this:

    #!/usr/bin/python

    import sys

    filelist = []
    with_stdin=0

    if len(sys.argv) > 1:
    for file in sys.argv[1:]:
    if file == "-":
    with_stdin=1
    continue
    filelist.append(file)
    else:
    with_stdin=1

    if with_stdin:
    for file in sys.stdin:
    filelist.append(file)


    for file in filelist:
    print "Processing file: %s" % file



    It's a bit clumsy, but seems to do what I guess you want.


    HTH
    Hexamorph, Jan 24, 2008
    #10
  11. > -----Original Message-----
    > From: python-list-bounces+jr9445= [mailto:python-
    > list-bounces+jr9445=] On Behalf Of Donn
    > Sent: Thursday, January 24, 2008 12:03 PM
    > To: Micha³ Bentkowski
    > Cc:
    > Subject: Re: piping into a python script
    >
    > I have tested getopt and it strips the lone '-' out. I can get it from


    Try 'foo.py -- -'. The '--' normally tells the parser to stop parsing args. Ex: date > -foo.txt; rm -foo.txt; rm -- -foo.txt


    I think this will tell you if stdin is being piped in or not:
    import sys
    import os
    print os.isatty(sys.stdin.fileno())

    D:\>type a.txt | python a.py
    False

    D:\>python a.py
    True


    Also if you're lazy, look at the StringIO class:

    if options.filelist is None and len(args) < 1: # read from stdin
    f = sys.stdin
    elif options.filelist is not None and len(args) < 1: # read filenames from file
    f = open(options.filelist, 'r')
    elif options.filelist is None and len(args) > 0: # filenames on command line
    f = StringIO.StringIO('\n'.join(args))
    else: ## Thanks for playing.
    parser.print_help()
    exit(1)

    if f:
    for filename in f:



    > -- Segio Aragones (Groo the Wanderer Number 99)


    Ah yes, Groo. Ever wonder who would win if Groo and Forrest Gump fought each other?
    Reedick, Andrew, Jan 24, 2008
    #11
  12. Donn Ingle

    Donn Guest

    Thanks for the tips, I'll decode and try 'em all out.

    > Ah yes, Groo. Ever wonder who would win if Groo and Forrest Gump fought
    > each other?

    Heh ;) I reckon they'd both die laughing. Be fun to watch -- if anyone else
    survived!

    \d

    --
    "A computer without Windows is like chocolate cake without mustard."
    -- Anonymous Coward /.

    Fonty Python and other dev news at:
    http://otherwiseingle.blogspot.com/
    Donn, Jan 24, 2008
    #12
  13. Marc 'BlackJack' Rintsch <> wrote:
    > On Thu, 24 Jan 2008 17:17:25 +0200, Donn Ingle wrote:
    >
    > > Given these two examples:
    > > 1.
    > > ./fui.py *.py
    > > 2.
    > > ls *.py | ./fui.py
    > >
    > > How can I capture a list of the arguments?
    > > I need to get all the strings (file or dir names) passed via the normal
    > > command line and any that may come from a pipe.
    > >
    > > There is a third case:
    > > 3.
    > > ls *.jpg | ./fui.py *.png
    > > Where I would be gathering strings from two places.
    > >
    > > I am trying to write a command-line friendly tool that can be used in
    > > traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.
    > >
    > > I have tried:
    > > 1. pipedIn = sys.stdin.readlines()
    > > Works fine for example 2, but example 1 goes into a 'wait for input' mode
    > > and that's no good. Is there a way to tell when no input is coming from a
    > > pipe at all?

    >
    > Usually Linux tools that can get the data from command line or files treat
    > a single - as file name special with the meaning of: read from stdin.
    >
    > So the interface if `fui.py` would be:
    >
    > 1. ./fui.py *.a
    > 2. ls *.a | ./fui.py -
    > 3. ls *.a | ./fui.py *.b -


    Did anyone mention the (standard library) fileinput module? (I missed
    the start of this thread.)

    http://docs.python.org/lib/module-fileinput.html

    11.2 fileinput -- Iterate over lines from multiple input streams

    This module implements a helper class and functions to quickly write a
    loop over standard input or a list of files.

    The typical use is:

    import fileinput
    for line in fileinput.input():
    process(line)

    This iterates over the lines of all files listed in sys.argv[1:],
    defaulting to sys.stdin if the list is empty. If a filename is '-', it
    is also replaced by sys.stdin. To specify an alternative list of
    filenames, pass it as the first argument to input(). A single file
    name is also allowed.

    --
    Nick Craig-Wood <> -- http://www.craig-wood.com/nick
    Nick Craig-Wood, Jan 25, 2008
    #13
  14. Donn Ingle

    Donn Guest

    Andrew,
    Thanks for your tips. I managed to get a working script going. I am sure there
    will be stdin 'issues' to come, but I hope not.

    If anyone wants to have a look, it's on the cheese shop at:
    http://pypi.python.org/pypi/fui

    \d
    --
    "You know, I've gone to a lot of psychics, and they've told me a lot of
    different things, but not one of them has ever told me 'You are an undercover
    policewoman here to arrest me.'"
    -- New York City undercover policewoman

    Fonty Python and other dev news at:
    http://otherwiseingle.blogspot.com/
    Donn, Jan 25, 2008
    #14
  15. Donn Ingle

    Donn Ingle Guest

    Hexamorph wrote:

    > It's a bit clumsy, but seems to do what I guess you want.

    Hey, thanks for that! I will have a go.

    \d
    Donn Ingle, Jan 25, 2008
    #15
  16. Donn Ingle

    Donn Ingle Guest

    Nick Craig-Wood wrote:

    > This iterates over the lines of all files listed in sys.argv[1:],
    > defaulting to sys.stdin if the list is empty. If a filename is '-', it
    > is also replaced by sys.stdin. To specify an alternative list of
    > filenames, pass it as the first argument to input(). A single file
    > name is also allowed.

    Yeah it has been discussed. It seems the one problem with it is that it
    opens each file. I only want the filenames.

    Anyway, this has more-or-less been solved now.

    Thanks,
    \d
    Donn Ingle, Jan 25, 2008
    #16
  17. Donn Ingle

    anonymous Guest

    I'm not sure I understane the question but my contribution is :

    import sys
    names = sys.argv[1:]

    line = 'x'
    while line:
    line = sys.stdin.readline().strip()
    if line: names.append (line)

    print "names=", names

    Called using:
    ls | stdtest.py arg1 arg2 arg3

    Does this help?

    Andy
    anonymous, Feb 1, 2008
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Christian Long
    Replies:
    3
    Views:
    10,230
    Duncan Booth
    Jun 26, 2003
  2. Matt Price

    piping a stream to a python script

    Matt Price, Oct 27, 2004, in forum: Python
    Replies:
    0
    Views:
    275
    Matt Price
    Oct 27, 2004
  3. Matt Price

    piping a stream to a python script

    Matt Price, Oct 27, 2004, in forum: Python
    Replies:
    2
    Views:
    325
    Matt Price
    Oct 28, 2004
  4. runes
    Replies:
    5
    Views:
    503
    runes
    Apr 16, 2005
  5. mhenry1384
    Replies:
    6
    Views:
    907
    chris
    Aug 17, 2005
Loading...

Share This Page