how to determine if pipe is given

[greg]

To retrieve piped input to my program I can use something like
ARGF.readlines

However, if there is no piped data, this will wait for input from the
terminal- how can I avoid this?

Thanks,
Greg Weber

 

[Vincent Fourmond]

To retrieve piped input to my program I can use something like
ARGF.readlines

However, if there is no piped data, this will wait for input from the
terminal- how can I avoid this?

You can check if $stdin is a terminal with

$stdin.tty?

Vince

 

[ara.t.howard]

To retrieve piped input to my program I can use something like
ARGF.readlines

However, if there is no piped data, this will wait for input from the
terminal- how can I avoid this?

you can do something like

stdin =
if STDIN.tty?
''
else
STDIN.read
end

but that is an extremely bad idea because running the program under cron, or
even using something like

program &

will cause it to hang

the best/safest approach is to __always__ construct the command-line to give
clear notice that stdin will be following. the most common way to do this is
to pass '-' on the command line. for example


program -
program --verbosity=4 -

etc. then, the code is simple and always correct:

# parse options first

stdin = ARGV.delete '-'

if stdin
buf = STDIN.read
# ...
else
# ...
end


regards.


-a

 

[greg]

thanks, a

This kind of sucks though.

apparently in perl, if there is a piped input, '-' will show up
automatically in ARGV. I think I will propose this change to Ruby.

 

[ara.t.howard]

thanks, a

This kind of sucks though.

why? is this hard?

stdin = ARGV.delete '-'

apparently in perl, if there is a piped input, '-' will show up
automatically in ARGV. I think I will propose this change to Ruby.

eeeks! that's is pure __evil__!

consider i have many, many programs which do things like

convert infile outfile
convert - outfile # infile on stdin
convert infile - # outfile on stdout
convert - - # infile on stdin, outfile on stdout
convert --infiles=- # list of infiles on stdin, auto-name outfiles

this is standard unix practice (do a man on gzip, tar, etc). now, if ruby
just started willy nilly adding a '-' to the command line this technique would
not work because it's not just the existence of '-' but the __position__ of it
which can have meaning.

auto-munging of ARGV is a bad idea imho.

regards.

-a

 

[gwtmp01]

thanks, a

This kind of sucks though.

apparently in perl, if there is a piped input, '-' will show up
automatically in ARGV. I think I will propose this change to Ruby.

Normally I would just nod my head in agreement with Ara, but not this
time.

I just have to say yuck with regard to using '-' and the idea that
command lines
should have have an explicit syntax to indicate that the program
should read from
stdin.

If your program is prepared to deal with piped input, why are
you concerned about input coming from the keyboard? If I found a
program
that worked as:

cat file | program

and as

program < file

but did not work as just

program

I would be surprised, to say the least.

Usually the issue is the other way around where an interactive
program (vim, for
example) *needs* a tty device on stdin and may complain when
STDIN.tty? fails, but
I'm not sure I understand the need to complain about STDIN being tied
to a tty device
for a program that is just reading a stream of data.

The '-' hack just makes me shudder. Something like '/dev/stdin'
would be marginally better
if you really need something like that (although not as portable).

 

[gwtmp01]

consider i have many, many programs which do things like

convert infile outfile
convert - outfile # infile on stdin
convert infile - # outfile on stdout
convert - - # infile on stdin, outfile on stdout
convert --infiles=- # list of infiles on stdin, auto-name
outfiles

While I understand the historical reason for such constructs, I much
prefer:

convert infile outfile
convert /dev/stdin outfile # infile on stdin
convert infile /dev/stdout # outfile on stdout
convert # infile on stdin, outfile on stdout

convert --infiles=- # list of infiles on stdin, auto-name
outfiles

Not sure about this last one. It certainly isn't all that common.
You can get close
to this via:

cat filenames | xargs convert

with maybe an option to convert to cause it to send output to auto-
named outfiles:

cat filenames | xargs convert --inplace

I realize that some Unix systems don't support /dev/stdin, /dev/
stdout, though...


Gary Wright

 

[ara.t.howard]

If your program is prepared to deal with piped input, why are you concerned
about input coming from the keyboard? If I found a program that worked as:

cat file | program

and as

program < file

but did not work as just

program

it may well work, but hang forever in cron though:

this is as easy to do as

infile = get_infile # oops, returned nil
system "program #{ infile }" # hangs forever

that's also very problematic for prorgams which take two arguments:

cat infile | program - outfile
cat infile | program - - | outfile_filter > outfile

note that you cannot simply say that the abscence of infile means stdin and
similar for outfile because

program infile_or_outfile?

i'm sure one could skin this differently, i just have debugged many hung
processes that make assumptions based on STDIN.tty?

kind regards.

-a

 

[ara.t.howard]

(e-mail address removed) wrote:

/ ...


It also isn't true. IMHO Larry Wall wouldn't countenance such a thing.

thanks - that's good to know. larry++.

-a

 

[ara.t.howard]

While I understand the historical reason for such constructs, I much prefer:

convert infile outfile
convert /dev/stdin outfile # infile on stdin
convert infile /dev/stdout # outfile on stdout
convert # infile on stdin, outfile on stdout
agreed.


Not sure about this last one. It certainly isn't all that common. You can
get close to this via:

harp:~ > printf "stdin\nstdout\nstderr\n" | grep -i -f- a.rb
STDERR.puts "#{ msg } (#{ tid })"
STDIN.gets

we have lots of code that does this because they take thousands of infiles,
hdrfiles, etc...

cheers.


-a

 

[gwtmp01]

it may well work, but hang forever in cron though:

Hmm. Doesn't cron arrange for stdin and stdout to be /dev/null, in
which case
the read will simply return EOF, correct?

I just tried some experiments on Mac OS X and it seems that cron
arranges for
stdin to be an empty pipe for cron jobs.

In any case, clearly the program has been invoked incorrectly for that
context (background cron job) and it pretty much doesn't matter what
structure
you choose for the command line arguments it is always possible to
screw it
up and provide the wrong arguments or plumb the wrong input or output
sources.

i'm sure one could skin this differently, i just have debugged many
hung
processes that make assumptions based on STDIN.tty?

I'm not sure I understand. The OP was suggesting that the program
was designed
to read from a pipe, so it really shouldn't care if it was reading
from a tty
device, should it?

I think of this like duck typing in Ruby. As long as you can read lines
from the file descriptor and your program is designed to read lines
of text,
should you really care if stdin is associated with a file, a pipe, a
network
connection, or a tty device? Why not just read and process the data?


Gary Wright

 

[ara.t.howard]

Hmm. Doesn't cron arrange for stdin and stdout to be /dev/null, in which
case the read will simply return EOF, correct?

I just tried some experiments on Mac OS X and it seems that cron arranges
for stdin to be an empty pipe for cron jobs.

hmmm. you are right. i had to look back at what had given me trouble before
- the crux of it was basically this

input_on_stdin = not STDIN.tty?

infile =
if input_on_stdin
STDIN.read
end

so, basically assuming that the user gave input on stdin simply because STDIN
was not a tty, which may or may not be true, as in the case where one is
running under cron or a system call from a daemon, etc.

i guess my point is that

'stdin is not a tty'

and

'input was given on stdin'

are two orthogonal ideas which, when merged, have bitten me in the past. so
that's why i've developed a preference for '-' to signal input on stdin - it's
non-ambiguous.

In any case, clearly the program has been invoked incorrectly for that
context (background cron job) and it pretty much doesn't matter what
structure you choose for the command line arguments it is always possible to
screw it up and provide the wrong arguments or plumb the wrong input or
output sources.
indeed.

I'm not sure I understand. The OP was suggesting that the program was
designed to read from a pipe, so it really shouldn't care if it was reading
from a tty device, should it?

well - that is true. maybe i confused things by making it more general. the
only reason i was making a fuss is that it's caused me problems - as i
mentioned. probably the noise wasn't worth it.

I think of this like duck typing in Ruby. As long as you can read lines
from the file descriptor and your program is designed to read lines of text,
should you really care if stdin is associated with a file, a pipe, a network
connection, or a tty device? Why not just read and process the data?

i think that you are quite right in this case.

cheers.

-a

 

[greg]

I wanted the ability to optionally pipe a file, but the program would
not require a file to be piped or given as an argument. I guess the
answer is that this cannot be done. I can only allow an optional file
argument, not a pipe.

 

[greg]

I wanted the ability to optionally pipe a file, but the program would
not require a file to be piped or given as an argument. I guess the
answer is that this cannot be done. I can only allow an optional file
argument, not a pipe.

 

[gwtmp01]

I wanted the ability to optionally pipe a file, but the program would
not require a file to be piped or given as an argument. I guess the
answer is that this cannot be done. I can only allow an optional
file
argument, not a pipe.

I'm not sure I understand what you mean by 'pipe a file'. Pipes and/or
file redirection are arranged by the shell not by the programs running
under the shell. The 'normal' way this is handled is:

program file # program opens and reads from named file
program # program simply reads from standard input

In the second case, standard input can be connected to all sorts of
things
depending how the shell sets up the environment:

program
program < /from/file
source | program

In all three cases, the program simply reads from standard input and is
oblivious to whether standard input is associated with the terminal, a
file, or a pipe. All the 'plumbing' is done by the shell prior to
executing
the program.

Ruby does give you a nice way to read from either standard input or
a list of files named on the command line via the special ARGF object.
So you simply let ARGF examine the command line arguments and decide if
it should get data from STDIN or if should open and read data from the
named files.

ARGF is a convenience, it isn't doing anything you couldn't do
explicitly
on your own by processing ARGV yourself.

I don't know if that confuses things or not as I don't really understand
your objective.


Gary Wright

 

[greg]

I wanted the ability to optionally pipe a file, but the program would
not require a file to be piped or given as an argument. I guess the
answer is that this cannot be done. I can only allow an optional file
argument, not a pipe.

 

[greg]

I guess the problem comes down to: can I determine if there is data
being piped or if there is data waiting on STDIN before reading. I am
really trying to avoid hanging from doing ARGF.readlines (or something
similiar) if there is no data being piped.

If a file is given as an argument it will show up in ARGV, and we know
it is there, and can act accordingly. Is there any way we can know
that piped data is there?

I avoid at all costs writing a program that will hang under any
conditions. For me, a program should always fail gracefully, it should
not lose control. If it is invoked incorrectly, there should be a
message explaining how to invoke it correctly, it should not just hang.

 

[ara.t.howard]

I guess the problem comes down to: can I determine if there is data
being piped or if there is data waiting on STDIN before reading. I am
really trying to avoid hanging from doing ARGF.readlines (or something
similiar) if there is no data being piped.

windows or *nix?


-a

 

[gwtmp01]

Ara's idea is how it is done in Unix, I do not know why he got so much
critisized for it

Actually I think the 'normal' thing in Unix is to simply do blocking
reads
from stdin if there are no file arguments. This is exactly what ARGF
does.
It is decidedly non-standard to *require* the use of a sentinel such
as '-' (or even better /dev/stdin) to cause a program to read from
stdin.
And I believe this is what the OP is looking for. I just don't think it
makes a lot of sense. For example, standard input redirection would
have
to be done as:

program - < /some/file/to/read

which is strange. Pipes also:

source | program -

In both cases if you leave off the '-' the program will report an error
rather than reading and processing the data. The next programmer
who comes along is going to be very confused by this behavior.

Can anyone think of a standard Unix program that will *only* read from
stdin in if you give an explicit sentinel as a file argument?


Gary Wright

 

[ara.t.howard]

In both cases if you leave off the '-' the program will report an error
rather than reading and processing the data. The next programmer
who comes along is going to be very confused by this behavior.

not if they work on unix! ;-)

Can anyone think of a standard Unix program that will *only* read from
stdin in if you give an explicit sentinel as a file argument?

#
# tarfile on stdout
#
harp:~ > tar -cf - directory > tarfile_on_stdout.tgz

#
# piped tarfile on stdin
#
harp:~ > tar -cf - directory | tar -xvf -


#
# copying files overnetwork via piped tarfile and unpacking on the other side
#
harp:~ > (cd /src; tar -cvf - foo) | (ssh other.machine 'cd /dst; tar -xf -')


#
# echo foobar into a gzip of stdin, sending compressed output to stdout, pipe
# that into another gzip which is decompressing stdin, dump that output back
# out to stdout
#
harp:~ > echo foobar | gzip - -c | gzip -d - -c
foobar

#
# use image magick's convert command to convert stdin -> stdout
#
harp :~ > convert - - < map.png > map2.png
harp :~ > file map2.png
map2.png: PNG image data, 713 x 569, 8-bit/color RGB, non-interlaced

#
# grep for list of patterns on stdin
#
harp:~ > echo alias | grep -f- .bashrc
alias new='ls -ltar'
alias p="fetchmail;pine -i -passfile /home/ahoward/.passfile"
alias g="glimpse -n -H"
alias gi='glimpseindex -B -t -f -H'
alias ldate='env TZ=America/Denver date'
alias ssh='ssh -X'
alias vi='vim'
alias mussel='tti -A (e-mail address removed)'
alias ss='screen -S '
alias sl='screen -list '
alias sdr='screen -d -r '
alias s='screen -D -R '
alias dark='eval `dircolors /etc/DIR_COLORS` && export DIR_COLORS=dark'
alias light='eval `dircolors /etc/DIR_COLORS.xterm` && export DIR_COLORS=light'
alias xt="xterm -font 7x13 -fb 7x13B -geometry 80x25 -sb -wf -j -ls -bg Black -fg grey &"



note that none of these are documented. i think that's because it's considered
'standard' bahaviour.


cheers.


-a