Net::HTTP Closes STDIN

[James Edward Gray II]

Kenneth Kalmer has brought up a HighLine issue and I'm trying to look
into it. Oddly, it seems to happen when interacting with the
Net::HTTP library. I've narrowed it done to the following example on
my box:

$ cat stdin_closed_issue.rb
require 'net/http'
require 'io/wait'

Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
$ ruby stdin_closed_issue.rb
true

Can anyone explain why $stdin is closed after the page read?

James Edward Gray II

 

[Eric Hodel]

Kenneth Kalmer has brought up a HighLine issue and I'm trying to
look into it. Oddly, it seems to happen when interacting with the
Net::HTTP library. I've narrowed it done to the following example
on my box:

$ cat stdin_closed_issue.rb
require 'net/http'
require 'io/wait'

Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
$ ruby stdin_closed_issue.rb
true

Can anyone explain why $stdin is closed after the page read?

$stdin isn't closed, its at the end of file. Use #closed? to test if
an IO has been closed or not.

$ cat test.rb
require 'net/http'
require 'io/wait'

Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.closed?
p $stdin.eof?

$ ruby test.rb
false # hit ^D to close $stdin on the sending side.
true
$
$ echo hi | ruby test.rb
false
false

 

[James Edward Gray II]

$stdin isn't closed, its at the end of file. Use #closed? to test
if an IO has been closed or not.

Right, good point.

However, when I call eof?() before the Net::HTTP call it behaves
differently (stalls and prints false). Why does it not behave the
same after that page read?

James Edward Gray II

 

[Eric Hodel]

Right, good point.

However, when I call eof?() before the Net::HTTP call it behaves
differently (stalls and prints false). Why does it not behave the
same after that page read?

$ cat test.rb
puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?

require 'net/http'
require 'io/wait'

Net::HTTP.start('localhost', 80) do |http|
body = http.get('/').body
end

puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?

$ ruby test.rb
closed? false
type some text here
eof? false
closed? false
eof? false
$

For the first #eof? no data written, so Ruby waits until something's
been flushed. I typed some text and hit return to flush the
terminal's stdout (Ruby's $stdin). At the second #eof? no input on
$stdin has been consumed, so Ruby doesn't need to check for #eof?
again by waiting.

$ cat test.rb
puts "eof? %p" % $stdin.eof?
gets
puts "eof? %p" % $stdin.eof?
$ ruby test.rb
aoeu
eof? false
aoeu
eof? false

 

[James Edward Gray II]

$ cat test.rb
puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?

require 'net/http'
require 'io/wait'

Net::HTTP.start('localhost', 80) do |http|
body = http.get('/').body
end

puts "closed? %p" % $stdin.closed?
puts "eof? %p" % $stdin.eof?

$ ruby test.rb
closed? false
type some text here
eof? false
closed? false
eof? false
$

For the first #eof? no data written, so Ruby waits until
something's been flushed.

OK, that makes sense. However, why does moving that eof?() check
below a Net::HTTP page fetch change this? In that case, why does
Ruby not need for something to be flushed in that case and why is eof?
() then +true+?

James Edward Gray II

 

[Eric Hodel]

OK, that makes sense. However, why does moving that eof?() check
below a Net::HTTP page fetch change this?

I don't see this behavior. With your original example I have to hit
^D to get a prompt back.

$ ruby -v
ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]

In that case, why does Ruby not need for something to be flushed

Because #eof? reads a character if feof(3) is not true and there's no
data pending:

rb_io_eof(io)
VALUE io;
{
OpenFile *fptr;
int ch;

GetOpenFile(io, fptr);
rb_io_check_readable(fptr);

if (feof(fptr->f)) return Qtrue;
if (READ_DATA_PENDING(fptr->f)) return Qfalse;
READ_CHECK(fptr->f);
clearerr(fptr->f);
TRAP_BEG;
ch = getc(fptr->f);
^^^^^^^^^^^^^^^^^^^

Analysis of sampling pid 15640 every 10.000000 milliseconds
Call graph:
[...]
300 rb_io_eof
300 getc
[...]

in that case and why is eof?() then +true+?

#eof? will only be true when there's no more data to read.

 

[James Edward Gray II]

OK, that makes sense. However, why does moving that eof?() check
below a Net::HTTP page fetch change this?

I don't see this behavior. With your original example I have to
hit ^D to get a prompt back.

$ ruby -v
ruby 1.8.5 (2006-12-04 patchlevel 2) [i686-darwin8.8.2]

I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don't get prompted for input and it still tells me $stdin
is closed:

Firefly:~/Desktop$ ruby -v
ruby 1.8.5 (2006-12-25 patchlevel 12) [i686-darwin8.8.1]
Firefly:~/Desktop$ cat stdin_eof_issue.rb
require 'net/http'
require 'io/wait'

Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
Firefly:~/Desktop$ ruby stdin_eof_issue.rb
true

James Edward Gray II

 

[James Edward Gray II]

I still don't get prompted for input and it still tells me $stdin
is closed:

I'm sorry, I meant it still claims $stdin is at eof?().

James Edward Gray II

 

[Jacob Fugal]

I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don't get prompted for input and it still tells me $stdin
is [eof]:

I can confirm on my machine:

lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]
lukfugl@hephaestus:~$ cat test.rb
require 'net/http'
require 'io/wait'

Net::HTTP.start('www.ruby-lang.org', 80) do |http|
body = http.get('/en/license.txt').body
end
p $stdin.eof?
lukfugl@hephaestus:~$ ruby test.rb
true

One note: the first time I ran it, I thought it was waiting for input
too, but it was really just a few second delay while fetching
http://www.ruby-lang.org/en/license.txt.

Jacob Fugal

 

[Eric Hodel]

I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don't get prompted for input and it still tells me $stdin
is [eof]:

I can confirm on my machine:

Then your getc may do something different than mine (BSD-ish).

 

[James Edward Gray II]

I just upgraded to Ruby 1.8.5 to see if it would make a difference,
but I still don't get prompted for input and it still tells me
$stdin
is [eof]:

I can confirm on my machine:

Then your getc may do something different than mine (BSD-ish).

In other words, you don't feel this is a Ruby issue, right?

James Edward Gray II

 

[gwtmp01]

Kenneth Kalmer has brought up a HighLine issue and I'm trying to
look into it. Oddly, it seems to happen when interacting with the
Net::HTTP library. I've narrowed it done to the following example
on my box:

Here is a simpler example that illustrates the issue:

Thread.new { }
p $stdin.eof?

As Eric Hodel pointed out, EOF on a terminal device is detected by
trying to read from
the device. It appears that *prior* to the creation of a thread, a
read on a terminal
device will simply block waiting for some activity on the device;
either actual data
or indication of an eof condition (someone typing control-d).

*After* a thread has been created, Ruby's green thread implementation
kicks in. This
means that the read caused by IO#eof? on the terminal device is not
allowed to block.
Instead it fails as an interrupted system call, which is interpreted
by IO#eof? as an
end of file condition. This is true even if the main thread is the
only running thread.

James' original example used Net::HTTP, which uses Timeout, which
creates a thread to
gain control of code that may block (i.e., an HTTP connection to a
remote host).

I don't know enough about Ruby's thread implementation to know if
this is a bug or
a feature. My hunch is that IO#eof? needs to be fixed so that an
interrupted system
call doesn't get interpreted as an end of file condition. Another
change would be to
turn off the software interrupt that is used to multiplex IO activity
by threads when
only the main thread is running.

Gary Wright

 

[ara.t.howard]

Here is a simpler example that illustrates the issue:

Thread.new { }
p $stdin.eof?

As Eric Hodel pointed out, EOF on a terminal device is detected by trying to
read from the device. It appears that *prior* to the creation of a thread,
a read on a terminal device will simply block waiting for some activity on
the device; either actual data or indication of an eof condition (someone
typing control-d).

*After* a thread has been created, Ruby's green thread implementation kicks
in. This means that the read caused by IO#eof? on the terminal device is
not allowed to block. Instead it fails as an interrupted system call, which
is interpreted by IO#eof? as an end of file condition. This is true even if
the main thread is the only running thread.

James' original example used Net::HTTP, which uses Timeout, which creates a
thread to gain control of code that may block (i.e., an HTTP connection to a
remote host).

I don't know enough about Ruby's thread implementation to know if this is a
bug or a feature. My hunch is that IO#eof? needs to be fixed so that an
interrupted system call doesn't get interpreted as an end of file condition.
Another change would be to turn off the software interrupt that is used to
multiplex IO activity by threads when only the main thread is running.

Gary Wright

sounds quite reasonable. however, on my box:

harp:~ > cat a.rb
Thread.new { }
p $stdin.eof?

harp:~ > ruby a.rb # and then type ctrl-d
true

harp:~ > ruby -v
ruby 1.8.4 (2005-12-01) [i686-linux]

harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686

so looks to be a bsd/mac issue?

-a

 

[Jacob Fugal]

... on my box:

harp:~ > cat a.rb
Thread.new { }
p $stdin.eof?

harp:~ > ruby a.rb # and then type ctrl-d
true

harp:~ > ruby -v
ruby 1.8.4 (2005-12-01) [i686-linux]

harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686

so looks to be a bsd/mac issue?

On my box:

lukfugl@hephaestus:~$ cat a.rb
Thread.new { }
p $stdin.eof?

lukfugl@hephaestus:~$ ruby a.rb # no CTRL-D, terminates immediately
true

lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]

lukfugl@hephaestus:~$ uname -srm
Linux 2.6.17-2-686 i686

So it's present for me with ruby 1.8.5 on linux 2.6.17. I did just
notice that my ruby is compiled i486-linux -- that *may* have
something to do with it, but I don't think so.

Jacob Fugal

 

[gwtmp01]

Here is a simpler example that illustrates the issue:

Thread.new { }
p $stdin.eof?

sounds quite reasonable. however, on my box:

harp:~ > cat a.rb
Thread.new { }
p $stdin.eof?

harp:~ > ruby a.rb # and then type ctrl-d
true

harp:~ > ruby -v
ruby 1.8.4 (2005-12-01) [i686-linux]

harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686

so looks to be a bsd/mac issue?

My post was based on ruby 1.8.5 on Darwin 8.8.0 Power Macintosh

The anomaly does not occur with ruby-1.9 on the same OS/hardware.

Gary Wright

 

[Eric Hodel]

Here is a simpler example that illustrates the issue:

Thread.new { }
p $stdin.eof?

As Eric Hodel pointed out, EOF on a terminal device is detected by
trying to read from the device. It appears that *prior* to the
creation of a thread, a read on a terminal device will simply block
waiting for some activity on the device; either actual data or
indication of an eof condition (someone typing control-d).

*After* a thread has been created, Ruby's green thread
implementation kicks in. This means that the read caused by
IO#eof? on the terminal device is not allowed to block.

If there are other threads and the IO isn't readable, ruby will run
those until they complete. Your thread immediately stopped.

Since the thread in your example has completed there's no threads to
switch to, so the main thread blocks waiting for a character (on a
BSD-ish libc). If there was a thread running, the main thread would
remain blocked (as if it were blocked on getc(3)) and the other
threads were running.

To figure out what your platform is doing use a tool that gives
system-call information like ktrace.

 

[ara.t.howard]

On my box:

lukfugl@hephaestus:~$ cat a.rb
Thread.new { }
p $stdin.eof?

lukfugl@hephaestus:~$ ruby a.rb # no CTRL-D, terminates immediately
true

lukfugl@hephaestus:~$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]

lukfugl@hephaestus:~$ uname -srm
Linux 2.6.17-2-686 i686

So it's present for me with ruby 1.8.5 on linux 2.6.17. I did just notice
that my ruby is compiled i486-linux -- that *may* have something to do with
it, but I don't think so.

strange:

[nrt@anchor ~]$ ruby -v -e' Thread.new{} and p STDIN.eof? ' # ctrl-d
ruby 1.8.5 (2006-12-11 patchlevel 5000) [i686-linux]
true

[nrt@anchor ~]$ uname -srm
Linux 2.6.9-42.0.3.ELsmp i686

and:

harp:~ > ruby19 -v -e' Thread.new{} and p STDIN.eof? ' # ctrl-d
ruby 1.9.0 (2006-07-26) [i686-linux]
true

harp:~ > uname -srm
Linux 2.4.21-47.0.1.EL i686

this could get tricky to debug! none of my linux versions or ruby versions
show this...

just to be clear (to everyone): have you all compiled ruby from source? all
my installs are from source using nothing but '--prefix' given to ./configure.
if anyone has a package (cough) manager or vendor (cough cough) install i'd
reccomend compiling from source without any options and testing again.

kind regards.

-a

 

[gwtmp01]

If there are other threads and the IO isn't readable, ruby will run
those until they complete. Your thread immediately stopped.

I coded it that way on purpose to see if the issue depended on
whether another thread was running. On my platform it doesn't
matter. It also doesn't matter if the main thread joins the now dead
thread. It is the initiation of
another thread that changes the behavior of IO#eof? on certain
platforms. The two platforms that have been reported
are:
ruby 1.8.5 Linux 2.6.17-2-686 i686
ruby 1.8.5 Darwin 8.8.0 Power Macintosh (Mac OS X 10.4.8)

Since the thread in your example has completed there's no threads
to switch to, so the main thread blocks waiting for a character (on
a BSD-ish libc). If there was a thread running, the main thread
would remain blocked (as if it were blocked on getc(3)) and the
other threads were running.

But my point (and James' point) was that on my (his) platform it
*doesn't* block when it logically should.

To figure out what your platform is doing use a tool that gives
system-call information like ktrace.

I did that and came up with the simpler example in the process. If
you run just:

p $stdin.eof?

Ruby blocks waiting for input. If you first create a thread
which immediately terminates (my simple example) then Ruby doesn't
block (ruby 1.8.4 on Mac OS X) instead the read call is interrupted
with a software interrupt, which I interpreted as being party of Ruby's
thread implementation.

Gary Wright

 

[James Edward Gray II]

this could get tricky to debug!

Bingo. It's actually a pretty significant problem for HighLine
because after you do something like fetching a web page, HighLine
believes $stdin to e closed and won't allow you to ask the user any
more questions. I fear we will need to disable HighLine's eof?()
check the work around this.

just to be clear (to everyone): have you all compiled ruby from
source?

I compiled from source. I did --enable-pthread as well.

James Edward Gray II

 

[James Edward Gray II]

But my point (and James' point) was that on my (his) platform it
*doesn't* block when it logically should.

Thank you. I was beginning to think I was going crazy.

James Edward Gray II