Net/HTTP is flaky?

[Just Another Victim of the Ambient Morality]

Inexplicably, the following code fails:


http = Net::HTTP.new("somesite.com")
puts http.get("/")[1]


...with this error:


ruby/1.8/net/http.rb:925:in '[]': undefined method 'downcase' for 1:Fixnum
(NoMethodError)
from web.rb:5


Now, I do have access to a Linux system with Ruby 1.6 and it sometimes
succeeds but often fails with this error:


/usr/lib/ruby/1.6/net/protocol.rb:221:in `error!': 403 "Forbidden"
(Net:rotoFatalError)
from /usr/lib/ruby/1.6/net/http.rb:1217:in `value'
from /usr/lib/ruby/1.6/net/http.rb:605:in `get'
from ./web.rb:16


How is this possible? How can it sometimes succeed and sometimes fail?
Why can't this reliably work?
Thank you...

 

[Logan Capaldo]

Inexplicably, the following code fails:


http = Net::HTTP.new("somesite.com")
puts http.get("/")[1]


...with this error:


ruby/1.8/net/http.rb:925:in '[]': undefined method 'downcase' for
1:Fixnum
(NoMethodError)
from web.rb:5

The interface to Net::HTTP changed between 1.6 and 1.8
You want
http.get('/').body

Now, I do have access to a Linux system with Ruby 1.6 and it
sometimes
succeeds but often fails with this error:


/usr/lib/ruby/1.6/net/protocol.rb:221:in `error!': 403 "Forbidden"
(Net:rotoFatalError)
from /usr/lib/ruby/1.6/net/http.rb:1217:in `value'
from /usr/lib/ruby/1.6/net/http.rb:605:in `get'
from ./web.rb:16


How is this possible? How can it sometimes succeed and
sometimes fail?
Why can't this reliably work?
Thank you...

It doesn't reliably work because this is the internet. Also the
particular error you are seeing means that the webserver has
forbidden access to the url you are trying to, um, access.

 

[Just Another Victim of the Ambient Morality]

The interface to Net::HTTP changed between 1.6 and 1.8
You want
http.get('/').body

Ah, thanks...

It doesn't reliably work because this is the internet. Also the
particular error you are seeing means that the webserver has forbidden
access to the url you are trying to, um, access.

But I can access the page just fine with a web browser. For instance,
go to "/" at "en.wikipedia.org" and it will reliably work with a browser.
Try it with Ruby and it will usually fail, although I did get it to work
occasionally, last night, which I don't get.
Hell, "/" at "music.com" has _never_ worked and there doesn't appear to
be any kind of redirect or anything.
Does any of this make sense?
Thank you...

 

[Eric Hodel]

But I can access the page just fine with a web browser. For instance,
go to "/" at "en.wikipedia.org" and it will reliably work with a
browser.

No, it redirects to /wiki/Main_Page

Try it with Ruby and it will usually fail, although I did get it to
work
occasionally, last night, which I don't get.

$ ruby -rnet/http
Net::HTTP.start 'en.wikipedia.org' do |http| res = http.get '/'; p
res; end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

Matches browser behavior.

Hell, "/" at "music.com" has _never_ worked and there doesn't
appear to
be any kind of redirect or anything.

$ ruby -rnet/http
Net::HTTP.start 'music.com' do |http| res = http.get '/'; p res; end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

Redirects to www.music.com in the browser, Net::HTTP matches browser
behavior.

 

[Just Another Victim of the Ambient Morality]

$ ruby -rnet/http
Net::HTTP.start 'en.wikipedia.org' do |http| res = http.get '/'; p res;
end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

Matches browser behavior.


$ ruby -rnet/http
Net::HTTP.start 'music.com' do |http| res = http.get '/'; p res; end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

Redirects to www.music.com in the browser, Net::HTTP matches browser
behavior.

Okay, so the site doesn't exist where I think it does and I get
redirected... in my browser. Net::HTTP doesn't get redirected, it simply
fails. _This_ behaviour doesn't match my browser. Is there any way I can
get redirected or find out where it wants to redirect me and go there,
myself?
Thanks...

 

[Joe Van Dyk]

Okay, so the site doesn't exist where I think it does and I get
redirected... in my browser. Net::HTTP doesn't get redirected, it simply
fails. _This_ behaviour doesn't match my browser. Is there any way I can
get redirected or find out where it wants to redirect me and go there,
myself?

You might want to check out the API docs for Net::HTTP.

Copy/pasted from
http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html:


require 'net/http'
require 'uri'

def fetch(uri_str, limit = 10)
# You should choose better exception.
raise ArgumentError, 'HTTP redirect too deep' if limit == 0

response = Net::HTTP.get_response(URI.parse(uri_str))
case response
when Net::HTTPSuccess then response
when Net::HTTPRedirection then fetch(response['location'], limit - 1)
else
response.error!
end
end

print fetch('http://www.ruby-lang.org')

 

[Eric Hodel]

Okay, so the site doesn't exist where I think it does and I get
redirected... in my browser. Net::HTTP doesn't get redirected, it
simply
fails.

No, it doesn't fail, it gives you a redirect. Follow it.

_This_ behaviour doesn't match my browser. Is there any way I can
get redirected or find out where it wants to redirect me and go there,
myself?

Follow the redirect. Look at open-uri for example code.

 

[Adam Shelly]

You might want to check out the API docs for Net::HTTP.

Ok, so Net:HTTP doesn't fail, it just returns pretty low-level
information. If you want browser behavior for redirects, you have to
roll it yourself. I ran into something similar 2 days ago when I was
trying to fetch pages which required basic authentication (Which, I
just learned, is the mechanism behind those urername & password login
popups that the browser generates). I did manage to roll my own,
based on some examples from this mailing list.
Then I had to write a wrapper around that because sometimes my URI's
pointed to file:// instead of http:// - another thing the browser
handles invisibly.
Then there are those pages that use cookies to track logins...

So my question is, is there an existing library out there that
implements all these things a browser does? Where get(uri).page
returns the source of the same page I would see if I put the URI in
firefox?

-Adam

 

[Just Another Victim of the Ambient Morality]

On Jul 19, 2006, at 2:55 PM, Just Another Victim of the Ambient Morality
wrote:

No, it doesn't fail, it gives you a redirect. Follow it.

You know, it would be easier to follow the redirect if I knew that it
was giving me a redirect and if I knew how to follow one were I to be given
it...

Follow the redirect. Look at open-uri for example code.

You know, this is where Ruby's lack of documentation is really biting me
in the ass. I found some rudimentary docs on how to use it but, you know
what? For the life of me, no amount of googling can reveal exactly where I
can get the open-uri module... Seriously, where do I download this thing
from and how could I have known that?
Thank you, from a very frustrated would-be Ruby programmer...

 

[Eric Hodel]

You know, it would be easier to follow the redirect if I knew
that it
was giving me a redirect and if I knew how to follow one were I to
be given
it...

Well, if you expect to use Net::HTTP you need to know both HTTP and
the library. You might prefer WWW::Mechanize instead.

You'll get Top Quality Answers if you tell us what you really want to
do. Help with the intricacies of a library may not let us guide you
down the right path

Quoting myself:

That shows Net::HTTP returning a redirect when you perform a get.

You know, this is where Ruby's lack of documentation is really
biting me in the ass. I found some rudimentary docs on how to use
it but, you know what?

Unfortunately the Net::HTTP documentation isn't enabled in 1.8. You
can use the 1.9 documentation however:

http://ruby-doc.org/core-1.9/classes/Net/HTTP.html

As luck would have it, it has a "Following Redirection" section a few
pages down.

For the life of me, no amount of googling can reveal exactly where
I can get the open-uri module... Seriously, where do I download
this thing from and how could I have known that?

It ships with ruby, require 'open-uri'.

Thank you, from a very frustrated would-be Ruby programmer...

I've been working on getting lots more documentation enabled in 1.8.
I hope to get the net/ stuff turned on, but I don't know if I'll have
enough time to review/merge documentation from HEAD. (I'd really,
really like help with this, and Hugh Sasse has brought in a ton of
new documentation.)

 

[Logan Capaldo]

Ok, so Net:HTTP doesn't fail, it just returns pretty low-level
information. If you want browser behavior for redirects, you have to
roll it yourself. I ran into something similar 2 days ago when I was
trying to fetch pages which required basic authentication (Which, I
just learned, is the mechanism behind those urername & password login
popups that the browser generates). I did manage to roll my own,
based on some examples from this mailing list.
Then I had to write a wrapper around that because sometimes my URI's
pointed to file:// instead of http:// - another thing the browser
handles invisibly.
Then there are those pages that use cookies to track logins...

So my question is, is there an existing library out there that
implements all these things a browser does? Where get(uri).page
returns the source of the same page I would see if I put the URI in
firefox?

WWW::Mechanize is a programmatic web browser for ruby:
http://mechanize.rubyforge.org/
http://rubyforge.org/projects/mechanize

 

[Justin Collins]

Unfortunately the Net::HTTP documentation isn't enabled in 1.8. You
can use the 1.9 documentation however:

http://ruby-doc.org/core-1.9/classes/Net/HTTP.html

As luck would have it, it has a "Following Redirection" section a few
pages down.

It's in the regular standard library documentation as well:

http://ruby-doc.org/stdlib/
http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html

-Justin

 

[Joe Van Dyk]

You know, it would be easier to follow the redirect if I knew that it
was giving me a redirect and if I knew how to follow one were I to be given
it...



You know, this is where Ruby's lack of documentation is really biting me
in the ass. I found some rudimentary docs on how to use it but, you know
what? For the life of me, no amount of googling can reveal exactly where I
can get the open-uri module... Seriously, where do I download this thing
from and how could I have known that?
Thank you, from a very frustrated would-be Ruby programmer...

open-uri is included in the Ruby standard library.

The process for finding its documentation isn't that difficult.

Go to http://www.ruby-doc.org. Click on the 1.8.4 standard library
link. Click on the open-uri library link that's in the left frame.
Tada!