Using net/HTTP - How to login and then retrieve data?

[Bob C.]

I'm new to Ruby, coming from Java. I need to write an HTTP client in
Ruby. I'm working in a Windows environment with Ruby 1.8 and RubyMine.

I've been working with net/http, and have written the code below, to
login to an HTTP server, then issue a POST to pull over some data, and
then logout. I can successfully Login, but the subsequent POST returns
an error indicating that I have Invalid Username/Password. I am also new
to HTTP processing, so I am not sure of the sequence of events that need
to take place for a successful Login-POST-Logout.

Please review my code and let me know what I need to do for the POST to
work.

Thanks!

--Bob

require 'net/http'

#LOGIN
url1 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/login')
req = Net::HTTP::Get.new(url1.path)
req.basic_auth 'myuser', 'mypass'
req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0'})
res1 = Net::HTTP.start(url1.host, url1.port) {|http| http.request(req)}
puts res1.body # This returns a 200 response and a listing of methods
that can be called, therefore this Login snippet works

#POST
url2Str = 'http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search'
url2 = URI.parse(url2Str)
path = 'http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search'
http = Net::HTTP.new(url2.host, 80)
data =
'SearchType=Property&Limit=15000&Format=COMPACT&Query=%28L_UpdateDate%3D2011-01-03T23:49:57%2B%29,%28L_UpdateDate%3D2011-01-04T23:49:57%2D%29&Class=BC_4'
headers = {
'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0',
'RETS-Version' => 'RETS/1.7'
}
res2 = http.post(url2Str, data, headers)
puts res2.body # This returns error <Net::HTTPUnauthorized 401 Invalid
Username/Password combination readbody=true>

#LOGOUT
url3 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/logout')
req = Net::HTTP::Get.new(url3.path)
req.basic_auth 'myuser', 'mypass'
req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0'})
res3 = Net::HTTP.start(url3.host, url3.port) {|http| http.request(req)}
puts res3.body # This returns a 200 response

 

[Rob Biedenharn]

I'm new to Ruby, coming from Java. I need to write an HTTP client in
Ruby. I'm working in a Windows environment with Ruby 1.8 and RubyMine.

I've been working with net/http, and have written the code below, to
login to an HTTP server, then issue a POST to pull over some data, and
then logout. I can successfully Login, but the subsequent POST returns
an error indicating that I have Invalid Username/Password. I am also
new
to HTTP processing, so I am not sure of the sequence of events that
need
to take place for a successful Login-POST-Logout.

Please review my code and let me know what I need to do for the POST
to
work.

Thanks!

--Bob

require 'net/http'

#LOGIN
url1 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/
login')
req = Net::HTTP::Get.new(url1.path)
req.basic_auth 'myuser', 'mypass'

Do you need to also set .basic_auth on the POST request?
Do you need to store a cookie? (In which case, you might prefer using
httpclient gem rather than net/http directly.)

You can also do:

http = Net::HTTP.new(url.host, url.port)
http.read_timeout = 120
if url.scheme == 'https' || url.port == 443
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
http.timeout = 120
end
http.start do |connection|
request = Net::HTTP:ost.new(url.request_uri)
request.basic_auth 'user', 'password'

request.body = 'something if needed'

response = connection.request(request)

# do other stuff with the same connection...
end

req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0'})
res1 = Net::HTTP.start(url1.host, url1.port) {|http|
http.request(req)}
puts res1.body # This returns a 200 response and a listing of methods
that can be called, therefore this Login snippet works

#POST
url2Str = 'http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/
search'
url2 = URI.parse(url2Str)
path = 'http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search'
http = Net::HTTP.new(url2.host, 80)
data =
'SearchType=Property&Limit=15000&Format=COMPACT&Query=%28L_UpdateDate
%3D2011-01-03T23:49:57%2B%29,%28L_UpdateDate%3D2011-01-04T23:49:57%2D
%29&Class=BC_4'
headers = {
'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0',
'RETS-Version' => 'RETS/1.7'
}
res2 = http.post(url2Str, data, headers)
puts res2.body # This returns error <Net::HTTPUnauthorized 401
Invalid
Username/Password combination readbody=true>

#LOGOUT
url3 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/logout')
req = Net::HTTP::Get.new(url3.path)
req.basic_auth 'myuser', 'mypass'
req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0'})
res3 = Net::HTTP.start(url3.host, url3.port) {|http|
http.request(req)}
puts res3.body # This returns a 200 response

Rob Biedenharn
(e-mail address removed) http://AgileConsultingLLC.com/
(e-mail address removed) http://GaslightSoftware.com/

 

[Richard Conroy]

[Note: parts of this message were removed to make it a legal post.]

I'm new to Ruby, coming from Java. I need to write an HTTP client in
Ruby. I'm working in a Windows environment with Ruby 1.8 and RubyMine.

I've been working with net/http, and have written the code below, to
login to an HTTP server, then issue a POST to pull over some data, and
then logout. I can successfully Login, but the subsequent POST returns
an error indicating that I have Invalid Username/Password. I am also new
to HTTP processing, so I am not sure of the sequence of events that need
to take place for a successful Login-POST-Logout.

Well Net::HTTP is a bit on the low level side. You really need to know the
details of HTTP in order to be productive.

There are a lot of alternative HTTP client libraries out there that may give
you what you want a lot easier (abstracting the HTTP details away);
OTOH:
RestClient
HttpClient
Patron
Httparty
Resourceful
All of those are pure Ruby, and should work fine on windows.
For pure web scraping/form interacting clients you can take it up a level.
WATIR is well worth a look if you are on windows (your QA guys will want a
demo, as it has a visual mode)
There is also Webrat and Capybara which are outstanding, normally used for
assertion level testing.

If Java is your thing, you might want to consider JRuby specific options as
well, like Celerity (which uses native Java HttpClient IIRC)

Net::HTTP forces you down into a very low level of HTTP mechanics. For most
use cases this is not necessary, and its a tricky library to work with if
your needs are simpler.
Many of those other options are easier to learn and may be more directly
applicable to your problem (logging in, interacting & leaving)

 

[Bob C.]

Yes, POST also requires basic_auth.

No, a cookie is not required.

For now I'd like to stick with using net/http, and possibly later look
at things like httpclient.

When I code according to your example (see below) I am still failing on
the SEARCH operation due to error <Net::HTTPPreconditionFailed 412
Precondition Failed-User not logged in. readbody=true>. This doesn't
make sense because I have successfully logged in, and I am re-using the
connection.

I appreciate your help Rob, because I didn't know how to re-use the
connection - Thanks!

--Bob

Here's my current code:

require 'net/http'

url1 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/login')
http = Net::HTTP.new(url1.host, url1.port)
http.read_timeout = 120
http.start do |connection|

#LOGIN
request1 = Net::HTTP:ost.new(url1.request_uri)
request1.basic_auth 'myuser', 'mypass'
request1.set_form_data({'RETS-Version' => 'RETS/1.7', 'Accept' =>
'*/*', 'User-Agent' => 'Mozilla/4.0'})
#request1.body = 'something if needed'
response1 = connection.request(request1)
puts response1.body


#SEARCH
url2 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search')
request2 = Net::HTTP:ost.new(url2.request_uri)
request2.basic_auth 'myuser', 'mypass'
request2.set_form_data({'RETS-Version' => 'RETS/1.7', 'Accept' =>
'*/*', 'User-Agent' => 'Mozilla/4.0',
'SearchType' => 'Property',
'Limit' => '15000',
'Format' => 'COMPACT',
'Query' =>
'%28L_UpdateDate%3D2011-01-03T23:49:57%2B%29,%28L_UpdateDate%3D2011-01-04T23:49:57%2D%29',
'Class' => 'BC_4'
})
response2 = connection.request(request2)
puts response2.inspect

#LOGOUT
url3 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/logout')
request3 = Net::HTTP:ost.new(url3.request_uri)
request3.set_form_data({'RETS-Version' => 'RETS/1.7', 'Accept' =>
'*/*', 'User-Agent' => 'Mozilla/4.0'})
request3.basic_auth 'myuser', 'mypass'
response3 = connection.request(request3)
puts response3.body
end
#<Net::HTTPOK 200 OK readbody=true>

#<Net::HTTPPreconditionFailed 412 Precondition Failed-User not logged
in. readbody=true>

#<Net::HTTPOK 200 OK readbody=true>



Rob Biedenharn wrote in post #976647:

 

[soldier.coder]

http://railstips.org/blog/archives/2008/07/29/it-s-an-httparty-and-everyone-is-invited/

 

[Bob C.]

soldier.coder wrote in post #976863:http://railstips.org/blog/archives/2008/07/29/it-s-an-httparty-and-everyone-is-invited/

Thanks for that - looks like an interesting gem. I will try it out.

BTW: I have solved my problem - was using an encoded "Query" param that
was screwing up the server processing.

--Bob