Using net/HTTP - How to login and then retrieve data?

Discussion in 'Ruby' started by Bob C., Jan 21, 2011.

  1. Bob C.

    Bob C. Guest

    I'm new to Ruby, coming from Java. I need to write an HTTP client in
    Ruby. I'm working in a Windows environment with Ruby 1.8 and RubyMine.

    I've been working with net/http, and have written the code below, to
    login to an HTTP server, then issue a POST to pull over some data, and
    then logout. I can successfully Login, but the subsequent POST returns
    an error indicating that I have Invalid Username/Password. I am also new
    to HTTP processing, so I am not sure of the sequence of events that need
    to take place for a successful Login-POST-Logout.

    Please review my code and let me know what I need to do for the POST to



    require 'net/http'

    url1 =
    req =
    req.basic_auth 'myuser', 'mypass'
    req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
    'User-Agent' => 'Mozilla/4.0'})
    res1 = Net::HTTP.start(, url1.port) {|http| http.request(req)}
    puts res1.body # This returns a 200 response and a listing of methods
    that can be called, therefore this Login snippet works

    url2Str = ''
    url2 = URI.parse(url2Str)
    path = ''
    http =, 80)
    data =
    headers = {
    'Accept' => '*/*',
    'User-Agent' => 'Mozilla/4.0',
    'RETS-Version' => 'RETS/1.7'
    res2 =, data, headers)
    puts res2.body # This returns error <Net::HTTPUnauthorized 401 Invalid
    Username/Password combination readbody=true>

    url3 =
    req =
    req.basic_auth 'myuser', 'mypass'
    req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
    'User-Agent' => 'Mozilla/4.0'})
    res3 = Net::HTTP.start(, url3.port) {|http| http.request(req)}
    puts res3.body # This returns a 200 response
    Bob C., Jan 21, 2011
    1. Advertisements

  2. Do you need to also set .basic_auth on the POST request?
    Do you need to store a cookie? (In which case, you might prefer using
    httpclient gem rather than net/http directly.)

    You can also do:

    http =, url.port)
    http.read_timeout = 120
    if url.scheme == 'https' || url.port == 443
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE
    http.timeout = 120
    http.start do |connection|
    request =
    request.basic_auth 'user', 'password'

    request.body = 'something if needed'

    response = connection.request(request)

    # do other stuff with the same connection...

    Rob Biedenharn
    Rob Biedenharn, Jan 21, 2011
    1. Advertisements

  3. [Note: parts of this message were removed to make it a legal post.]

    Well Net::HTTP is a bit on the low level side. You really need to know the
    details of HTTP in order to be productive.

    There are a lot of alternative HTTP client libraries out there that may give
    you what you want a lot easier (abstracting the HTTP details away);
    All of those are pure Ruby, and should work fine on windows.
    For pure web scraping/form interacting clients you can take it up a level.
    WATIR is well worth a look if you are on windows (your QA guys will want a
    demo, as it has a visual mode)
    There is also Webrat and Capybara which are outstanding, normally used for
    assertion level testing.

    If Java is your thing, you might want to consider JRuby specific options as
    well, like Celerity (which uses native Java HttpClient IIRC)

    Net::HTTP forces you down into a very low level of HTTP mechanics. For most
    use cases this is not necessary, and its a tricky library to work with if
    your needs are simpler.
    Many of those other options are easier to learn and may be more directly
    applicable to your problem (logging in, interacting & leaving)
    Richard Conroy, Jan 21, 2011
  4. Bob C.

    Bob C. Guest

    Yes, POST also requires basic_auth.

    No, a cookie is not required.

    For now I'd like to stick with using net/http, and possibly later look
    at things like httpclient.

    When I code according to your example (see below) I am still failing on
    the SEARCH operation due to error <Net::HTTPPreconditionFailed 412
    Precondition Failed-User not logged in. readbody=true>. This doesn't
    make sense because I have successfully logged in, and I am re-using the

    I appreciate your help Rob, because I didn't know how to re-use the
    connection - Thanks!


    Here's my current code:

    require 'net/http'

    url1 =
    http =, url1.port)
    http.read_timeout = 120
    http.start do |connection|

    request1 =
    request1.basic_auth 'myuser', 'mypass'
    request1.set_form_data({'RETS-Version' => 'RETS/1.7', 'Accept' =>
    '*/*', 'User-Agent' => 'Mozilla/4.0'})
    #request1.body = 'something if needed'
    response1 = connection.request(request1)
    puts response1.body

    url2 =
    request2 =
    request2.basic_auth 'myuser', 'mypass'
    request2.set_form_data({'RETS-Version' => 'RETS/1.7', 'Accept' =>
    '*/*', 'User-Agent' => 'Mozilla/4.0',
    'SearchType' => 'Property',
    'Limit' => '15000',
    'Format' => 'COMPACT',
    'Query' =>
    'Class' => 'BC_4'
    response2 = connection.request(request2)
    puts response2.inspect

    url3 =
    request3 =
    request3.set_form_data({'RETS-Version' => 'RETS/1.7', 'Accept' =>
    '*/*', 'User-Agent' => 'Mozilla/4.0'})
    request3.basic_auth 'myuser', 'mypass'
    response3 = connection.request(request3)
    puts response3.body
    #<Net::HTTPOK 200 OK readbody=true>

    #<Net::HTTPPreconditionFailed 412 Precondition Failed-User not logged
    in. readbody=true>

    #<Net::HTTPOK 200 OK readbody=true>

    Rob Biedenharn wrote in post #976647:
    Bob C., Jan 22, 2011
  5. Bob C.

    Bob C. Guest

    Bob C., Jan 24, 2011
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.