NTLM authentication with httpclient

Discussion in 'Ruby' started by Jim Clark, Dec 1, 2007.

  1. Jim Clark

    Jim Clark Guest

    I have rewritten my net/http script that I had questions on a couple of
    days ago using httpclient but am stuck on the NTLM authentication piece.
    This is what I have so far:

    #!c:/ruby/bin/ruby.exe
    CERT_FILE = "c:/certs/jim_nopw2.pem"
    CA_CERT = "c:/ca_certs/servers_ca.cer"

    require 'rubygems'
    require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

    client = HTTPClient.new
    client.ssl_config.set_trust_ca(CA_CERT)
    client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

    # This fetches the page but 401 error because not authenticated through NTLM
    resp = client.get("https://some_website.com/default1.asp")

    # Not sure how to turn on NTLM authentication and feed arguments. I
    tried this but still get 401 error
    #resp = client.request('GET', "https://some_website.com/default1.asp",
    {'username' => 'DOMAIN\username', 'password' => 'password'})
    puts resp.content
    puts resp.status

    Thanks again for any and all help.

    -Jim
    Jim Clark, Dec 1, 2007
    #1
    1. Advertising

  2. Jim Clark

    yermej Guest

    On Dec 1, 1:02 am, Jim Clark <> wrote:
    > I have rewritten my net/http script that I had questions on a couple of
    > days ago using httpclient but am stuck on the NTLM authentication piece.
    > This is what I have so far:
    >
    > #!c:/ruby/bin/ruby.exe
    > CERT_FILE = "c:/certs/jim_nopw2.pem"
    > CA_CERT = "c:/ca_certs/servers_ca.cer"
    >
    > require 'rubygems'
    > require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed
    >
    > client = HTTPClient.new
    > client.ssl_config.set_trust_ca(CA_CERT)
    > client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)
    >
    > # This fetches the page but 401 error because not authenticated through NTLM
    > resp = client.get("https://some_website.com/default1.asp")
    >
    > # Not sure how to turn on NTLM authentication and feed arguments. I
    > tried this but still get 401 error
    > #resp = client.request('GET', "https://some_website.com/default1.asp",
    > {'username' => 'DOMAIN\username', 'password' => 'password'})
    > puts resp.content
    > puts resp.status
    >
    > Thanks again for any and all help.
    >
    > -Jim


    There are examples included in the rubyntlm gem. I'm running Windows
    so they're in C:\ruby\lib\ruby\gems\1.8\gems\rubyntlm-0.1.1\examples\.
    YMMV.
    yermej, Dec 1, 2007
    #2
    1. Advertising

  3. Jim Clark

    Jim Clark Guest

    yermej wrote:
    >> There are examples included in the rubyntlm gem. I'm running Windows
    >> so they're in C:\ruby\lib\ruby\gems\1.8\gems\rubyntlm-0.1.1\examples\.
    >> YMMV.
    >>

    I'm really trying to do this in httpclient versus getting down into
    rubyntlm which shouldn't be necessary. I took another look at the
    httpclient.rb source and found what I was looking for on the get method
    which allows an extheader hash to be passed with arguments. Now I feel I
    am real close but I am still getting a 401 error. The script is now
    looking like:

    #!c:/ruby/bin/ruby.exe
    CERT_FILE = "c:/certs/jim_nopw2.pem"
    CA_CERT = "c:/ca_certs/servers_ca.cer"

    require 'rubygems'
    require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

    client = HTTPClient.new
    client.ssl_config.set_trust_ca(CA_CERT)
    client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

    resp = client.get("https://some_website.com/default1.asp"), nil,
    {'username' => 'DOMAIN\username', 'password' => 'password'})
    puts resp.content
    puts resp.status

    When I look at resp.content in my debugger, I see that the @header_item
    array has the normal info (content type, content length, server, etc.)
    and also has ["WWW-Authenticate", "Negotiate"] and the second
    ["WWW-Authenticate", "NTLM"]. When I trace through the get method, I see
    that @negotiate_auth has a hash @auth which == {} and @auth_default
    which == null. I guess if I can figure out how to populate these
    correctly then I'm probably good to go.

    This will have to wait until tomorrow or Sunday night and a few more
    drinks. However, I'll send another S.O.S. that if someone has already
    figured this out and can show me a good code snippet, I'd be more than
    willing to share a few drinks (if you are around Seattle) or offer high
    praise and many thanks from afar. :)

    -Jim
    Jim Clark, Dec 1, 2007
    #3
  4. Note: parts of this message were removed by the gateway to make it a legal Usenet post.

    On Dec 1, 2007 4:08 AM, Jim Clark <> wrote:

    >
    > When I look at resp.content in my debugger, I see that the @header_item
    > array has the normal info (content type, content length, server, etc.)
    > and also has ["WWW-Authenticate", "Negotiate"] and the second
    > ["WWW-Authenticate", "NTLM"]. When I trace through the get method, I see
    > that @negotiate_auth has a hash @auth which == {} and @auth_default
    > which == null. I guess if I can figure out how to populate these
    > correctly then I'm probably good to go.
    >



    Keep in mind that NTLM requires at least two round trips to the server in
    each connection. Your first GET/POST/whatever request needs to have an NTLM
    "type 1" message in the Authorize header. The server will then respond with
    a 401, but the response will contain an NTLM "type 2" message that you use
    to create an NTLM "type 3" response. You then send your GET/POST again, with
    the type 3 response in the Authorize header. At that point, if all goes
    well, you get a 2xx from the server.

    Your best bet is to use rubyntlm as a previous commenter suggested.
    Francis Cianfrocca, Dec 1, 2007
    #4
  5. Jim Clark

    Phrogz Guest

    On Dec 1, 12:02 am, Jim Clark <> wrote:
    > I have rewritten my net/http script that I had questions on a couple of
    > days ago using httpclient but am stuck on the NTLM authentication piece.
    > This is what I have so far:


    FWIW, I've worked around this in the past by using a download of curl
    that supports ntlm, and wrapped that in a shell command.

    Note that if you want binary data (which it doesn't sound like you do)
    and are on windows, you'll need to go through a temporary file to work
    around this bug:
    http://rubyforge.org/tracker/?func=detail&aid=14565&group_id=426&atid=1698
    Phrogz, Dec 1, 2007
    #5
  6. Jim Clark

    Jim Clark Guest

    Francis Cianfrocca wrote:
    > Keep in mind that NTLM requires at least two round trips to the server in
    > each connection. Your first GET/POST/whatever request needs to have an NTLM
    > "type 1" message in the Authorize header. The server will then respond with
    > a 401, but the response will contain an NTLM "type 2" message that you use
    > to create an NTLM "type 3" response. You then send your GET/POST again, with
    > the type 3 response in the Authorize header. At that point, if all goes
    > well, you get a 2xx from the server.
    >
    > Your best bet is to use rubyntlm as a previous commenter suggested

    I admit that I'm a bit stubborn sometimes. If my wife subscribed to this
    list, I'm sure she would provide ample evidence. :)

    Since the httpclient2 2.1.2 release (2007-09-22) specifically says it
    supports "NTLM auth for WWW-Authenticate", I'm reluctant to give up on
    it. When I look at the httpclient.rb source, the request method may loop
    up to 5 times depending and the NegotiateAuth class has all the logic in
    it to handle the NTLM. Tracing through, the @www_auth.set_auth method
    never seems to be called at least using the client.get call. I tried to
    force the set_auth call like this:

    def create_request(method, uri, query, body, extheader, proxy)
    if extheader.is_a?(Hash)
    extheader = extheader.to_a
    end
    if cookies = @cookie_manager.find(uri)
    extheader << ['Cookie', cookies]
    end
    boundary = nil
    content_type = extheader.find { |key, value|
    key.downcase == 'content-type'
    }
    if content_type && content_type[1] =~ /boundary=(.+)\z/
    boundary = $1
    end
    req = HTTP::Message.new_request(method, uri, query, body, proxy,
    boundary)
    + myuser, mypassword = nil, nil
    extheader.each do |key, value|
    req.header.set(key, value)
    + myuser = value if key == "username"
    + mypassword = value if key == "password"
    end
    + @www_auth.set_auth(uri, myuser, mypassword) if !myuser.nil? &&
    !mypassword.nil?
    if content_type.nil? and !body.nil?
    req.header.set('content-type', 'application/x-www-form-urlencoded')
    end
    req
    end

    And another change in line 1703 to allow access to @www_auth,
    - attr_reader :www_auth
    + attt_accessor :www_auth

    Sorry for not providing a real diff file. I don't have it on my windows
    machine and I don't feel like fetching it at this late hour.

    Point is that @www_auth.set_auth is now called within each of the
    do_get_block calls on line 1956 of the request method but I am still
    getting a 401.1 not authorized error.

    I'm about done on following up on this though. In working through the
    certificate problems for my httpclient script yesterday, I tried the
    same changes in my Perl LWP script tonight and it is now working with
    SSL, client certificates and NTLM authentication. Since I need a working
    script, I'm moving forward now with my Perl script and dropping any more
    efforts on the Ruby version. I realize I may get a few boo/hisses by
    saying that on a Ruby list but I've already lost too many hours sleep
    already!

    If Hiroshi or anyone else gets NTLM working, please provide an example
    script to show how it's done. I'd love to use this library because I
    like what I see in httpclient.

    Regards,
    Jim
    Jim Clark, Dec 2, 2007
    #6
  7. Jim Clark

    yermej Guest

    On Dec 2, 5:07 am, Jim Clark <> wrote:

    > If Hiroshi or anyone else gets NTLM working, please provide an example
    > script to show how it's done. I'd love to use this library because I
    > like what I see in httpclient.
    >
    > Regards,
    > Jim


    I don't have an NTLM server to test against at this point, but from
    looking at the code you might want to try (with the original
    httpclient.rb) calling set_auth on the client rather than passing in
    an extra header, thusly:

    CERT_FILE = "c:/certs/jim_nopw2.pem"
    CA_CERT = "c:/ca_certs/servers_ca.cer"

    require 'rubygems'
    require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

    client = HTTPClient.new
    client.ssl_config.set_trust_ca(CA_CERT)
    client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

    client.set_auth(nil, 'DOMAIN\username', 'password')

    resp = client.get("https://some_website.com/default1.asp")
    puts resp.content
    puts resp.status

    I think I'll be able to play with this once I'm back at work tomorrow.
    I'll post more if I can get it working. As you said, all the pieces
    are there, it's just a matter of figuring out how to use them
    correctly.
    yermej, Dec 2, 2007
    #7
  8. Jim Clark

    yermej Guest

    On Dec 2, 3:11 pm, yermej <> wrote:
    > On Dec 2, 5:07 am, Jim Clark <> wrote:
    >
    > > If Hiroshi or anyone else getsNTLMworking, please provide an example
    > > script to show how it's done. I'd love to use this library because I
    > > like what I see in httpclient.

    >
    > > Regards,
    > > Jim

    >
    > I don't have anNTLMserver to test against at this point, but from
    > looking at the code you might want to try (with the original
    > httpclient.rb) calling set_auth on the client rather than passing in
    > an extra header, thusly:
    >
    > CERT_FILE = "c:/certs/jim_nopw2.pem"
    > CA_CERT = "c:/ca_certs/servers_ca.cer"
    >
    > require 'rubygems'
    > require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed
    >
    > client = HTTPClient.new
    > client.ssl_config.set_trust_ca(CA_CERT)
    > client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)
    >
    > client.set_auth(nil, 'DOMAIN\username', 'password')
    >
    > resp = client.get("https://some_website.com/default1.asp")
    > puts resp.content
    > puts resp.status
    >
    > I think I'll be able to play with this once I'm back at work tomorrow.
    > I'll post more if I can get it working. As you said, all the pieces
    > are there, it's just a matter of figuring out how to use them
    > correctly.


    I've played with this some and there seem to be a couple problems. Or
    I'm just doing things completely wrong.

    It seems that calling set_auth with the first parameter as nil should
    end up setting the @auth_default instance variable in the
    NegotiateAuth object. It does not. So, using the target URL as the
    first parameter is the first step to getting things almost working.

    Once I did that, I was getting to the response stage of the
    authentication, but it wasn't successful. At this point, that seems
    like it might be a problem with the way HTTPClient uses rubyntlm or it
    might not support the version our server is using or...

    It's a bit soon to say, but it seems that there might be a bug in
    HTTPClient. I'll post again once I've figured it out.
    yermej, Dec 3, 2007
    #8
  9. Jim Clark

    yermej Guest

    Re: NTLM authentication with httpclient - SOLVED

    It works now! I just made a stupid mistake in my testing. This is an
    example that should work:

    > > CERT_FILE = "c:/certs/jim_nopw2.pem"
    > > CA_CERT = "c:/ca_certs/servers_ca.cer"

    >
    > > require 'rubygems'
    > > require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

    >
    > > client = HTTPClient.new
    > > client.ssl_config.set_trust_ca(CA_CERT)
    > > client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

    >
    > > client.set_auth("https://some_website.com/default1.asp", 'DOMAIN\username', 'password')

    >
    > > resp = client.get("https://some_website.com/default1.asp")
    > > puts resp.content
    > > puts resp.status


    I forgot to include the domain in my previous attempts as it isn't
    required in the browser. I'm guessing the browser grabs the target/
    domain during the ntlm process and prepends it to the username. It
    wouldn't be hard to modify httpclient to do the same, but isn't
    completely necessary.

    I think it should also work to call set_auth with nil as the first
    parameter and have that set the default credentials, but that doesn't
    work the way httpclient is currently coded. NegotiateAuth#set seems to
    allow for it though. I hope this helps and will allow you to continue
    on your way with Ruby.
    yermej, Dec 3, 2007
    #9
  10. Jim Clark

    Jim Clark Guest

    Re: NTLM authentication with httpclient - SOLVED

    yermej wrote:
    > It works now! I just made a stupid mistake in my testing. This is an
    > example that should work

    Very cool and thank you!

    Regards,
    Jim
    Jim Clark, Dec 3, 2007
    #10
  11. Hi

    I'm sorry to be asking questions on such and old thread. But I am having
    similar problems. I've gotten the CA_CERT by exporting it with Firefox
    from the EWS server that I am trying to connect to. The thing I am stuck
    on is the CERT_FILE. How do I get this file. And why is the same file
    used in both of the parameters. The method seems to be expecting a key
    and a certificate.

    I've tried generating my own private key but no luck. If I leave the
    set_client_cert_file line out I don't get any errors regarding the
    certificate, but I get a 401 response from the server.

    Any advice would be appreciated.



    > CERT_FILE = "c:/certs/jim_nopw2.pem"
    > CA_CERT = "c:/ca_certs/servers_ca.cer"
    >
    > require 'rubygems'
    > require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed
    >
    > client = HTTPClient.new
    > client.ssl_config.set_trust_ca(CA_CERT)
    > client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)
    >
    > client.set_auth(nil, 'DOMAIN\username', 'password')
    >
    > resp = client.get("https://some_website.com/default1.asp")
    > puts resp.content
    > puts resp.status
    >

    --
    Posted via http://www.ruby-forum.com/.
    Tim Stephenson, Feb 27, 2010
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Carlos Fersura

    WebControls and NTLM Authentication

    Carlos Fersura, Nov 3, 2003, in forum: ASP .Net
    Replies:
    0
    Views:
    339
    Carlos Fersura
    Nov 3, 2003
  2. Pavel
    Replies:
    2
    Views:
    2,892
    Gerbrand van Dieijen
    Aug 29, 2004
  3. ravi

    httpclient authentication

    ravi, Apr 7, 2005, in forum: Java
    Replies:
    1
    Views:
    4,681
    Holla
    Apr 13, 2005
  4. Will
    Replies:
    5
    Views:
    2,610
  5. Matthijs
    Replies:
    0
    Views:
    833
    Matthijs
    Dec 10, 2008
Loading...

Share This Page