NTLM authentication with httpclient

[Jim Clark]

I have rewritten my net/http script that I had questions on a couple of
days ago using httpclient but am stuck on the NTLM authentication piece.
This is what I have so far:

#!c:/ruby/bin/ruby.exe
CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"

require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

# This fetches the page but 401 error because not authenticated through NTLM
resp = client.get("https://some_website.com/default1.asp")

# Not sure how to turn on NTLM authentication and feed arguments. I
tried this but still get 401 error
#resp = client.request('GET', "https://some_website.com/default1.asp",
{'username' => 'DOMAIN\username', 'password' => 'password'})
puts resp.content
puts resp.status

Thanks again for any and all help.

-Jim

[yermej]

Post by Jim Clark
I have rewritten my net/http script that I had questions on a couple of
days ago using httpclient but am stuck on the NTLM authentication piece.
#!c:/ruby/bin/ruby.exe
CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"
require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed
client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)
# This fetches the page but 401 error because not authenticated through NTLM
resp = client.get("https://some_website.com/default1.asp")
# Not sure how to turn on NTLM authentication and feed arguments. I
tried this but still get 401 error
#resp = client.request('GET', "https://some_website.com/default1.asp",
{'username' => 'DOMAIN\username', 'password' => 'password'})
puts resp.content
puts resp.status
Thanks again for any and all help.
-Jim

There are examples included in the rubyntlm gem. I'm running Windows
so they're in C:\ruby\lib\ruby\gems\1.8\gems\rubyntlm-0.1.1\examples\.
YMMV.

[Jim Clark]

Post by yermej
There are examples included in the rubyntlm gem. I'm running Windows
so they're in C:\ruby\lib\ruby\gems\1.8\gems\rubyntlm-0.1.1\examples\.
YMMV.

I'm really trying to do this in httpclient versus getting down into
rubyntlm which shouldn't be necessary. I took another look at the
httpclient.rb source and found what I was looking for on the get method
which allows an extheader hash to be passed with arguments. Now I feel I
am real close but I am still getting a 401 error. The script is now
looking like:

#!c:/ruby/bin/ruby.exe
CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"

require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

resp = client.get("https://some_website.com/default1.asp"), nil,
{'username' => 'DOMAIN\username', 'password' => 'password'})
puts resp.content
puts resp.status

When I look at resp.content in my debugger, I see that the @header_item
array has the normal info (content type, content length, server, etc.)
and also has ["WWW-Authenticate", "Negotiate"] and the second
["WWW-Authenticate", "NTLM"]. When I trace through the get method, I see
that @negotiate_auth has a hash @auth which == {} and @auth_default
which == null. I guess if I can figure out how to populate these
correctly then I'm probably good to go.

This will have to wait until tomorrow or Sunday night and a few more
drinks. However, I'll send another S.O.S. that if someone has already
figured this out and can show me a good code snippet, I'd be more than
willing to share a few drinks (if you are around Seattle) or offer high
praise and many thanks from afar. :-)

-Jim

[Francis Cianfrocca]

Post by Jim Clark
array has the normal info (content type, content length, server, etc.)
and also has ["WWW-Authenticate", "Negotiate"] and the second
["WWW-Authenticate", "NTLM"]. When I trace through the get method, I see
which == null. I guess if I can figure out how to populate these
correctly then I'm probably good to go.

Keep in mind that NTLM requires at least two round trips to the server in
each connection. Your first GET/POST/whatever request needs to have an NTLM
"type 1" message in the Authorize header. The server will then respond with
a 401, but the response will contain an NTLM "type 2" message that you use
to create an NTLM "type 3" response. You then send your GET/POST again, with
the type 3 response in the Authorize header. At that point, if all goes
well, you get a 2xx from the server.

Your best bet is to use rubyntlm as a previous commenter suggested.

[Jim Clark]

Post by Francis Cianfrocca
Keep in mind that NTLM requires at least two round trips to the server in
each connection. Your first GET/POST/whatever request needs to have an NTLM
"type 1" message in the Authorize header. The server will then respond with
a 401, but the response will contain an NTLM "type 2" message that you use
to create an NTLM "type 3" response. You then send your GET/POST again, with
the type 3 response in the Authorize header. At that point, if all goes
well, you get a 2xx from the server.
Your best bet is to use rubyntlm as a previous commenter suggested

I admit that I'm a bit stubborn sometimes. If my wife subscribed to this
list, I'm sure she would provide ample evidence. :-)

Since the httpclient2 2.1.2 release (2007-09-22) specifically says it
supports "NTLM auth for WWW-Authenticate", I'm reluctant to give up on
it. When I look at the httpclient.rb source, the request method may loop
up to 5 times depending and the NegotiateAuth class has all the logic in
it to handle the NTLM. Tracing through, the @www_auth.set_auth method
never seems to be called at least using the client.get call. I tried to
force the set_auth call like this:

def create_request(method, uri, query, body, extheader, proxy)
if extheader.is_a?(Hash)
extheader = extheader.to_a
end
if cookies = @cookie_manager.find(uri)
extheader << ['Cookie', cookies]
end
boundary = nil
content_type = extheader.find { |key, value|
key.downcase == 'content-type'
}
if content_type && content_type[1] =~ /boundary=(.+)\z/
boundary = $1
end
req = HTTP::Message.new_request(method, uri, query, body, proxy,
boundary)
+ myuser, mypassword = nil, nil
extheader.each do |key, value|
req.header.set(key, value)
+ myuser = value if key == "username"
+ mypassword = value if key == "password"
end
+ @www_auth.set_auth(uri, myuser, mypassword) if !myuser.nil? &&
!mypassword.nil?
if content_type.nil? and !body.nil?
req.header.set('content-type', 'application/x-www-form-urlencoded')
end
req
end

And another change in line 1703 to allow access to @www_auth,
- attr_reader :www_auth
+ attt_accessor :www_auth

Sorry for not providing a real diff file. I don't have it on my windows
machine and I don't feel like fetching it at this late hour.

Point is that @www_auth.set_auth is now called within each of the
do_get_block calls on line 1956 of the request method but I am still
getting a 401.1 not authorized error.

I'm about done on following up on this though. In working through the
certificate problems for my httpclient script yesterday, I tried the
same changes in my Perl LWP script tonight and it is now working with
SSL, client certificates and NTLM authentication. Since I need a working
script, I'm moving forward now with my Perl script and dropping any more
efforts on the Ruby version. I realize I may get a few boo/hisses by
saying that on a Ruby list but I've already lost too many hours sleep
already!

If Hiroshi or anyone else gets NTLM working, please provide an example
script to show how it's done. I'd love to use this library because I
like what I see in httpclient.

Regards,
Jim

[yermej]

Post by Jim Clark
If Hiroshi or anyone else gets NTLM working, please provide an example
script to show how it's done. I'd love to use this library because I
like what I see in httpclient.
Regards,
Jim

I don't have an NTLM server to test against at this point, but from
looking at the code you might want to try (with the original
httpclient.rb) calling set_auth on the client rather than passing in
an extra header, thusly:

CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"

require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed

client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)

client.set_auth(nil, 'DOMAIN\username', 'password')

resp = client.get("https://some_website.com/default1.asp")
puts resp.content
puts resp.status

I think I'll be able to play with this once I'm back at work tomorrow.
I'll post more if I can get it working. As you said, all the pieces
are there, it's just a matter of figuring out how to use them
correctly.

[yermej]

If Hiroshi or anyone else getsNTLMworking, please provide an example
script to show how it's done. I'd love to use this library because I
like what I see in httpclient.
Regards,
Jim

I don't have anNTLMserver to test against at this point, but from
looking at the code you might want to try (with the original
httpclient.rb) calling set_auth on the client rather than passing in
CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"
require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed
client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)
client.set_auth(nil, 'DOMAIN\username', 'password')
resp = client.get("https://some_website.com/default1.asp")
puts resp.content
puts resp.status
I think I'll be able to play with this once I'm back at work tomorrow.
I'll post more if I can get it working. As you said, all the pieces
are there, it's just a matter of figuring out how to use them
correctly.

I've played with this some and there seem to be a couple problems. Or
I'm just doing things completely wrong.

It seems that calling set_auth with the first parameter as nil should
end up setting the @auth_default instance variable in the
NegotiateAuth object. It does not. So, using the target URL as the
first parameter is the first step to getting things almost working.

Once I did that, I was getting to the response stage of the
authentication, but it wasn't successful. At this point, that seems
like it might be a problem with the way HTTPClient uses rubyntlm or it
might not support the version our server is using or...

It's a bit soon to say, but it seems that there might be a bug in
HTTPClient. I'll post again once I've figured it out.

[yermej]

It works now! I just made a stupid mistake in my testing. This is an

Post by Jim Clark
CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"
require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed
client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)
client.set_auth("https://some_website.com/default1.asp", 'DOMAIN\username', 'password')
resp = client.get("https://some_website.com/default1.asp")
puts resp.content
puts resp.status

I forgot to include the domain in my previous attempts as it isn't
required in the browser. I'm guessing the browser grabs the target/
domain during the ntlm process and prepends it to the username. It
wouldn't be hard to modify httpclient to do the same, but isn't
completely necessary.

I think it should also work to call set_auth with nil as the first
parameter and have that set the default credentials, but that doesn't
work the way httpclient is currently coded. NegotiateAuth#set seems to
allow for it though. I hope this helps and will allow you to continue
on your way with Ruby.

[Jim Clark]

Post by yermej
It works now! I just made a stupid mistake in my testing. This is an
example that should work

Very cool and thank you!

Regards,
Jim

[Tim Stephenson]

Hi

I'm sorry to be asking questions on such and old thread. But I am having
similar problems. I've gotten the CA_CERT by exporting it with Firefox
from the EWS server that I am trying to connect to. The thing I am stuck
on is the CERT_FILE. How do I get this file. And why is the same file
used in both of the parameters. The method seems to be expecting a key
and a certificate.

I've tried generating my own private key but no luck. If I leave the
set_client_cert_file line out I don't get any errors regarding the
certificate, but I get a 401 response from the server.

Any advice would be appreciated.

Post by Jim Clark
CERT_FILE = "c:/certs/jim_nopw2.pem"
CA_CERT = "c:/ca_certs/servers_ca.cer"
require 'rubygems'
require 'httpclient' # using v2.1.2 and rubyntlm 0.1.1 installed
client = HTTPClient.new
client.ssl_config.set_trust_ca(CA_CERT)
client.ssl_config.set_client_cert_file(CERT_FILE, CERT_FILE)
client.set_auth(nil, 'DOMAIN\username', 'password')
resp = client.get("https://some_website.com/default1.asp")
puts resp.content
puts resp.status

[Phrogz]

Post by Jim Clark
I have rewritten my net/http script that I had questions on a couple of
days ago using httpclient but am stuck on the NTLM authentication piece.

FWIW, I've worked around this in the past by using a download of curl
that supports ntlm, and wrapped that in a shell command.

Note that if you want binary data (which it doesn't sound like you do)
and are on windows, you'll need to go through a temporary file to work
around this bug:
http://rubyforge.org/tracker/?func=detail&aid=14565&group_id=426&atid=1698