Posting over https

[Andrew Stewart]

Hello Rubyists,

I am struggling to post data over https and would be grateful for any
help.

I know the server works because I can hit it with curl and get the
response I expect:

curl -d <url-encoded form data> url

When I try to do the same from my Ruby code the server doesn't like it.

The URL to hit looks like:

https://some.host.com/foo/bar.asp?service=someservice

My code looks like:

require 'net/https'
require 'uri'

url = <as above>
uri = URI.parse(url)
request = Net::HTTP.new(uri.host, uri.port)
request.use_ssl = true
request.verify_mode = OpenSSL::SSL::VERIFY_NONE
data = { 'key1' => 'value1',
'key2' => 'value2',
'keyn' => 'valuen' }.map {|k,v| "#{URI.escape
(k.to_s)}=#{URI.escape(v.to_s)}"}.join('&')
response = request.post("#{uri.path}?#{uri.query}", data)

When I tried this with curl, I used the value of data as my url-
encoded form data and everything was fine.

Any ideas?

Thanks in advance,
Andy Stewart

 

[Aaron Patterson]

Hello Rubyists,

I am struggling to post data over https and would be grateful for any
help.

I know the server works because I can hit it with curl and get the
response I expect:

curl -d <url-encoded form data> url

When I try to do the same from my Ruby code the server doesn't like it.

The URL to hit looks like:

https://some.host.com/foo/bar.asp?service=someservice

My code looks like:

require 'net/https'
require 'uri'

url = <as above>
uri = URI.parse(url)
request = Net::HTTP.new(uri.host, uri.port)
request.use_ssl = true
request.verify_mode = OpenSSL::SSL::VERIFY_NONE
data = { 'key1' => 'value1',
'key2' => 'value2',
'keyn' => 'valuen' }.map {|k,v| "#{URI.escape
(k.to_s)}=#{URI.escape(v.to_s)}"}.join('&')
response = request.post("#{uri.path}?#{uri.query}", data)

When I tried this with curl, I used the value of data as my url-
encoded form data and everything was fine.

Any ideas?

Try using Mechanize. You can post forms over https with it, and you
won't have to deal with encoding query parameters or setting up
Net::HTTP.

http://mechanize.rubyforge.org/

--Aaron

 

[Andrew Stewart]

Hi,

Try using Mechanize. You can post forms over https with it, and you
won't have to deal with encoding query parameters or setting up
Net::HTTP.

OK, but I would still like to understand what's wrong with my code so
I can learn and improve.

Thanks anyway for the pointer,

Andy Stewart

 

[Richard Conroy]

Hi,




OK, but I would still like to understand what's wrong with my code so
I can learn and improve.

Its admirable, but I think Net::HTTP is a bit too low level, and requires a
bit too much HTTP knowledge for many applications. Mechanize seems
to be the business, and possibly Hpricot might also have similar
functionality.

I found Net::HTTP a bit too frustrating to use and the lack of good examples
bothered me a lot. It seems that the real work along these lines (programmatic
HTTP/HTML processing) is done with Mechanize & Hpricot.

 

[ara.t.howard]

Hello Rubyists,

I am struggling to post data over https and would be grateful for any help.

I know the server works because I can hit it with curl and get the response I
expect:

curl -d <url-encoded form data> url

When I try to do the same from my Ruby code the server doesn't like it.

The URL to hit looks like:

https://some.host.com/foo/bar.asp?service=someservice

My code looks like:

require 'net/https'
require 'uri'

url = <as above>
uri = URI.parse(url)
request = Net::HTTP.new(uri.host, uri.port)
request.use_ssl = true
request.verify_mode = OpenSSL::SSL::VERIFY_NONE
data = { 'key1' => 'value1',
'key2' => 'value2',
'keyn' => 'valuen' }.map {|k,v| "#{URI.escape
(k.to_s)}=#{URI.escape(v.to_s)}"}.join('&')
response = request.post("#{uri.path}?#{uri.query}", data)

When I tried this with curl, I used the value of data as my url-encoded form
data and everything was fine.

Any ideas?

Thanks in advance,
Andy Stewart

fortytwo :/var/www/html > cat form.cgi
#! /usr/local/bin/ruby
require 'cgi'
require 'yaml'
(c=CGI.new).out("type" => "text/plain"){ c.params.to_yaml }



fortytwo :/var/www/html > cat post.rb
require 'cgi'
require 'net/https'
require 'uri'

url = ARGV.shift || "https://fortytwo.merseine.nu/form.cgi"
uri = URI.parse url

data = {
'key1' => 'value1',
'key2' => 'value2',
'keyn' => 'valuen',
}

e = lambda{|x| CGI.escape x}
q = lambda{|h| h.map{|k,v| [e[k], e[v]].join '='}.join('&')}
form = q[data]

class HTTPS < Net::HTTP
def initialize *a, &b
super
self.use_ssl = true
self.verify_mode = OpenSSL::SSL::VERIFY_NONE
end
end

HTTPS.start(uri.host, uri.port) do |https|
res = https.post uri.path, form
puts res.body
end


fortytwo :/var/www/html > ruby post.rb https://fortytwo.merseine.nu/form.cgi
---
key1:
- value1
key2:
- value2
keyn:
- valuen



-a

 

[Andrew Stewart]

Hello again,

fortytwo :/var/www/html > ruby post.rb https://fortytwo.merseine.nu/
form.cgi
---
key1:
- value1
key2:
- value2
keyn:
- valuen

Many thanks for the pointers to Mechanize, Hpricot and this fully-
fledged code sample. I do appreciate your taking the time to
respond, and so quickly.

Adapting my code in the light of Ara's code above, with
'self.set_debug_output $stderr' added to the HTTPS initialize method,
I compared the client-server conversation with curl's, which I knew
worked. This led me to discover that the server expected the content-
type header to be set to 'application/x-www-form-urlencoded'.

For the record, here's the code which works with the server I am
talking to (Protx's payment gateway). Now that I have it working,
I'll take Aaron and Richard's advice and look into replacing it with
higher-level code using Mechanize.

require 'net/https'
require 'uri'

class HTTPS < Net::HTTP
def initialize *a, &b
super
self.use_ssl = true
self.verify_mode = OpenSSL::SSL::VERIFY_NONE
#self.set_debug_output $stderr
end
end

url = 'https://some.host.com/foo/bar.asp?service=someservice'
uri = URI.parse url
data = {
'key1' => 'value1',
'key2' => 'value2',
'keyn' => 'valuen'
}
e = lambda {|x| URI.escape x}
q = lambda {|h| h.map {|k,v| [e[k], e[v]].join '='}.join('&')}
form = q[data]
HTTPS.start(uri.host, uri.port) do |https|
response = https.post("#{uri.path}?#{uri.query}", form, {'content-
type' => 'application/x-www-form-urlencoded'})
puts response.body
end

Regards,
Andy Stewart