Proxy Server troubles

T

Tanner Burson

I've been working on a proxy server implementation for a project idea
I have. But I've found that it only seems to work on certain pages,
and not on others. I've distilled the problem down to a simple
implementation, but it still occurs. The code below will return a
(mostly) correct website if pointed at 'www.msn.com' but if pointed at
say 'slashdot.org' the browser refuses to display it, in both cases
the page is loaded correctly. If anyone has any ideas I'd really
appreciate it.

===proxy.rb

require 'socket'
require 'net/http'
local_host = '192.168.0.6'
local_port = 8080

host = 'slashdot.org'
server = TCPServer.new(local_host,local_port)
while true
Thread.start(server.accept) do |s|
data = ''
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
p "====Received: #{data}"
net = Net::HTTP.new(host,80)
response = net.get('/')
s << "HTTP/#{response.http_version}
#{response.code}/#{response.message} \r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end

====end
 
J

jm

I gave it a quick run. I had to change
local_host = nil
so that it would bind and fixed the line wrap in the posted code.
Telnetted to port 8080 on localhost and hit enter twice and it came
back with a lot of html from slashdot. So it works on

$ ruby -v
ruby 1.8.1 (2003-12-25) [powerpc-darwin]

Can't see anything obviously wrong. Do you have a transparent proxy or
anything between you and the site as this can cause havoc?

J.
 
J

James Britt

jm said:
I gave it a quick run. I had to change
local_host = nil
so that it would bind and fixed the line wrap in the posted code.
Telnetted to port 8080 on localhost and hit enter twice and it came back
with a lot of html from slashdot.
Click to expand...

I managed the same, on win32, but could not get the results to render in
Firefox or IE. It fetches the page OK (judging by the debugging p
calls), but whatever is sent back to the browser is off in some way.



James
 
T

Tanner Burson

I gave it a quick run. I had to change
local_host = nil
so that it would bind and fixed the line wrap in the posted code.
Telnetted to port 8080 on localhost and hit enter twice and it came
back with a lot of html from slashdot. So it works on

$ ruby -v
ruby 1.8.1 (2003-12-25) [powerpc-darwin]

Can't see anything obviously wrong. Do you have a transparent proxy or
anything between you and the site as this can cause havoc?
Click to expand...


Not that I'm aware of. I should have been more specific the first
time, try setting it as a proxy in your browser settings, I've yet to
get it to load correctly in a browser.
 
T

Tanner Burson

I managed the same, on win32, but could not get the results to render in
Firefox or IE. It fetches the page OK (judging by the debugging p
calls), but whatever is sent back to the browser is off in some way.
Click to expand...

This so far has been exactly the problem. Some sites (ie: msn.com )
work just fine, but others won't work at all. I've compared the
output of both requests but haven't been able to determine any
difference other than possibly the amount of content being sent back.
I'm about at my wit's end over this one, any advice would be great.
 
T

Tanner Burson

* Tanner Burson <[email protected]> [2005-04-22 10:49:35 +0900]:

Have you tried setting your HTTP_USER_AGENT to something like:

Mozilla/5.0
Click to expand...

When I'm using the full script (not included for brevity), it runs as
a true proxy and relays all the request headers straight on through to
the server. The code I provided is just a simple sample that still
portrays the same problems. It doesn't appear to be anything related
to the request itself, as the response comes back successfully. There
is just something about it the browser doesn't like.

My first thought was that it didn't like receiving the whole page at
once, so I tried reading the body in chunks and sending it on, but it
didn't seem to solve it either.
 
J

jm

I managed the same, on win32, but could not get the results to render
in Firefox or IE. It fetches the page OK (judging by the debugging p
calls), but whatever is sent back to the browser is off in some way.
Click to expand...

Alright. this is wierd I just tried it with firefox on macosx and it
rendered find, except for all the broken images, etc due to the
reference to '/', ie

net = Net::HTTP.new(host,80)
response = net.get('/')

Made a few modification. Really needs a clean up and there's probably
libraries out there which will do this cleaner. I've been staring at
perl code most of the day wishing it wasn't . Anyway, I've included the
full script below.

require 'socket'
require 'net/http'
local_host = nil
local_port = 8080

#
# returns proto,host,path or nil
#
def url_split(h)
reg = Regexp.new('^(\w*)\:\/\/([\w\.]+)(\/.*)')
md = reg.match(h)
return nil unless md
return md[1],md[2],md[3]
end

host = 'slashdot.org'
server = TCPServer.new(local_host,local_port)
while true
Thread.start(server.accept) do |s|
header = ''
data = ''
# this is a little crude
header = s.gets # added
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
header = header.split(' ') # added
proto,host,path = url_split(header[1]) # added
p "==== header[1]: #{header[1]}" # added
p "==== #{proto} #{host} #{path}" # added
p "====Received: #{data}"
net = Net::HTTP.new(host,80)
response = net.get(path) # changed
s << "HTTP/#{response.http_version}
#{response.code}/#{response.message} \r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end
 
J

Jim Freeze

* jm said:
Alright. this is wierd I just tried it with firefox on macosx and it
rendered find, except for all the broken images, etc due to the
reference to '/', ie
Click to expand...

Uhh, this may be another dumb suggestion, but are you
by chance sending inspect data with #p instead of #puts
to the browser?
 
C

Craig Moran

I am behind a proxy server at work, yet got this to work. I commented
out your Net::HTTP.new line and added what I believe to be the Proxy
equivalent. Then I set Mozilla Firefox's proxy to be localhost:8080.
NOTE: I do not have the ability to use another machine to test as I
am very restricted with net access here. I launched slashdot.org and
monitored the proxy.rb window while all of the HTML flew by. It came
up and rendered properly in the browser.

I am on Windows 2000
C:\>ruby -v
ruby 1.8.2 (2004-12-25) [i386-mswin32]

Here's the code. Hope it helps:

require 'socket'
require 'net/http'
local_host = 'localhost'
local_port = 8080

host = 'slashdot.org'
server = TCPServer.new(local_host, local_port)
while true
Thread.start(server.accept) do |s|
data = ''
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
p "====Received: #{data}"
# net = Net::HTTP.new(host,80)
net = Net::HTTP::proxy("proxy", "80").start(host)
response = net.get('/')
s <<
"HTTP/#{response.http_version}#{response.code}/#{response.message}
\r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end
 
C

Craig Moran

I neglected to mention that if you use the proxy version of Net::HTTP,
you can get it to work without a proxy by putting in "" for the proxy
and port. Here's the proxy version:

net = Net::HTTP::proxy("proxy", "80").start(host)

and the non-proxy version:

net = Net::HTTP::proxy("", "").start(host)

HTH,
Craig

I am behind a proxy server at work, yet got this to work. I commented
out your Net::HTTP.new line and added what I believe to be the Proxy
equivalent. Then I set Mozilla Firefox's proxy to be localhost:8080.
NOTE: I do not have the ability to use another machine to test as I
am very restricted with net access here. I launched slashdot.org and
monitored the proxy.rb window while all of the HTML flew by. It came
up and rendered properly in the browser.

I am on Windows 2000
C:\>ruby -v
ruby 1.8.2 (2004-12-25) [i386-mswin32]

Here's the code. Hope it helps:

require 'socket'
require 'net/http'
local_host = 'localhost'
local_port = 8080

host = 'slashdot.org'
server = TCPServer.new(local_host, local_port)
while true
Thread.start(server.accept) do |s|
data = ''
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
p "====Received: #{data}"
# net = Net::HTTP.new(host,80)
net = Net::HTTP::proxy("proxy", "80").start(host)
response = net.get('/')
s <<
"HTTP/#{response.http_version}#{response.code}/#{response.message}
\r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end

I've been working on a proxy server implementation for a project idea
I have. But I've found that it only seems to work on certain pages,
and not on others. I've distilled the problem down to a simple
implementation, but it still occurs. The code below will return a
(mostly) correct website if pointed at 'www.msn.com' but if pointed at
say 'slashdot.org' the browser refuses to display it, in both cases
the page is loaded correctly. If anyone has any ideas I'd really
appreciate it.

===proxy.rb

require 'socket'
require 'net/http'
local_host = '192.168.0.6'
local_port = 8080

host = 'slashdot.org'
server = TCPServer.new(local_host,local_port)
while true
Thread.start(server.accept) do |s|
data = ''
while s.gets && !$_.nil?
data += $_
break if $_ == "\r\n" || $_ == "\n"
end
p "====Received: #{data}"
net = Net::HTTP.new(host,80)
response = net.get('/')
s << "HTTP/#{response.http_version}
#{response.code}/#{response.message} \r\n"
p "====Got back"
response.each_header do |key,value|
tmp = key + ": " + value + "\r\n"
p tmp
s << tmp
end
p response.body
s << "\r\n" << response.body
s.close # We're done with this request, bring on the next.
end
end

====end
Click to expand...
Click to expand...
 
T

Tanner Burson

I neglected to mention that if you use the proxy version of Net::HTTP,
you can get it to work without a proxy by putting in "" for the proxy
and port. Here's the proxy version:

net = Net::HTTP::proxy("proxy", "80").start(host)

and the non-proxy version:

net = Net::HTTP::proxy("", "").start(host)
Click to expand...

This hangs until the timeout is reached. But if I use nil instead of
empty string it continues, but still fails loading correctly in the
browser. But I think it may have to do with the headers sent with a
proxied request, can you send me the sent headers portion of the
output when used with a working proxy server?
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Help with raycaster 0
Count of http requests 0
xmlrpc client through proxy to https server 0
How to fetch Cookie from response 2
Nuby: using net/http*s* from behind a proxy server 0
Errors on REXML reading an HTML. 1
Performance issues with net/http 0
proxy testing code with multi-thread 0

Members online

No members online now.
Total: 313 (members: 0, guests: 313)
Robots: 306

Forum statistics

Threads
474,431
Messages
2,571,677
Members
48,796
Latest member
Greg L.

Latest Threads

Top