HTTP Proxy via HTTP Layer by Perl?

Discussion in 'Perl Misc' started by nntp, Oct 19, 2004.

  1. nntp

    nntp Guest

    I am not sure how proxy works. The basic is it got a request then forward
    it.

    I need a perl program for doing that. There are windows, binary based
    proxys. However, I have to have broadband to use it, then it will cost $50 a
    month for only fast down, but slow up. If I have a Perl/PHP proxy, and use
    web server port 80, then I only need to pay $10 web hosting fee for super
    fast connections up and down.

    I searched cpan but there is none like that. I went to hotscripts. The only
    proxy is web based. But I need http layer so that I just enter the ip and
    port in browser then I am ready to go.

    Anyone knows such programs?
    nntp, Oct 19, 2004
    #1
    1. Advertising

  2. nntp

    nntp Guest

    > I am not sure how proxy works. The basic is it got a request then forward
    > it.
    >
    > I need a perl program for doing that. There are windows, binary based
    > proxys. However, I have to have broadband to use it, then it will cost $50

    a
    > month for only fast down, but slow up. If I have a Perl/PHP proxy, and use
    > web server port 80, then I only need to pay $10 web hosting fee for super
    > fast connections up and down.
    >
    > I searched cpan but there is none like that. I went to hotscripts. The

    only
    > proxy is web based. But I need http layer so that I just enter the ip and
    > port in browser then I am ready to go.
    >
    > Anyone knows such programs?
    >


    There are some on cpan. Because I don't know much about the technical terms,
    so I am not sure which is which

    http://search.cpan.org/~rwahby/Net-HTTPTunnel-0.4/HTTPTunnel.pm
    http://search.cpan.org/~book/HTTP-Proxy-0.13/lib/HTTP/Proxy.pm

    I could not match what it does to what I am trying to do.
    nntp, Oct 19, 2004
    #2
    1. Advertising

  3. "nntp" <> wrote in
    news::

    >> I am not sure how proxy works. The basic is it got a request then
    >> forward it.
    >>
    >> I need a perl program for doing that. There are windows, binary based
    >> proxys. However, I have to have broadband to use it, then it will
    >> cost $50 a month for only fast down, but slow up. If I have a
    >> Perl/PHP proxy, and use web server port 80, then I only need to pay
    >> $10 web hosting fee for super fast connections up and down.


    I am not sure why this is relevant. This is a programming newsgroup. If you
    are working on a program, and have a question regarding that program which
    we hope is in Perl, then you post it here and get help.

    >> I searched cpan but there is none like that. I went to hotscripts.
    >> The only proxy is web based. But I need http layer so that I just
    >> enter the ip and port in browser then I am ready to go.
    >>
    >> Anyone knows such programs?


    Yes: http://www.research.att.com/~hpk/wsp/

    Sinan.
    A. Sinan Unur, Oct 19, 2004
    #3
  4. nntp

    nntp Guest

    "A. Sinan Unur" <> дÈëÓʼþ
    news:Xns9587AE698E474asu1cornelledu@132.236.56.8...
    > "nntp" <> wrote in
    > news::
    >
    > >> I am not sure how proxy works. The basic is it got a request then
    > >> forward it.
    > >>
    > >> I need a perl program for doing that. There are windows, binary based
    > >> proxys. However, I have to have broadband to use it, then it will
    > >> cost $50 a month for only fast down, but slow up. If I have a
    > >> Perl/PHP proxy, and use web server port 80, then I only need to pay
    > >> $10 web hosting fee for super fast connections up and down.

    >
    > I am not sure why this is relevant. This is a programming newsgroup. If

    you
    > are working on a program, and have a question regarding that program which
    > we hope is in Perl, then you post it here and get help.

    I posted two modules from cpan and ask if they are what I need

    >
    > >> I searched cpan but there is none like that. I went to hotscripts.
    > >> The only proxy is web based. But I need http layer so that I just
    > >> enter the ip and port in browser then I am ready to go.
    > >>
    > >> Anyone knows such programs?

    >
    > Yes: http://www.research.att.com/~hpk/wsp/

    I don't understand the program. I don't see any demo either. It looks like
    to me a fake clicking software. I am not asking for that. I need about 5-10
    private proxy servers set up around the world for a trading business. I need
    the lowest cost way.



    >
    > Sinan.
    nntp, Oct 19, 2004
    #4
  5. nntp wrote:

    > I need a perl program for doing that.


    Why must it be Perl? Seriously - if you're looking for a canned
    solution, then you should be looking for the functionality you need,
    period, and ignoring the implementation language.

    Also - why are you posting this to a Perl group? Unless you're writing
    your own proxy in Perl, this is not the place to post your question.
    This group is about *writing* Perl, not discussing applications that
    happen to have been written in Perl.

    > There are windows, binary based
    > proxys. However, I have to have broadband to use it, then it will cost $50 a
    > month for only fast down, but slow up. If I have a Perl/PHP proxy, and use
    > web server port 80, then I only need to pay $10 web hosting fee for super
    > fast connections up and down.


    I don't follow you here, and to be honest I think you're confused. Using
    a proxy that's out on the web won't magically make your dialup line as
    fast as broadband. The connection between your proxy server and the
    target site will be fast, yes, but the connection between you and the
    proxy will be your dialup connection.

    > I searched cpan but there is none like that. I went to hotscripts. The only
    > proxy is web based. But I need http layer so that I just enter the ip and
    > port in browser then I am ready to go.


    Some serious advice: Learn how the web works before you try to make it
    work better. Really. That's not meant as a slam or an attack, just
    honest advice. You appear to be very confused, and you're not making
    much sense.

    Having said that, one of the most popular cacheing servers in use right
    now is Squid:

    <http://www.squid-cache.org>

    The first link on the Squid site's "recommended reading" list looks to
    me like a fairly good introduction to cacheing. You *seriously* need to
    read this to get better acquainted with the technology:

    <http://www.mnot.net/cache_docs/>

    There are also a couple of O'Reilly books available, if you prefer dead
    trees:

    <http://book.web-cache.com>

    sherm--

    --
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Hire me! My resume: http://www.dot-app.org
    Sherm Pendley, Oct 19, 2004
    #5
  6. nntp

    nntp Guest


    > Some serious advice: Learn how the web works before you try to make it
    > work better. Really. That's not meant as a slam or an attack, just
    > honest advice. You appear to be very confused, and you're not making
    > much sense.
    >
    > Having said that, one of the most popular cacheing servers in use right
    > now is Squid:
    >

    I need proxy, not cache.
    for example, let's call the script proxy.pl
    i make it sitting on root dir, set .htaccess use proxy.pl as default index
    then i use the virtual server in apache to proint to this dir....
    i enter the url in proxy and port 80 in my IE
    then when i go to citibank.com, it go through than proxy.
    i need https as well.

    > <http://www.squid-cache.org>
    >
    > The first link on the Squid site's "recommended reading" list looks to
    > me like a fairly good introduction to cacheing. You *seriously* need to
    > read this to get better acquainted with the technology:
    >
    > <http://www.mnot.net/cache_docs/>
    >
    > There are also a couple of O'Reilly books available, if you prefer dead
    > trees:
    >
    > <http://book.web-cache.com>
    >
    > sherm--
    >
    > --
    > Cocoa programming in Perl: http://camelbones.sourceforge.net
    > Hire me! My resume: http://www.dot-app.org
    nntp, Oct 19, 2004
    #6
  7. "nntp" <> wrote in
    news::

    >
    > "A. Sinan Unur" <> дÈëÓʼþ
    > news:Xns9587AE698E474asu1cornelledu@132.236.56.8...
    >> "nntp" <> wrote in
    >> news::
    >>
    >> >> I am not sure how proxy works. The basic is it got a request then
    >> >> forward it.
    >> >>
    >> >> I need a perl program for doing that. There are windows, binary
    >> >> based proxys. However, I have to have broadband to use it, then it
    >> >> will cost $50 a month for only fast down, but slow up. If I have a
    >> >> Perl/PHP proxy, and use web server port 80, then I only need to
    >> >> pay $10 web hosting fee for super fast connections up and down.

    >>
    >> I am not sure why this is relevant. This is a programming newsgroup.
    >> If you are working on a program, and have a question regarding that
    >> program which we hope is in Perl, then you post it here and get help.


    > I posted two modules from cpan and ask if they are what I need


    How do you propose that we should be able to figure out what you _really_
    need if you refuse to specify it.

    The easiest way to see if HTTP::proxy does what you want is to write a
    program that uses it to see if it does what you want. It probably does but
    then you haven't really explained what you want to do.

    >> >> I searched cpan but there is none like that. I went to hotscripts.
    >> >> The only proxy is web based. But I need http layer so that I just
    >> >> enter the ip and port in browser then I am ready to go.
    >> >>
    >> >> Anyone knows such programs?

    >>
    >> Yes: http://www.research.att.com/~hpk/wsp/


    > I don't understand the program. I don't see any demo either. It looks
    > like to me a fake clicking software. I am not asking for that.


    It is a proxy application written in pure Perl. Thought it might have
    hepled you figure out how proxying works. You could also look at the source
    code for HTTP::proxy obviously.

    > I need about 5-10 private proxy servers set up around the world for
    > a trading business. I need the lowest cost way.


    Again, this is still not relevant.

    Sinan.
    A. Sinan Unur, Oct 19, 2004
    #7
  8. nntp

    nntp Guest

    > > I need a perl program for doing that. There are windows, binary based
    > > proxys. However, I have to have broadband to use it, then it will cost

    $50
    > a
    > > month for only fast down, but slow up. If I have a Perl/PHP proxy, and

    use
    > > web server port 80, then I only need to pay $10 web hosting fee for

    super
    > > fast connections up and down.
    > >
    > > I searched cpan but there is none like that. I went to hotscripts. The

    > only
    > > proxy is web based. But I need http layer so that I just enter the ip

    and
    > > port in browser then I am ready to go.
    > >
    > > Anyone knows such programs?
    > >

    >
    > There are some on cpan. Because I don't know much about the technical

    terms,
    > so I am not sure which is which
    >
    > http://search.cpan.org/~rwahby/Net-HTTPTunnel-0.4/HTTPTunnel.pm
    > http://search.cpan.org/~book/HTTP-Proxy-0.13/lib/HTTP/Proxy.pm
    >
    > I could not match what it does to what I am trying to do.
    >


    This is the code I got online, but it even crash in shell mod in windows, so
    I don't know if it will work in CGI enviroment

    #####################
    #!/usr/usc/bin/perl
    #-------------------------------------------------------------------------
    #-- proxy.pl - A simple http proxy server. --
    #-- --
    #-- To run, type proxy.pl [port-number] at the shell prompt. --
    #-- Default port number is 5364. --
    #-- --
    #-------------------------------------------------------------------------
    #require "sys/socket.ph";
    use Socket;

    srand (time||$$);
    #--- Define a friendly exit handler
    $SIG{'KILL'} = $SIG{QUIT} = $SIG{INT} = 'exit_handler';
    sub exit_handler {
    print "\n\n --- Proxy server is dying ...\n\n";
    close(SOCKET);
    exit;

    }
    #--- Setup socket

    $| = 1;
    $proxy_port = shift(@ARGV);
    $proxy_port = 5364 unless $proxy_port =~ /\d+/;

    $socket_format = 'S n a4 x8';
    &listen_to_port(SOCKET, $proxy_port);
    $local_host = `hostname`;
    chop($local_host);
    $local_host_ip = (gethostbyname($local_host))[4];
    print " --- Proxy server running on $local_host port: $proxy_port \n\n";
    #--- Loop forever taking requests as they come
    while (1) {
    #--- Wait for request
    print " --- Waiting to be of service ...\n";
    ($addr = accept(CHILD,SOCKET)) || die "accept $!";
    ($port,$inetaddr) = (unpack($socket_format,$addr))[1,2];
    @inetaddr = unpack('C4',$inetaddr);
    print "Connection from ", join(".", @inetaddr), " port: $port \n";
    #--- Fork a subprocess to handle request.
    #--- Parent proces continues listening.
    if (fork) {
    wait; # For now we wait for the child to finish
    next; # We wait so that printouts don't mix
    }
    #--- Read first line of request and analyze it.
    #--- Return and edited version of the first line and the request method.
    ($first,$method) = &analyze_request;
    #--- Send request to remote host
    print URL $first;
    print $first;
    while (<CHILD>) {
    print $_;
    next if (/Proxy-Connection:/);
    print URL $_;
    last if ($_ =~ /^[\s\x00]*$/);
    }
    if ($method eq "POST") {
    $data = <CHILD>;
    print $data;
    print URL $data;
    }
    print URL "\n";
    #--- Wait for response and transfer it to requestor.
    print " --- Done sending. Response: \n\n";
    $header = 1;
    $text = 0;
    while (<URL>) {
    print CHILD $_;
    if ($header || $text) { # Only print header & text lines to STDOUT
    print $_;
    if ($header && $_ =~ /^[\s\x00]*$/) {
    $header = 0;
    }
    # if ($header && $_ =~ /^Content-type: text/) {
    # $text = 1;
    # }
    }
    }
    close(URL);
    close(CHILD);
    exit; # Exit from child process
    }
    #-------------------------------------------------------------------------
    #-- analyze_request --
    #-- --
    #-- Analyze a new request. First read in first line of request. --
    #-- Read URL from it, process URL and open connection. --
    #-- Return an edited version of the first line and the request --
    #-- method. --
    #-------------------------------------------------------------------------
    sub analyze_request {
    #--- Read first line of HTTP request
    $first = <CHILD>;

    $url = ($first =~ m|(http://\S )|)[0];
    print "Request for URL: $url \n";

    #--- Check if first line is of the form GET http://host-name ...
    ($method, $remote_host, $remote_port) =
    ($first =~ m!(GET|POST|HEAD) http://([^/:]+):?(\d*)! );
    #--- If not, bad request.

    if (!$remote_host) {
    print $first;
    while (<CHILD>) {
    print $_;
    last if ($_ =~ /^[\s\x00]*$/);
    }
    print "Invalid HTTP request from ", join(".", @inetaddr), "\n";
    # print CHILD "Content-type: text/plain","\n\n";
    print CHILD "I don't understand your request.\n";
    close(CHILD);
    exit;
    }
    #--- If requested URL is the proxy server then ignore request
    $remote_ip = (gethostbyname($remote_host))[4];
    if (($remote_ip eq $local_host_ip) && ($remote_port eq $proxy_port)) {
    print $first;
    while (<CHILD>) {
    print $_;
    last if ($_ =~ /^[\s\x00]*$/);
    }
    print " --- Connection to proxy server ignored.\n";
    # print CHILD "Content-type: text/plain","\n\n";
    print CHILD "It's not nice to make me loop on myself!.\n";
    close(CHILD);
    exit;
    }
    #--- Setup connection to target host and send request
    $remote_port = "http" unless ($remote_port);
    &open_connection(URL, $remote_host, $remote_port);
    #--- Remove remote hostname from URL
    $first =~ s/http:\/\/[^\/]+//;
    ($first, $method);
    }
    #-------------------------------------------------------------------------
    #-- listen_to_port(SOCKET, $port) --
    #-- --
    #-- Create a socket that listens to a specific port --
    #-------------------------------------------------------------------------
    sub listen_to_port {
    local ($port) = $_[1];
    local ($socket_format, $proto, $packed_port, $cur, $max_requests);
    $max_requests = 3; # Max number of outstanding requests
    $socket_format = 'S n a4 x8';
    $proto = (getprotobyname('tcp'))[2];
    $packed_port = pack($socket_format, &AF_INET, $port, "\0\0\0\0");
    socket($_[0], &PF_INET, &SOCK_STREAM, $proto) || die "socket: $!";
    bind($_[0], $packed_port) || die "bind: $!";
    listen($_[0], $max_requests) || die "listen: $!";
    $cur = select($_[0]);
    $| = 1; # Disable buffering on socket.
    select($cur);
    }

    #-------------------------------------------------------------------------
    #-- open_connection(SOCKET, $remote_hostname, $port) --
    #-- --
    #-- Create a socket that connects to a certain host --
    #-- $local_host_ip is assumed to be local hostname IP address --
    #-------------------------------------------------------------------------
    sub open_connection {
    local ($remote_hostname, $port) = @_[1,2];
    local ($socket_format, $proto, $packed_port, $cur);
    local ($remote_addr, @remote_ip, $remote_ip);
    local ($local_port, $remote_port);
    if ($port !~ /^\d+$/) {
    $port = (getservbyname($port, "tcp"))[2];
    $port = 80 unless ($port);
    }
    $proto = (getprotobyname('tcp'))[2];
    $remote_addr = (gethostbyname($remote_hostname))[4];
    if (!$remote_addr) {
    die "Unknown host: $remote_hostname";
    }

    @remote_ip = unpack("C4", $remote_addr);
    $remote_ip = join(".", @remote_ip);
    print "Connecting to $remote_ip port $port.\n\n";
    $socket_format = 'S n a4 x8';
    $local_port = pack($socket_format, &AF_INET, 0, $local_host_ip);
    $remote_port = pack($socket_format, &AF_INET, $port, $remote_addr);
    socket($_[0], &AF_INET, &SOCK_STREAM, $proto) || die "socket: $!";
    bind($_[0], $local_port) || die "bind: $!";
    connect($_[0], $remote_port) || die "socket: $!";
    $cur = select($_[0]);

    $| = 1; # Disable buffering on socket.
    select($cur);
    }
    nntp, Oct 19, 2004
    #8
  9. nntp wrote:

    > I need proxy, not cache.


    Please - read the docs I pointed you to. Squid is a caching proxy, but
    you can easily - *if* you *read* the *docs* - disable its cache to turn
    it into a plain proxy.

    And your babble about making your dialup connection faster by putting a
    proxy on a $10/month hosting account is still utter nonsense.

    sherm--

    --
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Hire me! My resume: http://www.dot-app.org
    Sherm Pendley, Oct 19, 2004
    #9
  10. nntp

    nntp Guest


    > And your babble about making your dialup connection faster by putting a
    > proxy on a $10/month hosting account is still utter nonsense.
    >


    I don't use dialup.

    For example, if I buy a high speed internet to use as a windows/linux proxy,
    then I need to pay $50 monthly fee, and the speed is 3M down and 300K up.
    On a web proxy, the down/up rate is equal. Close to 1M, which is good, as
    proxy is two way in and out. So a cable proxy is only good at 300k at most.

    Thanks for the doc. I am going to read it Harder!!!.
    nntp, Oct 20, 2004
    #10
  11. nntp <> wrote:

    > This is the code I got online,



    It is written in an eight year old style.

    I wouldn't trust a programmer or program that does it the way it
    was done 8 years ago.

    It is very likely worth less than what you paid for it...


    > $local_host = `hostname`;
    > chop($local_host);



    The modern way to remove newlines is chomp().


    > sub listen_to_port {
    > local ($port) = $_[1];
    > local ($socket_format, $proto, $packed_port, $cur, $max_requests);



    The modern type of variables are lexical, rather than package, variables.

    my($port) = $_[1];
    my($socket_format, $proto, $packed_port, $cur, $max_requests);


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
    Tad McClellan, Oct 20, 2004
    #11
  12. nntp wrote:

    > For example, if I buy a high speed internet to use as a windows/linux proxy,
    > then I need to pay $50 monthly fee, and the speed is 3M down and 300K up.
    > On a web proxy, the down/up rate is equal. Close to 1M, which is good, as
    > proxy is two way in and out. So a cable proxy is only good at 300k at most.


    Okay, let me see if I can sort this out. I like a challenge. ;-)

    You want to set up a web server. If you put it on your home cable
    connection, it won't be as quick because that connection limits the
    outgoing bandwidth. (Also, although you don't mention it, most cable
    providers strictly prohibit servers.)

    Now, on this web server of yours, you're looking for a CGI that forwards
    the requests it receives to another server, retrieves the results, and
    then sends the results to the browser.

    You *might* be able to get away with a simple little script that uses
    the HTTP::proxy module. But, you'd need to run that - or most any other
    "canned" proxy application I can think of - as a daemon (aka server)
    process. That's not likely to be something you'll be allowed to do on a
    budget $10/month hosting account. You could do it with a dedicated
    server, but the cheapest I've seen that for is $79/month.

    If you want a CGI, you'll probably need to write one. You'll need to
    send the request to the remote server and retrieve the results - use
    LWP::UserAgent for that. Of course, if you want links on the returned
    pages to work, you'll need to parse the returned HTML and fix those -
    HTML::parser will help with that.

    sherm--

    --
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Hire me! My resume: http://www.dot-app.org
    Sherm Pendley, Oct 20, 2004
    #12
  13. nntp

    Joe Smith Guest

    nntp wrote:
    > I am not sure how proxy works. The basic is it got a request then forward
    > it.


    Yes, that is what a proxy server does.
    The standard http proxy requires that you inform your browser that
    you are using a proxy, so that the browser (MSIE, Netscape) will use
    the proxy for all requests.

    > However, I have to have broadband to use it, then it will cost $50 a
    > month for only fast down, but slow up.


    That is typical for residential broadband access to the Internet.
    It allows high speed for download (to your computer from the Internet)
    but has slow speed for upload (from your computer to the Internet).


    > If I have a Perl/PHP proxy, and use web server port 80, then I only need to
    > pay $10 web hosting fee for super fast connections up and down.


    I seriously doubt that. Most $10/month web sites do not allow CGI.
    Even if you are allowed to run programs there, the super fast connections
    are between the web hosting server and the rest of the Internet.
    It will *not* make the connection between your computer and anywhere
    else faster.

    > I need about 5-10 private proxy servers set up around the
    > world for a trading business. I need the lowest cost way.


    It would make sense to have mirror servers around the world, not proxy
    servers. A proxy server may make things faster if you and everyone who
    does business with you are all on the same high speed network. But since
    you're not using broadband, a proxy won't help.

    If you are serious about doing business, you should sign up for a
    business-grade shared server at a colocation site. More than $50/month.
    Perl is not the solution.
    -Joe
    Joe Smith, Oct 23, 2004
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?YzY3NjIyOA==?=

    do it in database layer or application layer

    =?Utf-8?B?YzY3NjIyOA==?=, Jan 26, 2006, in forum: ASP .Net
    Replies:
    3
    Views:
    431
    Karl Seguin [MVP]
    Jan 27, 2006
  2. et
    Replies:
    2
    Views:
    1,911
  3. dan
    Replies:
    25
    Views:
    1,334
    Uncle Bob (Robert C. Martin)
    Oct 28, 2003
  4. Universe
    Replies:
    1
    Views:
    473
    Universe
    Oct 24, 2003
  5. Dhananjay
    Replies:
    1
    Views:
    1,111
    sloan
    Dec 18, 2006
Loading...

Share This Page