Timeout errors using Net::HTTP on Windows

Discussion in 'Ruby' started by Toby DiPasquale, Apr 17, 2007.

  1. Hi all,

    I'm having a problem here running some file upload code on Windows. I've
    installed Ruby 1.8.6 from the one-click installer on Windows 2000 Server
    in VMware and on real hardware with Windows XP. The client, when running
    on Windows, always times/errors out no matter what. Here's the error:

    c:/ruby/lib/ruby/1.8/net/http.rb:2094:in `error!': 408 "Request Timeout
    " (Net::HTTPServerException)
    from c:/ruby/lib/ruby/1.8/net/http.rb:2103:in `value'
    from put_client.rb:25:in `upload_file'
    from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
    from put_client.rb:20:in `upload_file'
    from put_client.rb:30

    Here is the client code that is causing the problems:

    <client>
    require 'net/https'

    def upload_file url, file
    uri = URI.parse url
    http = Net::HTTP.new uri.host, uri.port
    if uri.scheme == 'https'
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE
    end

    if file.respond_to? :read
    file.rewind
    stream = file
    length = File.size file.path
    else
    stream = File.open file
    length = File.size file
    end

    http.start do
    req = Net::HTTP::put.new uri.path
    req.body_stream = stream
    req.content_length = length
    resp = http.request req
    resp.value
    end
    end

    if __FILE__ == $0
    upload_file "http://localhost:8443/junk", ARGV.shift
    end
    </client>

    Here is some server code I was using to test:

    <server>
    require 'webrick'

    s = WEBrick::HTTPServer.new:)Port => 8443)

    class PutServlet < WEBrick::HTTPServlet::AbstractServlet
    def do_PUT(req, res)
    res['Content-Type'] = "text/html"
    res.body = "Uploaded #{req.body}"
    end
    end

    s.mount("/", PutServlet)
    trap("INT"){ s.shutdown }
    s.start
    </server>

    This only happens on Windows and it happens irrespective of the web
    server involved. I've tried this on Mac OS X (WEBrick and nginx) and it
    works fine. Running the client on Windows, though, I've tried this
    against WEBrick and nginx and both exhibit the same behavior when
    running the client from Windows.

    Any ideas? Thanks.

    --
    Toby DiPasquale

    --
    Posted via http://www.ruby-forum.com/.
    Toby DiPasquale, Apr 17, 2007
    #1
    1. Advertising

  2. Toby DiPasquale wrote:
    > I'm having a problem here running some file upload code on Windows. I've
    > installed Ruby 1.8.6 from the one-click installer on Windows 2000 Server
    > in VMware and on real hardware with Windows XP. The client, when running
    > on Windows, always times/errors out no matter what. Here's the error:
    >
    > c:/ruby/lib/ruby/1.8/net/http.rb:2094:in `error!': 408 "Request Timeout
    > " (Net::HTTPServerException)
    > from c:/ruby/lib/ruby/1.8/net/http.rb:2103:in `value'
    > from put_client.rb:25:in `upload_file'
    > from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
    > from put_client.rb:20:in `upload_file'
    > from put_client.rb:30


    This error is more indicative of what I've been seeing:

    c:/ruby/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill': execution expired
    (Timeout::Error)
    from c:/ruby/lib/ruby/1.8/timeout.rb:56:in `timeout'
    from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
    from c:/ruby/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
    from c:/ruby/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
    from c:/ruby/lib/ruby/1.8/net/protocol.rb:126:in `readline'
    from c:/ruby/lib/ruby/1.8/net/http.rb:2017:in `read_status_line'
    from c:/ruby/lib/ruby/1.8/net/http.rb:2006:in `read_new'
    from c:/ruby/lib/ruby/1.8/net/http.rb:1047:in `request'
    from put_client.rb:25:in `upload_file'
    from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
    from put_client.rb:21:in `upload_file'
    from put_client.rb:31


    I've been looking on the engines for answers to this issue, both before
    and after I posted, but it seems that whenever this has come up in the
    past, the query has gone unanswered or was unrelated to this particular
    problem.

    --
    Toby DiPasquale

    --
    Posted via http://www.ruby-forum.com/.
    Toby Dipasquale, Apr 17, 2007
    #2
    1. Advertising

  3. Toby Dipasquale wrote:
    > Toby DiPasquale wrote:
    >> I'm having a problem here running some file upload code on Windows. I've
    >> installed Ruby 1.8.6 from the one-click installer on Windows 2000 Server
    >> in VMware and on real hardware with Windows XP. The client, when running
    >> on Windows, always times/errors out no matter what. Here's the error:
    >>
    >> c:/ruby/lib/ruby/1.8/net/http.rb:2094:in `error!': 408 "Request Timeout
    >> " (Net::HTTPServerException)
    >> from c:/ruby/lib/ruby/1.8/net/http.rb:2103:in `value'
    >> from put_client.rb:25:in `upload_file'
    >> from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
    >> from put_client.rb:20:in `upload_file'
    >> from put_client.rb:30

    >
    > This error is more indicative of what I've been seeing:
    >
    > c:/ruby/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill': execution expired
    > (Timeout::Error)
    > from c:/ruby/lib/ruby/1.8/timeout.rb:56:in `timeout'
    > from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
    > from c:/ruby/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
    > from c:/ruby/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
    > from c:/ruby/lib/ruby/1.8/net/protocol.rb:126:in `readline'
    > from c:/ruby/lib/ruby/1.8/net/http.rb:2017:in `read_status_line'
    > from c:/ruby/lib/ruby/1.8/net/http.rb:2006:in `read_new'
    > from c:/ruby/lib/ruby/1.8/net/http.rb:1047:in `request'
    > from put_client.rb:25:in `upload_file'
    > from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
    > from put_client.rb:21:in `upload_file'
    > from put_client.rb:31
    >
    >
    > I've been looking on the engines for answers to this issue, both before
    > and after I posted, but it seems that whenever this has come up in the
    > past, the query has gone unanswered or was unrelated to this particular
    > problem.


    For posterity: as described and discussed here:

    http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/229801

    it turns out that threads and I/O don't mix in the One-Click Installer
    at this time. Neither, apparently, do they mix in the MinGW builds I
    made. Cygwin is not an option for me, so I will have to switch to a
    different language. Thanks anyway.

    --
    Toby DiPasquale

    --
    Posted via http://www.ruby-forum.com/.
    Toby Dipasquale, Apr 18, 2007
    #3
  4. Toby DiPasquale

    Emilio Tagua Guest

    On 4/17/07, Toby DiPasquale <> wrote:
    > Hi all,
    >
    > I'm having a problem here running some file upload code on Windows. I've
    > installed Ruby 1.8.6 from the one-click installer on Windows 2000 Server
    > in VMware and on real hardware with Windows XP. The client, when running
    > on Windows, always times/errors out no matter what. Here's the error:
    >
    > c:/ruby/lib/ruby/1.8/net/http.rb:2094:in `error!': 408 "Request Timeout
    > " (Net::HTTPServerException)
    > from c:/ruby/lib/ruby/1.8/net/http.rb:2103:in `value'
    > from put_client.rb:25:in `upload_file'
    > from c:/ruby/lib/ruby/1.8/net/http.rb:543:in `start'
    > from put_client.rb:20:in `upload_file'
    > from put_client.rb:30
    >
    > Here is the client code that is causing the problems:
    >
    > <client>
    > require 'net/https'
    >
    > def upload_file url, file
    > uri = URI.parse url
    > http = Net::HTTP.new uri.host, uri.port
    > if uri.scheme == 'https'
    > http.use_ssl = true
    > http.verify_mode = OpenSSL::SSL::VERIFY_NONE
    > end
    >
    > if file.respond_to? :read
    > file.rewind
    > stream = file
    > length = File.size file.path
    > else
    > stream = File.open file
    > length = File.size file
    > end
    >
    > http.start do
    > req = Net::HTTP::put.new uri.path
    > req.body_stream = stream
    > req.content_length = length
    > resp = http.request req
    > resp.value
    > end
    > end
    >
    > if __FILE__ == $0
    > upload_file "http://localhost:8443/junk", ARGV.shift
    > end
    > </client>
    >
    > Here is some server code I was using to test:
    >
    > <server>
    > require 'webrick'
    >
    > s = WEBrick::HTTPServer.new:)Port => 8443)
    >
    > class PutServlet < WEBrick::HTTPServlet::AbstractServlet
    > def do_PUT(req, res)
    > res['Content-Type'] = "text/html"
    > res.body = "Uploaded #{req.body}"
    > end
    > end
    >
    > s.mount("/", PutServlet)
    > trap("INT"){ s.shutdown }
    > s.start
    > </server>
    >
    > This only happens on Windows and it happens irrespective of the web
    > server involved. I've tried this on Mac OS X (WEBrick and nginx) and it
    > works fine. Running the client on Windows, though, I've tried this
    > against WEBrick and nginx and both exhibit the same behavior when
    > running the client from Windows.
    >
    > Any ideas? Thanks.


    You could try something like:
    def method
    begin
    response = Timeout::timeout(5) { Net::(whatever u want from NET Class) }
    ...
    rescue Timeout::Error
    puts 'Timeout!!!'
    ....
    end
    end

    Hope it helps you!
    Emilio Tagua, Apr 18, 2007
    #4
  5. Emilio Tagua wrote:
    > On 4/17/07, Toby DiPasquale <> wrote:
    >> from put_client.rb:25:in `upload_file'
    >> uri = URI.parse url
    >> else
    >> end
    >> require 'webrick'
    >> s.mount("/", PutServlet)
    >> Any ideas? Thanks.

    > You could try something like:
    > def method
    > begin
    > response = Timeout::timeout(5) { Net::(whatever u want from NET
    > Class) }
    > ...
    > rescue Timeout::Error
    > puts 'Timeout!!!'
    > ....
    > end
    > end
    >
    > Hope it helps you!


    This would not help as the problem it the use of the timeout module in
    the first place. The net/protocol module already wraps the call in a
    Timeout::timeout() call and this is the source of the issue. That call
    spawns a Ruby thread to enforce the timeout and the combination of the
    thread use and I/O is what is triggering the problem. In any case, I've
    already rewritten the code in Python and it works fine on all platforms.
    Thanks anyway.

    --
    Toby DiPasquale

    --
    Posted via http://www.ruby-forum.com/.
    Toby Dipasquale, Apr 23, 2007
    #5
  6. Toby DiPasquale

    Marcin Coles Guest

    Well, I'm no expert, and I certainly wasn't going to learn another
    language, so I decided to do some tests because this was a problem for
    me too.

    I went into protocol.rb, to the rbuf_fill method (where it actually
    starts the timeout thread).

    the code there was

    def rbuf_fill
    timeout(@read_timeout) {
    @rbuf << @io.sysread(1024)
    }
    end

    now timeout takes 2 parameters - a time in seconds(?) and an exception
    class to raise (defaults to Error).

    When I changed the code to the following, it started to work for me.
    New code:

    def rbuf_fill
    timeout(@read_timeout,ProtocolError) {
    @rbuf << @io.sysread(1024)
    }
    end

    Obviously this is not exactly extensively tested - but an interesting
    results.

    Marcin


    Toby Dipasquale wrote:
    > Emilio Tagua wrote:
    >> On 4/17/07, Toby DiPasquale <> wrote:
    >>> from put_client.rb:25:in `upload_file'
    >>> uri = URI.parse url
    >>> else
    >>> end
    >>> require 'webrick'
    >>> s.mount("/", PutServlet)
    >>> Any ideas? Thanks.

    >> You could try something like:
    >> def method
    >> begin
    >> response = Timeout::timeout(5) { Net::(whatever u want from NET
    >> Class) }
    >> ...
    >> rescue Timeout::Error
    >> puts 'Timeout!!!'
    >> ....
    >> end
    >> end
    >>
    >> Hope it helps you!

    >
    > This would not help as the problem it the use of the timeout module in
    > the first place. The net/protocol module already wraps the call in a
    > Timeout::timeout() call and this is the source of the issue. That call
    > spawns a Ruby thread to enforce the timeout and the combination of the
    > thread use and I/O is what is triggering the problem. In any case, I've
    > already rewritten the code in Python and it works fine on all platforms.
    > Thanks anyway.
    >
    > --
    > Toby DiPasquale



    --
    Posted via http://www.ruby-forum.com/.
    Marcin Coles, May 1, 2007
    #6
  7. Interesting results!!! What an understatement!! How on earth did you
    figure that out? It worked like a charm. Initially my problem was just
    that the HTTP request took FOREVER... then i started randomly getting
    the timeout errors mentioned above. I went into the net/protocol.rb file
    and made this change and voila!. Not only did the errors go away, but
    the request takes literally 1/30th of the time it did before it
    eventually got the data or even timed out.. Marcin Coles, you are the
    genius of the day! Thank you!

    Marcin Coles wrote:
    > Well, I'm no expert, and I certainly wasn't going to learn another
    > language, so I decided to do some tests because this was a problem for
    > me too.
    >
    > I went into protocol.rb, to the rbuf_fill method (where it actually
    > starts the timeout thread).
    >
    > the code there was
    >
    > def rbuf_fill
    > timeout(@read_timeout) {
    > @rbuf << @io.sysread(1024)
    > }
    > end
    >
    > now timeout takes 2 parameters - a time in seconds(?) and an exception
    > class to raise (defaults to Error).
    >
    > When I changed the code to the following, it started to work for me.
    > New code:
    >
    > def rbuf_fill
    > timeout(@read_timeout,ProtocolError) {
    > @rbuf << @io.sysread(1024)
    > }
    > end
    >
    > Obviously this is not exactly extensively tested - but an interesting
    > results.
    >
    > Marcin


    --
    Posted via http://www.ruby-forum.com/.
    Ben Brightwell, Jun 5, 2007
    #7
  8. Toby DiPasquale

    Marcin Coles Guest

    Hey Ben
    I'm glad it worked for you too, and thanks for the compliment. I was
    happy to just get rid of those untrappable errors, so I didn't even
    profile to see whether there were any other speed improvements -
    performance is not a critical issue in my script.

    Cheers,
    M

    Ben Brightwell wrote:
    > Interesting results!!! What an understatement!! How on earth did you
    > figure that out? It worked like a charm. Initially my problem was just
    > that the HTTP request took FOREVER... then i started randomly getting
    > the timeout errors mentioned above. I went into the net/protocol.rb file
    > and made this change and voila!. Not only did the errors go away, but
    > the request takes literally 1/30th of the time it did before it
    > eventually got the data or even timed out.. Marcin Coles, you are the
    > genius of the day! Thank you!
    >
    > Marcin Coles wrote:
    >> Well, I'm no expert, and I certainly wasn't going to learn another
    >> language, so I decided to do some tests because this was a problem for
    >> me too.
    >>
    >> I went into protocol.rb, to the rbuf_fill method (where it actually
    >> starts the timeout thread).
    >>
    >> the code there was
    >>
    >> def rbuf_fill
    >> timeout(@read_timeout) {
    >> @rbuf << @io.sysread(1024)
    >> }
    >> end
    >>
    >> now timeout takes 2 parameters - a time in seconds(?) and an exception
    >> class to raise (defaults to Error).
    >>
    >> When I changed the code to the following, it started to work for me.
    >> New code:
    >>
    >> def rbuf_fill
    >> timeout(@read_timeout,ProtocolError) {
    >> @rbuf << @io.sysread(1024)
    >> }
    >> end
    >>
    >> Obviously this is not exactly extensively tested - but an interesting
    >> results.
    >>
    >> Marcin



    --
    Posted via http://www.ruby-forum.com/.
    Marcin Coles, Jun 5, 2007
    #8
  9. Toby DiPasquale

    Raghu Kumar Guest

    Hi,

    Thanks a lot... I was struck with this problem ... now I have no
    problem.

    But in my project I have lot of different web pages to scrap, is
    anything possible to tweak so that responses are fast ?

    Raghu

    Ben Brightwell wrote:
    > Interesting results!!! What an understatement!! How on earth did you
    > figure that out? It worked like a charm. Initially my problem was just
    > that the HTTP request took FOREVER... then i started randomly getting
    > the timeout errors mentioned above. I went into the net/protocol.rb file
    > and made this change and voila!. Not only did the errors go away, but
    > the request takes literally 1/30th of the time it did before it
    > eventually got the data or even timed out.. Marcin Coles, you are the
    > genius of the day! Thank you!
    >
    > Marcin Coles wrote:
    >> Well, I'm no expert, and I certainly wasn't going to learn another
    >> language, so I decided to do some tests because this was a problem for
    >> me too.
    >>
    >> I went into protocol.rb, to the rbuf_fill method (where it actually
    >> starts the timeout thread).
    >>
    >> the code there was
    >>
    >> def rbuf_fill
    >> timeout(@read_timeout) {
    >> @rbuf << @io.sysread(1024)
    >> }
    >> end
    >>
    >> now timeout takes 2 parameters - a time in seconds(?) and an exception
    >> class to raise (defaults to Error).
    >>
    >> When I changed the code to the following, it started to work for me.
    >> New code:
    >>
    >> def rbuf_fill
    >> timeout(@read_timeout,ProtocolError) {
    >> @rbuf << @io.sysread(1024)
    >> }
    >> end
    >>
    >> Obviously this is not exactly extensively tested - but an interesting
    >> results.
    >>
    >> Marcin



    --
    Posted via http://www.ruby-forum.com/.
    Raghu Kumar, Jun 26, 2007
    #9
  10. Hi Raghu,

    Since this weird Net timeout issue has a solution now, I'd recommend
    that you spin out a series of threads to scrape your websites. That way,
    even if each thread spends a long time waiting on Net:HTTP, you still
    have lots of other threads working on other sites - aggregate wait time
    goes way down if you have 100 simultaneous HTTP requests all waiting on
    different sites. Be careful not to have too many threads request from
    the same site at the same time - not only is this bad manners but some
    systems will block you instantly if you do this.

    Pragprog guys have a good little tutorial on threading which even
    includes some info on net requests (though be sure to make the changes
    described above for execution expired issues first):

    http://www.rubycentral.com/book/tut_threads.html

    Steve

    Raghu Kumar wrote:
    > Hi,
    >
    > Thanks a lot... I was struck with this problem ... now I have no
    > problem.
    >
    > But in my project I have lot of different web pages to scrap, is
    > anything possible to tweak so that responses are fast ?
    >
    > Raghu
    >
    > Ben Brightwell wrote:
    >> Interesting results!!! What an understatement!! How on earth did you
    >> figure that out? It worked like a charm. Initially my problem was just
    >> that the HTTP request took FOREVER... then i started randomly getting
    >> the timeout errors mentioned above. I went into the net/protocol.rb file
    >> and made this change and voila!. Not only did the errors go away, but
    >> the request takes literally 1/30th of the time it did before it
    >> eventually got the data or even timed out.. Marcin Coles, you are the
    >> genius of the day! Thank you!
    >>
    >> Marcin Coles wrote:
    >>> Well, I'm no expert, and I certainly wasn't going to learn another
    >>> language, so I decided to do some tests because this was a problem for
    >>> me too.
    >>>
    >>> I went into protocol.rb, to the rbuf_fill method (where it actually
    >>> starts the timeout thread).
    >>>
    >>> the code there was
    >>>
    >>> def rbuf_fill
    >>> timeout(@read_timeout) {
    >>> @rbuf << @io.sysread(1024)
    >>> }
    >>> end
    >>>
    >>> now timeout takes 2 parameters - a time in seconds(?) and an exception
    >>> class to raise (defaults to Error).
    >>>
    >>> When I changed the code to the following, it started to work for me.
    >>> New code:
    >>>
    >>> def rbuf_fill
    >>> timeout(@read_timeout,ProtocolError) {
    >>> @rbuf << @io.sysread(1024)
    >>> }
    >>> end
    >>>
    >>> Obviously this is not exactly extensively tested - but an interesting
    >>> results.
    >>>
    >>> Marcin



    --
    Posted via http://www.ruby-forum.com/.
    Steve Midgley, Jun 30, 2007
    #10
  11. Toby DiPasquale

    Roger Pack Guest

    > This error is more indicative of what I've been seeing:

    > c:/ruby/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill': execution expired
    > (Timeout::Error)
    > from c:/ruby/lib/ruby/1.8/timeout.rb:56:in `timeout'



    > When I changed the code to the following, it started to work for me.
    > New code:
    >
    > def rbuf_fill
    > timeout(@read_timeout,ProtocolError) {
    > @rbuf << @io.sysread(1024)
    > }
    > end



    Anybody know if this was ever fixed? I'm unable to recreate it currently
    [maybe my boxes are too new so they don't trigger the problem?]

    > For posterity: as described and discussed here:
    > http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/229801
    > it turns out that threads and I/O don't mix in the One-Click Installer
    > at this time. Neither, apparently, do they mix in the MinGW builds...


    I don't think this was the root cause, however--the problem described
    here is that if you have nested timeout's
    Timeout.timeout(x) {
    begin
    Timeout.timeout(y) { stuff }
    rescue Timeout::Error
    # which timeout will cause this to trigger?
    end
    }

    The outermost timeout could timeout the inner one, causing the inner one
    to think it was "timed out" when it wasn't--the outer one was. Timeout
    and/or Thread#raise should be deprecated, I think.
    =r
    --
    Posted via http://www.ruby-forum.com/.
    Roger Pack, Aug 1, 2009
    #11
  12. Toby DiPasquale

    James Gray Guest

    On Aug 1, 2009, at 7:05 AM, Roger Pack wrote:

    >> For posterity: as described and discussed here:
    >> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/229801
    >> it turns out that threads and I/O don't mix in the One-Click
    >> Installer
    >> at this time. Neither, apparently, do they mix in the MinGW builds...

    >
    > I don't think this was the root cause, however--the problem described
    > here is that if you have nested timeout's
    > Timeout.timeout(x) {
    > begin
    > Timeout.timeout(y) { stuff }
    > rescue Timeout::Error
    > # which timeout will cause this to trigger?
    > end
    > }
    >
    > The outermost timeout could timeout the inner one, causing the inner
    > one
    > to think it was "timed out" when it wasn't--the outer one was.


    For information about how to correctly nest timeout() calls, please see:

    http://blog.segment7.net/articles/2006/04/11/care-and-feeding-of-timeout-timeout

    James Edward Gray II
    James Gray, Aug 1, 2009
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mark Goldin

    Errors, errors, errors

    Mark Goldin, Jan 17, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    920
    Mark Goldin
    Jan 17, 2004
  2. SenthilVel
    Replies:
    0
    Views:
    929
    SenthilVel
    Jun 7, 2006
  3. Lance Wynn
    Replies:
    1
    Views:
    1,811
    Lance Wynn
    Feb 3, 2008
  4. Freddie
    Replies:
    0
    Views:
    223
    Freddie
    Jun 29, 2004
  5. Mark Probert

    Timeout::timeout and Socket timeout

    Mark Probert, Oct 6, 2004, in forum: Ruby
    Replies:
    1
    Views:
    1,267
    Brian Candler
    Oct 6, 2004
Loading...

Share This Page