open - uri question

Discussion in 'Ruby' started by akanksha, Jul 26, 2006.

  1. akanksha

    akanksha Guest

    I am using open-uri for the first time. I need to visit a bunch of urls
    and gather some data. Here is a samll code snippet

    require 'open-uri' # allows the use of a file like API for URLs
    open( "http://no-way-outspaik375.spaces.msn.com/") { |file|
    lines = file.read
    puts lines

    }

    and here is the error I get
    ruby test.rb
    /usr/local/lib/ruby/1.8/open-uri.rb:290:in `open_http': 500 Internal
    Server Error (OpenURI::HTTPError)
    from /usr/local/lib/ruby/1.8/open-uri.rb:629:in `buffer_open'
    from /usr/local/lib/ruby/1.8/open-uri.rb:167:in `open_loop'
    from /usr/local/lib/ruby/1.8/open-uri.rb:165:in `open_loop'
    from /usr/local/lib/ruby/1.8/open-uri.rb:135:in `open_uri'
    from /usr/local/lib/ruby/1.8/open-uri.rb:531:in `open'
    from /usr/local/lib/ruby/1.8/open-uri.rb:86:in `open'
    from test.rb:2

    However
    require 'open-uri' # allows the use of a file like API for URLs
    open( "http://www.google.com/") { |file|
    lines = file.read
    puts lines

    }

    works just fine. What am I doing wrong??
     
    akanksha, Jul 26, 2006
    #1
    1. Advertising

  2. akanksha

    ChrisH Guest

    akanksha wrote:
    > I am using open-uri for the first time. I need to visit a bunch of urls
    > and gather some data. Here is a samll code snippet
    >
    > require 'open-uri' # allows the use of a file like API for URLs
    > open( "http://no-way-outspaik375.spaces.msn.com/") { |file|
    > lines = file.read
    > puts lines
    >
    > }
    >
    > and here is the error I get
    > ruby test.rb
    > /usr/local/lib/ruby/1.8/open-uri.rb:290:in `open_http': 500 Internal
    > Server Error (OpenURI::HTTPError)

    ....

    You can see some info on HTTP 500 errors here:
    http://www.checkupdown.com/status/E500.html

    Maybe the service was down?
    Or they may have it restricted to prevent scraping?
    You may need to provide some info to fool the site into
    thinking your a regular browser...

    Cheers
     
    ChrisH, Jul 26, 2006
    #2
    1. Advertising

  3. akanksha

    akanksha Guest

    > Maybe the service was down?

    The service was not down. Both urls open in a browser.



    > Or they may have it restricted to prevent scraping?
    > You may need to provide some info to fool the site into
    > thinking your a regular browser...


    How would I go about doing that ...could you plz point me to some
    info?
    Thank you.
     
    akanksha, Jul 26, 2006
    #3
  4. akanksha

    Guest

    On Thu, 27 Jul 2006, akanksha wrote:

    >> Maybe the service was down?

    >
    > The service was not down. Both urls open in a browser.
    >
    >
    >
    >> Or they may have it restricted to prevent scraping?
    >> You may need to provide some info to fool the site into
    >> thinking your a regular browser...

    >
    > How would I go about doing that ...could you plz point me to some
    > info?
    > Thank you.



    you need to set user-agent to a 'real' browser. something like 'Mozilla/4.0'

    -a
    --
    suffering increases your inner strength. also, the wishing for suffering
    makes the suffering disappear.
    - h.h. the 14th dali lama
     
    , Jul 26, 2006
    #4
  5. akanksha

    akanksha Guest

    yes that works and so does mechanize ....thanks!!!



    wrote:
    > On Thu, 27 Jul 2006, akanksha wrote:
    >
    > >> Maybe the service was down?

    > >
    > > The service was not down. Both urls open in a browser.
    > >
    > >
    > >
    > >> Or they may have it restricted to prevent scraping?
    > >> You may need to provide some info to fool the site into
    > >> thinking your a regular browser...

    > >
    > > How would I go about doing that ...could you plz point me to some
    > > info?
    > > Thank you.

    >
    >
    > you need to set user-agent to a 'real' browser. something like 'Mozilla/4.0'
    >
    > -a
    > --
    > suffering increases your inner strength. also, the wishing for suffering
    > makes the suffering disappear.
    > - h.h. the 14th dali lama
     
    akanksha, Jul 26, 2006
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Simon Harris
    Replies:
    0
    Views:
    6,383
    Simon Harris
    May 10, 2005
  2. Stanimir Stamenkov
    Replies:
    1
    Views:
    2,476
    Stanimir Stamenkov
    Aug 17, 2005
  3. Pavel
    Replies:
    2
    Views:
    1,651
    Peter Flynn
    Aug 4, 2004
  4. etheriau
    Replies:
    1
    Views:
    669
    Pavel
    Aug 23, 2004
  5. Jay 99
    Replies:
    2
    Views:
    192
    Jay 99
    Apr 4, 2009
Loading...

Share This Page