how come i cant grab all rss items from a feed

Discussion in 'Ruby' started by Adam Akhtar, Apr 4, 2009.

  1. Adam Akhtar

    Adam Akhtar Guest

    When i try and access the bbc feed it will only return the latest 40
    results. However if i use the same feed in google reader, it can return
    a lot more. Why is this and how do i modify the code below so that i can
    return more results.

    Heres some code

    require 'rss/1.0'
    require 'rss/2.0'
    require 'open-uri'
    require 'rss/parser'


    source =
    "http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/rss.xml"
    # url or local file
    content = "" # raw content of rss feed will be loaded here
    open(source) do |s| content = s.read end
    rss = RSS::parser.parse(content, false)

    (rss.items.length == 40ish)
    --
    Posted via http://www.ruby-forum.com/.
     
    Adam Akhtar, Apr 4, 2009
    #1
    1. Advertising

  2. Adam Akhtar

    Ben Lovell Guest

    [Note: parts of this message were removed to make it a legal post.]

    I'd suggest you check which headers your browser is sending. Particularly
    the Last-Modified and Etag. Replicate those and you will see the same amount
    of items.
    Ben

    On Sat, Apr 4, 2009 at 11:28 AM, Adam Akhtar <>wrote:

    > When i try and access the bbc feed it will only return the latest 40
    > results. However if i use the same feed in google reader, it can return
    > a lot more. Why is this and how do i modify the code below so that i can
    > return more results.
    >
    > Heres some code
    >
    > require 'rss/1.0'
    > require 'rss/2.0'
    > require 'open-uri'
    > require 'rss/parser'
    >
    >
    > source =
    > "http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/rss.xml"
    > # url or local file
    > content = "" # raw content of rss feed will be loaded here
    > open(source) do |s| content = s.read end
    > rss = RSS::parser.parse(content, false)
    >
    > (rss.items.length == 40ish)
    > --
    > Posted via http://www.ruby-forum.com/.
    >
    >
     
    Ben Lovell, Apr 4, 2009
    #2
    1. Advertising

  3. Adam Akhtar

    Adam Akhtar Guest

    hi ben, thanks for the reply. Im a bit confused though.

    When i type the address into the browser it shows 40 items just as the
    code above does.

    When i access the feed via google reader it shows more. I wondering what
    magic google is using to get more items than my browser or my rss code
    can.

    --
    Posted via http://www.ruby-forum.com/.
     
    Adam Akhtar, Apr 4, 2009
    #3
  4. Adam Akhtar wrote:
    > When i access the feed via google reader it shows more. I wondering what
    > magic google is using to get more items than my browser or my rss code
    > can.


    I think this could be because google caches items internally, from the
    moment someone subscribes to the feed for the first time. Then it just
    periodically updates the feed and caches new items. To simulate google
    behavior you would need to go back in time and start fetching items from
    the moment you need.

    Unless there's some api on the feed itself, in which case you'd have to
    ask owners of the feed.
    --
    Posted via http://www.ruby-forum.com/.
     
    Snaury Miyoto, Apr 4, 2009
    #4
  5. Adam Akhtar

    Ben Lovell Guest

    [Note: parts of this message were removed to make it a legal post.]

    >
    >
    > When i access the feed via google reader it shows more. I wondering what
    > magic google is using to get more items than my browser or my rss code
    > can.
    >
    >

    Ah I have to admit I glossed over the fact you were using Google Reader. As
    Snaury said, they would certainly employ caching at their end which should
    display results beyond the ones you are seeing.

    Ben
     
    Ben Lovell, Apr 4, 2009
    #5
  6. Adam Akhtar

    Adam Akhtar Guest

    ahhh thank you for confirming my suspicions. Customized Deloreans with
    flux capacitors are a bit hard to come by these days so I guess ill have
    to settle with 40 a day.

    Thank you for your help!

    --
    Posted via http://www.ruby-forum.com/.
     
    Adam Akhtar, Apr 5, 2009
    #6
  7. Adam Akhtar

    Ben Lovell Guest

    [Note: parts of this message were removed to make it a legal post.]

    If it is utterly important you _could_ possibly use the google reader api to
    get at those entries. Might be a little easier than going back in time ;)

    On Sun, Apr 5, 2009 at 4:16 AM, Adam Akhtar <> wrote:

    > ahhh thank you for confirming my suspicions. Customized Deloreans with
    > flux capacitors are a bit hard to come by these days so I guess ill have
    > to settle with 40 a day.
    >
    > Thank you for your help!
    >
    > --
    > Posted via http://www.ruby-forum.com/.
    >
    >
     
    Ben Lovell, Apr 5, 2009
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    5
    Views:
    797
    SpaceGirl
    Feb 25, 2005
  2. Motta
    Replies:
    1
    Views:
    559
    Andy Dingley
    Jun 9, 2004
  3. Scott Gordo
    Replies:
    5
    Views:
    739
  4. Nagaraj
    Replies:
    1
    Views:
    893
    Lionel B
    Mar 1, 2007
  5. Jonathan Groll
    Replies:
    1
    Views:
    297
    Kouhei Sutou
    Jun 27, 2009
Loading...

Share This Page