Removing duplicate entries/stories from a RSS feed?

Discussion in 'XML' started by gaikokujinkyofusho@gmail.com, Dec 6, 2006.

  1. Guest

    Hi, I have been enjoying being able to subscribe to RSS
    (http://kinja.com/user/thedigestibleaggie) for awhile and have come up
    with a fairly nice list of feeds but I have run into an annoying
    (though not critical) problem, duplicate stories. Apparently there is
    overlap with some of the sites I subscribe to so I get duplicate
    stories. Does anyone know of some sort of filter (software or online
    service) that can remove duplicate stories? Any help or suggestions
    would really be appreciated!

    Cheers

    -Gaiko
     
    , Dec 6, 2006
    #1
    1. Advertising

  2. Paul Lutus Guest

    wrote:

    > Hi, I have been enjoying being able to subscribe to RSS
    > (http://kinja.com/user/thedigestibleaggie) for awhile and have come up
    > with a fairly nice list of feeds but I have run into an annoying
    > (though not critical) problem, duplicate stories. Apparently there is
    > overlap with some of the sites I subscribe to so I get duplicate
    > stories. Does anyone know of some sort of filter (software or online
    > service) that can remove duplicate stories? Any help or suggestions
    > would really be appreciated!


    Write a script in a language that supports associative arrays (as do Java,
    Perl, Ruby, Python, and even JavaScript). Key the associative array to a
    unique key created out of elements in the various RSS feed items. Fill the
    associative array using the generated key.

    Unfortunately, it is rare for two RSS feed items to be truly identical.
    Often, they tell the same story with small differences in wording (to avoid
    accusations of plagiarism) and of course the URL is normally different.

    Without some complex coding to detect items that are almost the same, the
    above method will remove only genuinely identical items from different RSS
    feeds.

    --
    Paul Lutus
    http://www.arachnoid.com
     
    Paul Lutus, Dec 6, 2006
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. sri2097
    Replies:
    4
    Views:
    586
    sri2097
    Jan 10, 2006
  2. Replies:
    2
    Views:
    561
    Andy Dingley
    Dec 6, 2006
  3. Amit
    Replies:
    3
    Views:
    424
    Richard Herring
    Sep 5, 2005
  4. liuyuprc
    Replies:
    1
    Views:
    236
    Fredrik Lundh
    Sep 14, 2008
  5. ppnair
    Replies:
    0
    Views:
    418
    ppnair
    Oct 11, 2012
Loading...

Share This Page