Removing duplicate entries/stories from a RSS feed?

  • Thread starter gaikokujinkyofusho
  • Start date
G

gaikokujinkyofusho

Hi, I have been enjoying being able to subscribe to RSS
(http://kinja.com/user/thedigestibleaggie) for awhile and have come up
with a fairly nice list of feeds but I have run into an annoying
(though not critical) problem, duplicate stories. Apparently there is
overlap with some of the sites I subscribe to so I get duplicate
stories. Does anyone know of some sort of filter (software or online
service) that can remove duplicate stories? Any help or suggestions
would really be appreciated!

Cheers

-Gaiko
 
P

Paul Lutus

Hi, I have been enjoying being able to subscribe to RSS
(http://kinja.com/user/thedigestibleaggie) for awhile and have come up
with a fairly nice list of feeds but I have run into an annoying
(though not critical) problem, duplicate stories. Apparently there is
overlap with some of the sites I subscribe to so I get duplicate
stories. Does anyone know of some sort of filter (software or online
service) that can remove duplicate stories? Any help or suggestions
would really be appreciated!

Write a script in a language that supports associative arrays (as do Java,
Perl, Ruby, Python, and even JavaScript). Key the associative array to a
unique key created out of elements in the various RSS feed items. Fill the
associative array using the generated key.

Unfortunately, it is rare for two RSS feed items to be truly identical.
Often, they tell the same story with small differences in wording (to avoid
accusations of plagiarism) and of course the URL is normally different.

Without some complex coding to detect items that are almost the same, the
above method will remove only genuinely identical items from different RSS
feeds.
 
A

Andy Dingley

Paul said:
Unfortunately, it is rare for two RSS feed items to be truly identical.

They don't need to be - they have a link URL embedded in them and
that's perfectly adequate for disambiguation.

You might even have a feed with deliberate isPermaLink properties on
each item, for exactly that purpose.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top