Deduping quotations

[Roedy Green]

Have you ever noticed how the quotation websites have the same
quotations with tiny variations? or the same quote attributed to
several different authors. Sometimes there is a short and long version
of the same quotation.

I was wondering how you might detect these.


I thought you might do it by converting all to lower case, stripping
punctuation and normalising white space to a single space.

Then you would remove common words.

Then you need to match, where order matters, put precise matching does
not. Just how would that work?

 

[Arne Vajhøj]

Have you ever noticed how the quotation websites have the same
quotations with tiny variations? or the same quote attributed to
several different authors. Sometimes there is a short and long version
of the same quotation.

It is common.

Poor quoting can easily spread such variations.

I was wondering how you might detect these.

I thought you might do it by converting all to lower case, stripping
punctuation and normalising white space to a single space.
>
Then you would remove common words.

Then you need to match, where order matters, put precise matching does
not. Just how would that work?

Maybe:
- only look at the very specific words
- convert those to a standard form
- test if all of those are present

Arne

 

[Tom Anderson]

Have you ever noticed how the quotation websites have the same
quotations with tiny variations? or the same quote attributed to several
different authors. Sometimes there is a short and long version of the
same quotation.

I was wondering how you might detect these.

I thought you might do it by converting all to lower case, stripping
punctuation and normalising white space to a single space.

Then computing edit distances between all pairs of quotations:

http://en.wikipedia.org/wiki/Levenshtein_distance

And reporting those with distances below a certain threshold. I would
guess that for a database of a few hundred quotations, the analysis would
take under five minutes - probably under one minute, and probably a matter
of seconds.

Lucene has an implementation of this algorithm, and i imagine it's a fast
one. If you weren't satisfied with a speed of a your own implementation
(and it's really not difficult), you could try finding and using that.

tom