Comparing two book chapters (text files)

N

Nick Matzke

Hi all,

So I have an interesting challenge. I want to compare two book
chapters, which I have in plain text format, and find out (a) percentage
similarity and (b) what has changed.

Some features make this problem different than what seems to be the
standard text-matching problem solvable with e.g. difflib. Here is what
I mean:

* there is no guarantee that single lines from each file will be
directly comparable -- e.g., if a few words are inserted into a
sentence, then a chunk of the sentence will be moved to the next line,
then a chunk of that line moved to the next, etc.

* Also, there are cases where paragraphs have been moved around,
sections re-ordered, etc. So it can't just be a "linear" match.

I imagine this kind of thing can't be all that hard in the grand scheme
of things, but I couldn't find an easily applicable solution readily
available. I have advanced beginner python skills but am not quite
where I could do this kind of thing from scratch without some guidance
about the likely functions, libraries etc. to use.

PS: I am going to have to do this for multiple book chapters so various
software packages, e.g. for windows, are not really usable.

Any help is much appreciated!!

Cheers,
Nick



--
====================================================
Nicholas J. Matzke
Ph.D. student, Graduate Student Researcher
Huelsenbeck Lab
Center for Theoretical Evolutionary Genomics
4151 VLSB (Valley Life Sciences Building)
Department of Integrative Biology
University of California, Berkeley

Lab websites:
http://ib.berkeley.edu/people/lab_detail.php?lab=54
http://fisher.berkeley.edu/cteg/hlab.html
Dept. personal page:
http://ib.berkeley.edu/people/students/person_detail.php?person=370
Lab personal page: http://fisher.berkeley.edu/cteg/members/matzke.html
Lab phone: 510-643-6299
Dept. fax: 510-643-6264
Cell phone: 510-301-0179
Email: (e-mail address removed)

Mailing address:
Department of Integrative Biology
3060 VLSB #3140
Berkeley, CA 94720-3140

-----------------------------------------------------
"[W]hen people thought the earth was flat, they were wrong. When people
thought the earth was spherical, they were wrong. But if you think that
thinking the earth is spherical is just as wrong as thinking the earth
is flat, then your view is wronger than both of them put together."

Isaac Asimov (1989). "The Relativity of Wrong." The Skeptical Inquirer,
14(1), 35-44. Fall 1989.
http://chem.tufts.edu/AnswersInScience/RelativityofWrong.htm
====================================================
 
A

andrew cooke

So I have an interesting challenge.  I want to compare two book
chapters, which I have in plain text format, and find out (a) percentage
similarity and (b) what has changed.

no idea if it will help, but i found this yesterday - http://www.nltk.org/

it's a python toolkit for natural language processing. there's a book
at http://www.nltk.org/book with much more info.

andrew
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,734
Messages
2,569,441
Members
44,832
Latest member
GlennSmall

Latest Threads

Top