Identify the language of a String literal

[javadev]

Hello all

I was wondering if the following is possible:

Having read some text from a file in the form of a String, is it
possible to identify the language in which the text is? Maybe some sort
of a getLocale() method for a String....

Rgds
SS

 

[Gordon Beaton]

Having read some text from a file in the form of a String, is it
possible to identify the language in which the text is? Maybe some
sort of a getLocale() method for a String....

The String does not encode that information anywhere.

The problem isn't trivial and there are no foolproof solutions, but
common mechanisms for identifying the language of a text are to look
for known "stop words" (lists are available online), or to calculate
letter frequencies, and in both cases compare the results with what is
known about various languages.

/gordon

 

[Adam Maass]

Hello all

I was wondering if the following is possible:

Having read some text from a file in the form of a String, is it
possible to identify the language in which the text is? Maybe some sort
of a getLocale() method for a String....

The short answer: can't be done with absolute reliability. If you have a
large(ish) sample, you might be able to make some pretty good guesses based
on character frequencies or elsehow.


-- Adam Maass