Some "PERSONS" it found, everything between '' is [sic]:
'C'
'Cobol'
'B begat C.'
'”' (yes, a single fancy closing quote)
'engineer'
'’ initials'
'’70s'
'“Jobs’ genius'
Some of it is understandable given how annoying single-character identifiers are to parse and others are roughly reasonable in context.
Even given some of those quirks I bet it would be useful if you had bajillions of pages (cough ediscovery cough) to search through instead of a few general-audience articles.
” PERSON
its ORGANIZATION
’s Wear Daily PERSON
shoulder pad DISEASE
The rest is mostly correct (although random punctuation seems included at times) but it's nothing you couldn't get by word matching using a big dictionary. It reminded me to get tested for shoulder pads though, so there's that.
We have a bug with quotes/punctuation that we are fixing now (that account for 1 & 3). "its" is not necessarily a mistake, it likely refers to the owner. "shoulder pad" is a mistake due to the probabilistic nature of the algorithms used.
Took me a while but I retried. It's definitely better, to the point that I have to stretch a bit to find really bad errors. "B begat C" still makes it in as do a few similar ones but considering how pathological a case a single-letter identifier is I can forgive that.
I assume because the identifiers aren't unique that they're returned in the underlying api as text locations. Hmm now I want to throw EDGAR data at this.
Sorry Polish is not one of the supported language yet: Arabic; Chinese (Simplified); Chinese (Traditional); Cyrillic; Danish; Dutch; English; Farsi; Finnish; French; German; Greek; Hebrew; Hindi; Icelandic; Italian; Japanese; Korean; Norwegian (Bokmal); Norwegian (Nynorsk); Portuguese; Spanish; Swedish; Turkish; Urdu.
You'd get collisions if you're trying to determine the language of a single word, obviously. But a sentence or two should be no problem. You're just matching words in a dictionary.
Different languages are mostly quite distinct. It would be a more interesting challenge to recognize and understand, say, German dialects.
In fact you can already get very good results, even without a dictionary, just with a distribution of the trigrams in your input languages (which is immediate to compute from a corpus). Just compute the distribution of the sample and return the language that has the closest one.
Even a single letter frequency histogram is already a very powerful discriminator to detect language. Although I suppose scoring maximum likelihood on bi- or trigrams works better for very small (sub-sentence) bits of text.
Odd that Farsi is supported, not Polish, given that Poland's GDP is close to Iran's and that Poland has no sanctions against it. Perhaps IBM is planning to sell lots of Watson-powered systems in Afghanistan & Tajikistan?
Incidentally, Poland's GDP is several times that of Finland…
Polish as a language is more similar to other languages than Finnish or Farsi. Basically, it's easier to detect Farsi and Finnish because they're more distinct and in smaller language families than Polish, which is similar to many slavic languages.
