Language Identification

We introduce the language identification problem-a vital component in a multilingual text analysis pipeline. We discuss the document and word level formulations of the language identification task, briefly discuss supervised solutions, and then present low-supervision methods based on polyglot training that are highly applicable in low-resource settings. We then discuss code mixing, a linguistic phenomenon common in bilingual and multilingual speakers. We extend our language identification methods to model code mixing and measure the extent of English-Hindi code mixing in various social media data sets.