An Ensemble BERT Model for English-Hindi-Marathi Multilingual Question Answering

Multilingual Question Answering (MQA) is an application of NLP that deals with retrieving an accurate answer to the user’s question from the context despite the language of the question and context. Currently, multilingual-cased BERT models perform well for all fine-tuning settings, such as zero-shot, monolingual, cross-lingual, and multilingual settings of multilingual tasks of NLP. Of which the recent multilingual-cased BERT models are mBERT and IndicBERT, pre-trained on multiple languages of the world (104 languages) and India (17 languages), respectively. Hence, in this work, mBERT and IndicBERT are ensembled and then fine-tuned by training them on the in-house ILMQuAD dataset for all three languages. The obtained result evidenced that the ensemble BERT model with fine-tuning improves the result and becomes the new state-of-the-art performance for all three languages. The proposed ensemble BERT model is also evaluated on the MMQA, Translated SQuAD, MLQA, and XQuAD evaluation dataset for the English-Hindi MQA task.