Tidy analysis of TyDi: Analysing knowledge sharing in Multilingual domain

Recent trends in NLP have shown remarkable success across diverse set of tasks by virtue of pre-training transformer-based architectures [1] across a large corpora. One such architecture, multilingual-BERT (mBERT), has been shown to exhibit surprising cross-lingual abilities even in the absence of any cross-lingual training objective or aligned data. In this project, we work with a new cross-lingual QA dataset (TyDiQA-GoldP [2]) with an aim to investigate the cross-lingual transferability of mBERT by performing various experiments and ablation studies. Our experiments not only reveal excellent cross-lingual abilities of mBERT via zeroshot experiments but also discover that ensembling, gradient accumulation, and task specific knowledge transfer from a different cross-lingual dataset improves mBERT’s performance drastically and resulted in a 3.2 points of absolute improvement on F1 score over baseline. We realise that mBERT’s learning ability can be decoupled into task specific and language specific representations, allowing us to replace the need for expensive low-resource language data with easily available datasets in high-resource languages. Finally, we also explore qualitative improvements brought about by our methods for some languages for a more thorough treatment of our experiments. 1 Key Information to include • Mentor: Matthew Lamm