Keynote Talk 2 Training Large Language Models: Challenges and Opportunities.

Language models with large number of parameters trained on massive datasets can achieve state-of-the-art accuracies in various natural language processing applications including summarization, automatic dialogue generation, translation, semantic search, and code autocompletion. However, training such models is challenging as these models no longer fit in the largest GPU memory and can require a very long training time. Therefore, numerous innovations and breakthroughs are required in dataset, algorithms, software, and hardware altogether to make training these models a reality. In this talk, I present our efforts to train the Megatron-Turing Natural Language Generation model (MT-NLG), the largest and the most powerful monolithic transformer language model trained to date, with 530 billion parameters. I will also showcase several applications of MT-NLG and discuss future research and numerous opportunities that this model presents.