Comparative Study of Large Language Model Architectures on Frontier
CoRR(2024)
摘要
Large language models (LLMs) have garnered significant attention in both the
AI community and beyond. Among these, the Generative Pre-trained Transformer
(GPT) has emerged as the dominant architecture, spawning numerous variants.
However, these variants have undergone pre-training under diverse conditions,
including variations in input data, data preprocessing, and training
methodologies, resulting in a lack of controlled comparative studies. Here we
meticulously examine two prominent open-sourced GPT architectures, GPT-NeoX and
LLaMA, leveraging the computational power of Frontier, the world's first
Exascale supercomputer. Employing the same materials science text corpus and a
comprehensive end-to-end pipeline, we conduct a comparative analysis of their
training and downstream performance. Our efforts culminate in achieving
state-of-the-art performance on a challenging materials science benchmark.
Furthermore, we investigate the computation and energy efficiency, and propose
a computationally efficient method for architecture design. To our knowledge,
these pre-trained models represent the largest available for materials science.
Our findings provide practical guidance for building LLMs on HPC platforms.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要