Evaluating the Carbon Impact of Large Language Models at the Inference Stage

Large Language Models (LLMs), such as GPT-3, ChatGPT, and GPT-4, have demonstrated enormous potential across a range of tasks and attracted over 100 million users globally in recent months. However, these LLMs are resource-intensive and contribute significantly to carbon emissions. Currently, our understanding of their carbon impact remains insufficient due to the lack of reliable measurement tools, standard methodologies, and evaluation metrics. To bridge this gap, this paper conducts a thorough study on the carbon impact of various open-source LLMs, including GPT-J 6B, GPT Neo 2.7B, GPTNEO 1.3B, and GPT-2 at the inference stage, utilizing the Software Carbon Intensity (SCI) specification released by the Green Software Foundation. The primary contributions of our research are: (1) We propose a quantitative framework that measures and contrasts the environmental impacts of different LLMs; (2) We illustrate that high-carbon LLMs do not necessarily provide superior model quality than their low-carbon counterparts; and (3) We find that the carbon emissions are primarily driven by embodied carbon in LLMs and that employing GPUs, as opposed to CPUs, can substantially reduce carbon emissions.