Optimizing Large Models For Performance And Sustainability Through Evolutionary Pruning

Abstract The need for network optimization has become become evident when considering the rising computational costs and environmental impact caused by AI models. The substantial carbon footprint and resource consumption of Large Language Models (LLMs) underscore the urgency of optimizing network architectures. These models, with varying parameters, place a significant strain on energy resources, particularly in terms of CO2 emissions and electricity consumption. Achieving a balance between model performance and ecological responsibility is becoming increasingly critical in the era of large-scale AI models.Training and using such models consume substantial amounts of water and power, highlighting the necessity for sleeker, faster, and more efficient model training strategies. We address a crucial challenge related to the compression of deep neural networks, with the aim of enhancing their suitability for embedded devices. Deep neural networks typically exhibit high space and computational requirements, which can hinder their deployment on edge devices and limit their widespread adoption. This study investigates innovative techniques for pruning neural networks, presenting two distinct strategies, namely, "evolution of weights" and "smart pruning." These novel approaches are systematically compared to conventional pruning methods using established benchmark datasets. The proposed pruning method involves the continuous assessment of parameter importance throughout the training process, employing a weighted average methodology that surpasses the traditional magnitude-based pruning in terms of accuracy preservation during the compression process. The outcomes of this approach include accelerated computations, higher compression rates with minimal accuracy degradation, enhanced resilience against adversarial attacks, and the capacity to train models using smaller datasets. The authors have thoughtfully provided a PyTorch library for broad usage, simplifying the implementation of their approach across various model architectures.