Beyond Language Models: Byte Models Are Digital World Simulators

Traditional deep learning often overlooks bytes, the basic units of thedigital world, where all forms of information and operations are encoded andmanipulated in binary format. Inspired by the success of next token predictionin natural language processing, we introduce bGPT, a model with next byteprediction to simulate the digital world. bGPT matches specialized models inperformance across various modalities, including text, audio, and images, andoffers new possibilities for predicting, simulating, and diagnosing algorithmor hardware behaviour. It has almost flawlessly replicated the process ofconverting symbolic music data, achieving a low error rate of 0.0011 bits perbyte in converting ABC notation to MIDI format. In addition, bGPT demonstratesexceptional capabilities in simulating CPU behaviour, with an accuracyexceeding 99.99prediction, models like bGPT can directly learn from vast binary data,effectively simulating the intricate patterns of the digital world.