Exploring the Limits of Decoder-Only Models Trained on Public Speech Recognition Corpora

The emergence of industrial-scale speech recognition (ASR) models such asWhisper and USM, trained on 1M hours of weakly labelled and 12M hours of audioonly proprietary data respectively, has led to a stronger need for large scalepublic ASR corpora and competitive open source pipelines. Unlike the saidmodels, large language models are typically based on Transformer decoders, andit remains unclear if decoder-only models trained on public data alone candeliver competitive performance. In this work, we investigate factors such aschoice of training datasets and modeling components necessary for obtaining thebest performance using public English ASR corpora alone. Our Decoder-OnlyTransformer for ASR (DOTA) model comprehensively outperforms theencoder-decoder open source replication of Whisper (OWSM) on nearly all EnglishASR benchmarks and outperforms Whisper large-v3 on 7 out of 15 test sets. Werelease our codebase and model checkpoints under permissive license.