On The Open Prompt Challenge In Conditional Audio Generation

Text-to-audio generation (TTA) produces audio from a text description,\nlearning from pairs of audio samples and hand-annotated text. However,\ncommercializing audio generation is challenging as user-input prompts are often\nunder-specified when compared to text descriptions used to train TTA models. In\nthis work, we treat TTA models as a ``blackbox'' and address the user prompt\nchallenge with two key insights: (1) User prompts are generally\nunder-specified, leading to a large alignment gap between user prompts and\ntraining prompts. (2) There is a distribution of audio descriptions for which\nTTA models are better at generating higher quality audio, which we refer to as\n``audionese''. To this end, we rewrite prompts with instruction-tuned models\nand propose utilizing text-audio alignment as feedback signals via margin\nranking learning for audio improvements. On both objective and subjective human\nevaluations, we observed marked improvements in both text-audio alignment and\nmusic audio quality.