LLMBind: A Unified Modality-Task Integration Framework

In the multi-modal domain, the dependence of various models on specific inputformats leads to user confusion and hinders progress. To address thischallenge, we introduce LLMBind, a novel framework designed to unify adiverse array of multi-modal tasks. By harnessing a Mixture-of-Experts (MoE)Large Language Model (LLM), LLMBind processes multi-modal inputs and generatestask-specific tokens, enabling the invocation of corresponding models toaccomplish tasks. This unique approach empowers LLMBind to interpret inputs andgenerate outputs across various modalities, including image, text, video, andaudio. Furthermore, we have constructed an interaction dataset comprising 400kinstructions, which unlocks the ability of LLMBind for interactive visualgeneration and editing tasks. Extensive experimentation demonstrates thatLLMBind achieves very superior performance across diverse tasks and outperformsexisting models in user evaluations conducted in real-world scenarios.Moreover, the adaptability of LLMBind allows for seamless integration with thelatest models and extension to new modality tasks, highlighting its potentialto serve as a unified AI agent for modeling universal modalities.