Qwen2-Audio: Chat with Your Voice! · AI Engineering Radar

To achieve the objective of building an AGI system, the model should be capable of understanding information from different modalities. Thanks to the rapid development of large language models, LLMs are now capable of understanding language and reasoning. Previously we have taken a step forward to extend our LLM, i.e., Qwen, to more modalities, including vision and audio, and built Qwen-VL and Qwen-Audio. Today, we release Qwen2-Audio, the next version of Qwen-Audio, which is capable of accepting audio and text inputs and generating text outputs. Qwen2-Audio has the following features:

Voice Chat: for the first time, users can use the voice to give instructions to the audio-language model without ASR modules.
Audio Analysis: the model is capable of analyzing audio information, including speech, sound, music, etc., with text instructions.
Multilingual: the model supports more than 8 languages and dialects, e.g., Chinese, English, Cantonese, French, Italian, Spanish, German, and Japanese.

We open-weight Qwen2-Audio-7B and Qwen2-Audio-7B-Instruct in Hugging Face and ModelScope, and we have built a demo for users to interact with. Below are some examples to show the model performance:

Voice Chat#

Audio Analysis#

Performance#

We have conducted a series of experiments on benchmark datasets, including LibriSpeech, Common Voice 15, Fleurs, Aishell2, CoVoST2, Meld, Vocalsound, and AIR-Benchmark, to evaluate the performance of Qwen2-Audio in comparison with our previously released Qwen-Audio as well as the state-of-the-art models in each tasks. Below we demonstrate a figure to show how Qwen2-Audio performs against the competitors. In all the tasks, Qwen2-Audio significantly surpasses either previous SOTAs or Qwen-Audio.

More specific results on the datasets are listed below in the table.

Architecture#

Below is a demonstration of the training architecture. Specifically, we start from the Qwen language model and an audio encoder as the foundation models. We sequentially apply multi-task pretraining for the audio language alignment, and supervised finetuning as well as direct preference optimization to grasp the capabilities of downstream tasks and model human preference.

How to Use#

Now Qwen2-Audio is officially supported by Hugging Face Transfomers. We advise you install the latest version of transformers from source:}}

We demonstrate how to use Qwen2-Audio-7B-Instruct to play with voice chat and audio analysis. Here is an example of voice chat:}}

In voice chat mode, the user input is simply the audio without text, and the instructions from the users are inside the audio. Next is an example of audio analysis:}}

In comparison, in audio analysis mode, there is additional text instructions. However, switching between either mode only needs to modify your user inputs and there is no need to worry about others such as system prompt.

Next Step#

This time we bring a new audio language model, Qwen2-Audio, which supports voice chat and audio analysis at the same time and understands more than 8 languages and dialects. In the near future, we plan to train improved Qwen2-Audio models on larger pretraining datasets, enabling the model to support longer audio (over 30s). We also plan to build larger Qwen2-Audio models to explore the scaling laws of audio language models.