Audio Interaction Model: A Streaming Audio LLM That Decides When to Speak

Quick answer

The Audio Interaction Model reframes a large audio language model as a streaming agent that runs a continuous perceive-decide-respond loop: it ingests sound, ambient context, and instructions in real time, decides whether and when a response is warranted, and speaks on the fly rather than waiting for a turn to end. To support it, the authors build SoundFlow — an end-to-end pipeline of streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference — and release StreamAudio-2M, a 2.6M-item corpus covering 7 core abilities and 28 sub-tasks. The model stays competitive across 8 benchmarks, including real-time ASR and streaming audio instruction following.

The turn-taking problem in audio assistants

Most audio LLMs are built for a clean, offline transaction: a user speaks, the audio ends, the model transcribes the whole clip, then generates a reply. That breaks down for live interaction. Real conversation has no tidy turn boundary — people pause mid-thought, background sounds matter, and a useful assistant sometimes needs to jump in early or stay silent. Treating audio as a finished file forces latency (you must wait for the end) and removes any notion of timing or initiative. The paper’s framing is that the missing capability is not better transcription but a decision: at every moment, should the model keep listening, or respond now?

How the perceive-decide-respond loop works

The loop adds an explicit decision stage between perception and generation. As audio streams in, the model continuously perceives the acoustic scene and any spoken instruction, then makes a semantic judgment about response timing — is there enough information to act, and is now the right moment — before committing to generation. This is what lets the same model handle proactive behavior (intervening when something in the environment warrants it) and ordinary voice chat, rather than only answering when explicitly addressed. Crucially the decision is driven by semantic content, not a fixed silence timer, which is the usual crude trigger in voice interfaces.

What SoundFlow provides

SoundFlow is the engineering scaffolding that makes the loop trainable and deployable, organized as three pillars:

• Streaming-native data construction — data is built to look like a live stream, not pre-segmented clips, so the model learns on the same shape of input it sees at inference.
• Comprehension-aware training — training conditions the model to understand the unfolding scene well enough to make the decide step reliable, not just to map audio to text.
• Asynchronous low-latency inference — perception and response generation run asynchronously so the model can keep listening while it speaks, which is what keeps real-time interaction stable instead of stalling on each turn.

What’s in StreamAudio-2M

StreamAudio-2M is the data backbone: roughly 2.6 million items spanning 7 fundamental abilities and 28 sub-tasks. The breadth is the point — covering dialogue, voice chat, real-time transcription, instruction following, and environmental-sound reactions in one corpus is what makes a single unified model plausible instead of a stack of task-specific systems. The authors also construct Proactive-Sound-Bench specifically to measure proactive audio intervention, the capability that turn-based benchmarks simply do not test.

Key results

• StreamAudio-2M: ~2.6 million items across 7 core abilities and 28 sub-tasks — the corpus is sized for breadth rather than a single skill.
• Benchmarks: competitive performance across 8 benchmarks while running as a unified streaming model, rather than winning one task with a specialist.
• New capabilities: real-time ASR (transcribing as audio arrives) and streaming audio instruction following emerge from the streaming-native setup, not from a separate model per task.
• New evaluation: Proactive-Sound-Bench targets proactive intervention — deciding to respond to a sound event — which existing turn-based suites do not measure.

Why it matters now

Voice is becoming a primary interface for assistants and agents, and the bottleneck has shifted from recognition accuracy to interaction feel — latency, knowing when to speak, and reacting to the world rather than just the user. By making timing a first-class learned decision and shipping the data and inference plumbing to train it, this work points at the next generation of audio LLMs: ones that behave like a participant in a live scene instead of a transcription box. The honest read: the contribution is as much the StreamAudio-2M corpus and SoundFlow recipe as any single model number.

Limits and open questions

The paper itself frames the model as work in progress for the next generation of large audio language models, so this is a direction more than a finished product. The abstract reports competitive — not state-of-the-art — performance across its 8 benchmarks, and does not publish the latency figures or per-benchmark scores that would let you judge how much the streaming design actually costs or gains versus an offline model. Proactive intervention is powerful but double-edged: a model that decides to speak unprompted can also interrupt at the wrong moment, and Proactive-Sound-Bench is a new benchmark from the same authors, so external validation of the proactivity claims is still open. Anyone building on this should wait for released data, weights, or code before assuming the loop transfers to their setting.

FAQ

What is the Audio Interaction Model?

It is a unified streaming large audio language model that runs a perceive-decide-respond loop: it listens to sound and instructions in real time, decides whether and when to respond based on semantic content, and generates a reply on the fly instead of waiting for the audio to finish.

How is the Audio Interaction Model different from a normal audio LLM?

A normal audio LLM processes a finished clip offline, then replies. The Audio Interaction Model adds an explicit decision step about response timing and runs perception and generation asynchronously, so it can react during a live stream and even intervene proactively.

What is SoundFlow in the Audio Interaction Model paper?

SoundFlow is the end-to-end framework behind the model, built on three pillars: streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference for stable real-time interaction.

What is StreamAudio-2M?

StreamAudio-2M is the streaming corpus released with the paper — about 2.6 million items spanning 7 core abilities and 28 sub-tasks, used to train the unified model across dialogue, real-time ASR, and instruction following.

One line: make “should I speak now?” a learned decision, and an audio LLM stops being a transcription box and starts being a live participant. Read the original paper on arXiv.