Qwen Audio Agent: Full-Duplex Voice for AI Coding Agents
Voice coding has an awkward limitation: the conversation often stops as soon as you give the agent real work to do. Ask it to refactor a file, run tests, or investigate a bug, and the voice interaction goes quiet while the task runs in the background.
The problem is not speech recognition or the capability of the underlying model. It is architectural. In most voice agents, the conversation loop and the task execution loop are effectively the same loop, so a long-running coding task can block the interaction until it finishes.
Qwen Audio Agent takes a different approach. It separates conversation from execution using a two-layer architecture. A lightweight voice frontend remains available for real-time conversation, while a separate backend agent handles heavier coding tasks independently.
The two layers communicate through the Agent Client Protocol (ACP). That means the frontend can keep talking to you while the backend edits code, runs commands, or performs other longer-running work. Once the task finishes, the backend sends the result back and the agent can announce it naturally within the ongoing conversation.
The project is available at github.com/QwenAudio/qwen-audio-agent under the Apache 2.0 license. It does not ship its own model weights. Qwen Audio Agent is a runtime for building this kind of voice-agent workflow, not a standalone model.
The blocking problem
Traditional voice coding is a linear pipeline: speech to text, text to model, output pasted into editor. This works for dictation. It breaks for agent tasks. When you say "refactor this file," the entire system blocks until the refactor is done. You can't ask a follow-up question, request a status update, or give a new instruction. The conversational layer is coupled to the execution layer, so whenever the agent is busy, the conversation is paused.
Full-duplex and agent presence
Full-duplex means both directions of the conversation can be active simultaneously. In practice this means two things:
Barge-in: You can interrupt the agent mid-sentence. When you start speaking, the runtime detects it, truncates the agent's audio output stream, suppresses any in-flight partials that would otherwise leak into the next turn, and starts a new input turn. The interruption feels clean rather than glitchy because the runtime treats it as a state machine transition rather than a simple cutoff.
Asynchronous results: The agent can interrupt you. When a background task completes, the result flows back into the live conversation even if you're mid-sentence. You asked for a refactor, started talking about something else, and three minutes later the agent says "the refactor is done and the tests are passing" without you having to ask.
This combination is what the project calls "agent presence": the voice layer stays alive and responsive regardless of what the backend is doing.
The two-layer architecture
Layer 1: the realtime frontstage. A lightweight voice agent handles continuous audio capture, real-time transcription and synthesis, barge-in detection, simple fast queries that don't need heavy reasoning, and the task lifecycle: spawning backend tasks, polling status, canceling them if needed. This layer never blocks on work. Its only job is to keep the conversation alive.
Layers 2 and 3: the backend agent and durable execution. When the frontstage receives a complex task, it passes it through a Voice Gateway and TaskManager to an independent backend agent session. This is where Claude Code, Codex, Kimi Code, Qwen Code, DeepSeek Harness, or any ACP-compatible agent lives. The backend has access to the file system, terminal, and browser. It executes the slow work in its own process.
The backend is swappable because the two layers communicate over ACP rather than through a tight internal integration. You can use whichever coding agent you already have configured.
Interruptions as a state machine
The five-state interruption model: detect speech, mark interruption, emit interrupted event, suppress in-flight audio and transcript partials, start new turn. The suppress step is what prevents old audio from leaking into the new conversational turn. Without it, interruptions feel sticky or produce garbled responses as the old and new streams mix.
Setup
The desktop app is a signed universal build for Windows, macOS, and Linux. No Python environment, no CUDA configuration, no compilation required.
The voice frontend needs a real-time speech provider. The default path uses Alibaba's DashScope API, which requires an Alibaba Cloud account. DashScope handles low-latency transcription and synthesis in the cloud. Some documentation is in Chinese and may need browser translation.
The fully local alternative, added in v1.3.0, uses a Hugging Face speech-to-speech pipeline with an MLX backend for Apple Silicon acceleration. This keeps all audio processing on-device with no cloud API keys. The trade-off is that this path is less documented and latency is higher than the cloud option.
For the backend agent, configure whichever coding agent you already use: Claude Code needs an Anthropic API key; Codex needs an OpenAI key; Kimi Code, Qwen Code, and DeepSeek Harness have their own configuration flows. One-click backend agent install was added in v1.4.1.
The default wake word is "你好千问" (nǐ hǎo Qiānwèn). On-device wake detection runs via sherpa-onnx without a cloud round-trip.
What the runtime currently does
Beyond the core full-duplex loop, features added through v1.11.0 include:
Voice permission gates for destructive operations: file deletions, force pushes, and similar actions require spoken confirmation before the backend agent executes them.
Invisible memory (added v1.6.0): after each session, the runtime automatically extracts and stores context for future sessions. You can review and delete stored memories.
Scheduled reminders (v1.5.0): you can ask the agent to remind you of something in 30 minutes and the runtime handles the timer and announcement.
Computer use (v1.6.1): the backend agent can operate the computer directly, not just the file system and terminal.
Agent Skills (v1.11.0): you can install and manage skills like Archify or anti-slop from within the voice interface.
v2.0.0 is in active development with changes to the agent architecture, task lifecycle, multimodal input, memory, and extensibility.
The actual limitation
The biggest caveat is DashScope. Qwen Audio Agent's real-time voice experience depends on Alibaba's proprietary, paid, cloud-hosted service, which means the orchestration layer is open source but the most performance-sensitive part of the stack is not.
There is a fully local path, but it comes with trade-offs. The documentation is primarily in Chinese, setup is more involved, and latency is noticeably higher. For private, offline, or air-gapped environments, that may still be the right option. For most developers, however, DashScope delivers the smoother experience in exchange for a paid API and an external cloud dependency.
That makes the project most useful for teams that already use Claude Code, Codex, or another ACP-compatible agent and want to add voice without blocking the conversation whenever the agent starts working.
The more interesting takeaway is the architecture itself. A lightweight conversational frontend stays responsive while a separate backend handles heavier execution tasks. That pattern is useful well beyond Qwen Audio Agent, and this project is a practical example of how it can work in a real coding workflow.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.