Audio.cpp: A Unified Local Runtime for Audio AI Models

llama.cpp and whisper.cpp established that you could run powerful AI models locally with a single binary and a GGUF model file, no Python runtime required. audio.cpp applies the same approach to the full audio AI stack: text-to-speech, speech-to-text, voice cloning, voice conversion, speaker diarization, music generation, and audio processing, all through one CLI and one server binary.

Released by ShugoAI LLC under the Apache 2.0 license on June 25, 2026, the project is at github.com/0xShug0/audio.cpp. Version 0.6, released August 13, 2026, covers 49 model families and 70+ model variants. The contributor count is 237, making it a genuine open-source project rather than a solo effort.

The problem it solves

Running audio AI models locally used to mean navigating a complex chain of Python dependencies: specific PyTorch versions, CUDA toolkit versions that had to match your driver version and your GPU, and conflicting library requirements between different models. Each new model often meant a new isolated environment just to avoid breaking what already worked.

The ggml ecosystem, which powers whisper.cpp and llama.cpp, solved this for text by collapsing the dependency stack into two components: a GGUF model file containing the weights and metadata, and a native C++ binary that loads it and runs inference. No Python, no CUDA configuration, no library conflicts.

audio.cpp applies that same model to audio. There is no Python dependency at runtime. The binary handles TTS, ASR, voice cloning, and the rest under the same interface.

What it supports

The task coverage as of v0.6 includes text-to-speech, speech-to-text (ASR), voice activity detection, voice cloning, voice conversion, speaker diarization, music generation, stem separation, and denoising.

The backend architecture runs on top of ggml with selectable hardware targets at compile time: CPU (always available), NVIDIA CUDA, Vulkan (cross-platform GPU including AMD), Metal (Apple Silicon), and HIP/ROCm for AMD GPUs added in v0.5.

Voice cloning walkthrough

The setup requires cloning the repository and compiling from source. On macOS with Apple Silicon:

git clone https://github.com/0xShug0/audio.cpp.git

cd audio.cpp

./build-macos-metal.sh

This produces the binary at build/macos-metal-release/bin/audiocpp_cli. Compilation takes a few minutes. Pre-built binaries for Windows are available on the releases page.

Model download uses a provided Python script (Python is only needed for the one-time setup and model conversion step, not at runtime):

python3 -m audio_cpp.bin.download --repo-id <model_repository_on_huggingface>

With the binary and model ready, voice cloning takes a reference audio file (10–15 seconds of clean speech works well) and the text you want generated in that voice:

./build/macos-metal-release/bin/audiocpp_cli \
  --task tts \
  --family higgs_audio_tts \
  --model models/Higgs-Audio-v3-4B-gguf/higgs-v3-4b-f16.gguf \
  --backend metal \
  --voice-ref /path/to/reference.wav \
  --text "This is me talking on my MacBook about audio cpp." \
  --out cloned_output.wav

--voice-ref is what enables cloning. Without it, the model uses its default voice. With it, it attempts to match the speaker characteristics from the reference file. Generation for a short sentence on an M-series Mac takes a few seconds.

OpenAI-compatible server mode

The binary also runs as a local server with endpoints matching OpenAI's Audio API structure, including /v1/audio/speech and /v1/audio/transcriptions. Any application using the OpenAI SDK can switch to the local server by changing one line:

# Before: OpenAI cloud
from openai import OpenAI
client = OpenAI(api_key="YOUR_KEY")

# After: local audio.cpp server
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed",
)

Everything runs locally. No data leaves the machine, no rate limits apply, and no API costs accumulate. A native WebUI was added in v0.4 for browser-based access without any code changes.

Current limitations

The project is early-stage (v0.6) and should be treated accordingly.

Compilation from source is required on macOS and Linux. The model conversion step still relies on Python even though the runtime itself doesn't. Several models in the ecosystem have reported quality issues including robotic output and occasional crashes.

Performance benchmarks in the README were measured on a top-of-the-line GPU. Results on consumer hardware will be lower, though still usable for offline tasks. Real-time generation for interactive applications may not be achievable on all hardware configurations.

The GitHub contributors graph in the source article described this as a one-person project, but search results show 237 contributors as of August 2026. The development pace is fast with releases roughly every two to three weeks since June, and the scope has expanded considerably from the initial 20 models to 49 families.

For applications where audio quality, latency, or reliability are critical, validating output on your specific use case before shipping is worth doing. For local experimentation, offline use, or privacy-sensitive applications where cloud APIs aren't viable, audio.cpp is currently the most comprehensive single-binary option in the ggml ecosystem.

Got an article suggestion? Let us know

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.