# Running a Multilingual Speech Model on a $10 Microcontroller
The ESP32-S3 has **16 MB of flash storage**. Cactus Compute’s Whistle speech recognition model is **16.9 MB**. In other words, the model is already larger than the chip’s entire storage capacity before you even account for firmware.
That is the core engineering challenge behind this project: **how do you run a capable offline voice assistant on a $10 microcontroller with almost no storage to spare?**
The solution combines several compromises. First, decoder layers are removed from the model to shrink it below the flash limit. The flash layout is then repartitioned to reclaim as much space as possible. Finally, the loss in model capacity is partly offset with **keyword biasing and a retry heuristic** that improve recognition on the commands the device is most likely to hear.
The result is a compact voice assistant that can **transcribe commands in seven languages entirely offline**, with no cloud API or network connection required. The trade-off is speed: inference takes roughly **7 to 12 seconds** per request.
The interesting part is not just that the model runs at all. It is how much capability can be recovered through careful engineering after the model is forced into hardware that, on paper, is too small to hold it.
## What Whistle is
Whistle is a speech-to-text model from Cactus Compute (the same team that built Needle, covered in an earlier article here) released in October 2026. At 16.9 MB it's roughly nine times smaller than OpenAI's Whisper base model at 145 MB, which targets similar quality.
The model supports English, German, French, Spanish, Italian, Dutch, and Polish, with automatic language detection: you don't specify the language in advance, the model identifies it. Beyond transcription, it can produce word-level timestamps and speech embeddings for speaker identification or audio similarity tasks.
Whistle was designed to work with Needle in a two-stage pipeline: Whistle converts speech to text, Needle converts text to a structured tool call. On the ESP32-S3, both models can't fit simultaneously, so this project uses Whistle alone with a simpler custom parser for command interpretation.
## The hardware
The ESP32-S3 N16R8 is a dual-core 240 MHz microcontroller with 16 MB of flash and 8 MB of PSRAM. Broadly capable for its price, but the storage limit is severe for AI model deployment.
The additional components are a INMP441 digital I²S microphone, a tactile push button, a perf board for mounting, solder and jumper wires, and a power bank for offline operation. The ESP32-S3 board has a built-in RGB LED on GPIO48 that serves as the output indicator.
## Fitting the model onto the chip
The gap between model size (16.9 MB) and available flash (16.0 MB) means the full model can't fit at all. Even if it could, the firmware itself needs a couple of megabytes.
Whistle's decoder uses an Intelligence Ladder architecture: the decoder consists of 8 layers, and any prefix from 2 to 8 layers is a functional model on its own. The encoder always runs all 8 layers and can't be trimmed. The decoder can be. Each layer removed reduces model size and accuracy in proportion.
A 4-layer decoder version was chosen as the balance point. A small Python script parses the original `.cact` model file, identifies the weights corresponding to decoder layers 5-8, and removes them. The result is `whistle_d4.cact` at 14.52 MB.
With the trimmed model in hand, the ESP32's flash partition table is reconfigured to allocate almost all available space to the model and application, leaving 98 KB of margin. Tight, but workable.
## Wiring
The wiring connects the INMP441 microphone to the ESP32 via I²S and adds the push button on GPIO7:
- INMP441 VDD → ESP32 3V3
- INMP441 GND → ESP32 GND
- INMP441 SCK (serial clock) → GPIO4
- INMP441 WS (word select) → GPIO5
- INMP441 SD (serial data) → GPIO6
- Push button leg 1 → GPIO7
- Push button leg 2 → GND
Pressing the button pulls GPIO7 to ground. The firmware captures audio while the button is held, then releases and starts transcription. The built-in RGB LED on GPIO48 provides feedback.
## Inside the inference pipeline
Once audio is captured, it passes through four stages:
**Mel spectrogram conversion:** Raw audio at 16,000 samples/second is sliced into 10ms windows. A frequency transform converts each slice into a heat map of energy at different pitches. Takes about 0.1 seconds.
**Stem (convolutional layers):** Eight consecutive 10ms slices are combined into a single 80ms frame through convolutional downsampling. This compression reduces the data volume before the expensive attention computation. Takes about 0.9 seconds.
**Encoder (transformer attention):** Every 80ms frame can attend to every other frame simultaneously across the full clip. This is what lets the model use end-of-sentence context to correctly interpret an ambiguous sound from the beginning. The most computationally expensive stage at about 3.2 seconds.
**Decoder (text generation):** The decoder generates one token at a time using cross-attention to align with encoder output. The first token it generates is always a language identifier (``, ``, etc.), deciding the language before transcribing any words. Takes about 2.2 seconds.
Total pipeline: 7-12 seconds from button release to LED response.
## Performance and the surprising accuracy result
On synthetic voice tests, the 4-layer on-device model achieved 78% success rate for English color commands with clean audio, and 76% with realistic microphone noise mixed in. Previous work on the same board with text-based models required around 40 seconds per command. The 7-12 second range is a major improvement.
The more surprising finding: comparing the 4-layer on-device model against the full 8-layer model running on a computer, the trimmed model on the ESP32 outperformed the full model across all seven languages for the specific task of color command recognition.
The smaller model isn't inherently smarter. The accuracy gap comes from two software-level compensations applied on top of the model's raw output.
**Keyword biasing:** The firmware maintains a list of valid color and action words. When the model is deciding between tokens with similar probabilities (e.g., "read" vs. "red"), a small bonus is added to any token that appears on the keyword list. This biases uncertain decisions toward expected vocabulary without affecting confident predictions.
**Retry rule:** Because the model auto-detects language, it occasionally misidentifies the language and produces a garbled transcript. When a transcription pass fails to produce a valid color command, the firmware reruns transcription with a forced different language (e.g., French if the first pass assumed English). The second result is accepted only if its confidence score is within a threshold of the first. This heuristic catches language misidentification while avoiding false substitutions.
Together these two techniques narrow the gap between a 4-layer and 8-layer model for a constrained vocabulary task. For broader transcription without a known vocabulary, the 8-layer model would be expected to perform better.
## Offline, private, local
The device runs entirely without network connectivity. No Wi-Fi, no cloud endpoint, no data leaving the hardware. Voice commands are processed on the chip from capture to LED response. The 98 KB of remaining flash headroom doesn't leave much room for extension, but for a fixed-vocabulary command application the current configuration is complete.
The Whistle model is Apache 2.0 licensed and available on Hugging Face. The firmware and model-trimming scripts for this project are on GitHub under the same license. For teams working on other ESP32 projects wanting to integrate Whistle without the trimming process, Cactus's `cactus-needle` Python package includes a `whistle` module with a documented API for running inference on any GGUF-compatible runtime.