FreeToken: Running Massive MoE Models Locally at 2-4x Ollama's Speed

On August 17, 2026, a team from UC Berkeley and UT Austin published arXiv:2608.16157 and released FreeToken as open source under Apache 2.0. The authors include Song Han, Matei Zaharia, and Ion Stoica. FreeToken is an edge-native serving engine built specifically for Mixture-of-Experts models, capable of running a 35B parameter model at interactive speeds on an 8GB laptop GPU, a 284B model on a gaming desktop, and the 753B parameter GLM-5.2 on a single workstation GPU.

The project reached over 2,200 GitHub stars in its first week and is available as a desktop app for Windows and Linux, or as a Python CLI package.

Why MoE models are slow on consumer hardware

A standard dense LLM activates all of its parameters for every token. A MoE model activates only a small fraction: DeepSeek V4-Flash has 284 billion total parameters but activates only 13 billion per token. This sparsity is what makes MoE architecturally efficient.

The problem is storage. Those 13 billion active parameters could fit in a consumer GPU with 16–24GB of VRAM. But the remaining 271 billion inactive parameters have to live somewhere, and they won't fit in VRAM. They sit in system RAM, and the GPU fetches whichever experts are needed for each token over the PCIe bus.

PCIe bandwidth is the bottleneck. The gating network that chooses which experts to activate is dynamic, so you can't predict in advance which experts the next token will need. Existing engines like llama.cpp and Ollama use static placement to cope with this: when you load the model, they permanently assign some layers to the GPU and offload the rest to CPU/RAM.

The static approach works but is inefficient. Because which experts are needed changes with every token, a fixed layout is wrong most of the time. Experts needed by the GPU end up on the CPU, and data travels back and forth across the PCIe link rather than staying in fast VRAM.

How FreeToken works

FreeToken's core insight is to treat this as a scheduling problem rather than a placement problem. The entire model lives in system RAM as the single source of truth. The GPU's VRAM acts as a dynamic LRU cache for recently used expert weights.

MoE models exhibit locality of reference: for a sequence of related tokens, the gating network tends to reuse the same set of experts repeatedly. After the first few tokens, the VRAM cache fills with the "hot" experts that keep getting activated. Subsequent tokens have a high probability of finding their required experts already in the cache. The paper reports cache hit rates as high as 85% in typical workloads.

Double buffering for the prefill stage

Processing the initial prompt (prefill) is the hard case. Long prompts activate a wide variety of experts, potentially touching most of the model. FreeToken handles this with double buffering:

While the GPU computes for layer N, FreeToken predicts which experts will be needed for layer N+1 and starts streaming them from RAM into a secondary VRAM buffer in parallel. By the time the GPU finishes layer N, the layer N+1 weights are already loaded. This hides PCIe transfer latency behind GPU computation, keeping utilization high throughout prefill.

The Q* policy for cache misses

When the cache misses and an expert isn't in VRAM, FreeToken has a choice: transfer the expert to the GPU over PCIe, or run the computation on the CPU which already has the weights in RAM. Neither option is universally better, and the right answer depends on your specific hardware.

The Q* policy handles this dynamically. On startup, FreeToken benchmarks your actual PCIe bandwidth and CPU memory bandwidth. When a batch of cache misses occurs, it calculates the optimal split: on a gaming desktop with PCIe 5.0, it might route 41% of missing experts to the GPU and 59% to the CPU. On a laptop with a narrower PCIe 4.0 link but a fast CPU, those ratios might be 13% and 87%. The goal is for both paths to finish at the same time, minimizing total latency. No manual configuration required.

The FTW weight format

Standard inference engines load a model from disk, decode it, and repack the weights into the internal memory layout the engine uses. For hundred-gigabyte models, this repack step is slow. FreeToken stores weights on disk in the FTW (FreeToken Weight) format, which matches the engine's internal layout exactly. Loading is a direct memory-mapped read from disk to RAM with no intermediate transformation.

Performance: FreeToken vs. Ollama

The test was a real-world agentic coding task: using OpenCode to write a complete Pytest suite for a sample Python project, run on a workstation with an RTX 5090 (32GB VRAM), Intel Core Ultra 9 285K CPU, and 64GB DDR5 RAM.

Scenario 1: Model larger than VRAM (the main case)

Model: Qwen3.6 35B quantized to 8-bit (~38GB), 6GB larger than the GPU's 32GB VRAM.

Ollama split the model by layer, loading ~70% onto the GPU and leaving ~30% for the CPU. Every token required a CPU detour. The task took 14 minutes 20 seconds at 58.8 tokens/second median.

FreeToken kept all computation on the GPU, streaming experts from RAM on demand via caching and Q* scheduling. The same task took 4 minutes 40 seconds at 132.5 tokens/second median. That's a 2.25x speedup on the same hardware.

Expert cache size sweep

Reducing the expert cache from 57.8% of experts to 40% only incurred a 7.2% performance penalty. A small subset of hot experts handles the majority of the workload, and FreeToken's LRU cache reliably identifies them.

Scenario 2: Model fits in VRAM

Model: 4-bit quantized version (~22GB), fits comfortably in 32GB VRAM.

Here Ollama wins. Ollama loaded the full model into VRAM and achieved 239.6 tokens/second. FreeToken, running its scheduling overhead unnecessarily, reached 225.3 tokens/second. When your model fits in VRAM, use Ollama or a similarly simple engine.

Agentic session support

FreeToken also includes two features particularly relevant for agentic workloads:

Semantic-aware caching uses semantic anchor checkpoints for recurrent state and KV caches. Agentic sessions constantly re-enter prefill: every tool call result, every thinking block, every context edit triggers another prefill pass. Without semantic caching, this recomputes the same prefix repeatedly. With it, FreeToken can skip redundant recomputation when only the tail of the context changed.

Runtime VRAM re-allocation lets you resize the expert cache dynamically during a running session using ft ctl without restarting the engine or reloading weights. This means you can trade off between expert cache and KV cache size based on what a particular session needs.

When to use FreeToken vs. Ollama

Use FreeToken when your MoE model is too large to fit in VRAM. This is its purpose-built use case, and the 2–4x speedup over Ollama in this scenario is real and hardware-agnostic.

Use Ollama (or another simple engine) when your model already fits entirely in VRAM. FreeToken's scheduling overhead costs a small amount in this case, and the simpler engine will be faster.

FreeToken currently supports DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2 across quantization formats including MXFP4, NVFP4, FP8, and BF16. It exposes Anthropic and OpenAI-compatible APIs, so it integrates directly with Claude Code, Codex, OpenCode, and DeepSeek Harness without configuration changes on the client side.

The research paper is at arXiv:2608.16157. The GitHub repository is at github.com/FlashML-org/FreeToken.