Colibri: How to Run a 744-Billion-Parameter Model on Consumer Hardware
Frontier AI models have always required frontier hardware. A 744-billion-parameter model in 4-bit quantization still weighs about 372 GB, which requires more VRAM than eight NVIDIA A100 80GB GPUs just to load. Running one on a consumer machine has seemed categorically impossible. Colibri, released in July 2026 by open-source developer JustVugg, makes it work on a machine with as little as 25 GB of RAM.
This article covers how Colibri pulls this off, how to install it, and what real-world performance looks like on two very different hardware setups.
Why this is normally impossible
Each parameter in an LLM is a number. At 16-bit precision, 744 billion parameters would need roughly 1.5 TB of VRAM to load. Even with aggressive 4-bit quantization, the GLM-5.2 model comes in at about 372 GB, still far beyond any consumer GPU and most workstations.
The conventional approach to running a model is to load all its weights into fast memory before doing any inference. That assumption is what Colibri discards entirely.
How it works
The MoE architectural loophole
Colibri is purpose-built for GLM-5.2, which uses a Mixture-of-Experts (MoE) architecture. In a dense model, every parameter participates in every computation. MoE models divide their parameters across thousands of specialized sub-networks (experts), and a router selects only a small subset to activate for each token.
GLM-5.2 has 744 billion total parameters, but only about 40 billion are active for any single token. That's roughly 5% of the model doing work at any given moment.
This is the architectural property Colibri exploits. Since most of the model is idle at any given moment, you don't need all of it in fast memory at once. You only need the parts currently being used.
The hierarchical memory system
Colibri treats your machine's entire storage stack as a single tiered memory pool, loading model weights just-in-time as the router requests them. The developer describes this as a "JIT compiler, but for weights."
The three tiers:
- VRAM holds the most frequently accessed experts for near-zero latency access
- System RAM caches a larger working set of commonly used experts, pinned based on learned usage history
- NVMe SSD stores the full 372 GB model and streams expert weights on demand
The dense components that run on every token (attention layers, embeddings), about 17 billion parameters at roughly 9.9 GB in Int4, stay resident in RAM permanently. Individual expert networks live on the SSD and are streamed in when the router selects them. Once loaded, they're cached in case they're needed again.
The entire engine is written in pure C with no external dependencies. No CUDA, no PyTorch, no Python runtime required for the engine itself.
Adaptive caching and speculative decoding
Colibri learns which experts your queries tend to activate. The more you use it for a particular type of work, the better it gets at predicting which experts to pre-load at startup. Frequently used experts migrate up the hierarchy automatically, reducing I/O over time.
For generation speed, Colibri uses GLM-5.2's native Multi-Token Prediction head for speculative decoding. The model predicts several tokens ahead, batches them, and verifies in one pass. When predictions are correct, generation produces 2-3 tokens per forward pass instead of one.
Output quality is uncompromised: the developer has validated that Colibri's output is token-exact against a reference Transformers implementation. It also uses GLM-5.2's MLA attention, which reduces KV cache size by a factor of 57, keeping memory usage manageable during long conversations.
Installation
Hardware requirements:
- ~400 GB free disk space on a single drive (372 GB for the model plus working room)
- Internal NVMe SSD strongly recommended; external USB drives work but performance degrades significantly
- Minimum ~25 GB of RAM; 64 GB or more for comfortable use
- Linux, macOS, or Windows via WSL2; GCC or Clang required
Installation:
Clone the repository:
Build the engine:
Download the pre-converted Int4 model from Hugging Face. This is a ~370 GB download:
Start the server:
The engine starts and opens a chat interface at http://127.0.0.1:8000.
Performance on two hardware setups
The gap between "it works" and "it works well" comes down entirely to your memory hierarchy.
MacBook Pro M2 Max, 32 GB RAM, external USB-C SSD
Running the 372 GB model from an external USB-C drive was possible but painful. Time to first token was over two minutes. Generation speed was approximately 0.1 tokens per second.
The engine's built-in profiler revealed why: of the 158.5-second wall time for one turn, 145.3 seconds were I/O wait. The M2 Max was sitting idle almost the entire time, starved of data by the slow external drive. Actual computation time was 7.45 seconds. The CPU was barely working; the bottleneck was entirely disk bandwidth.
High-end PC, RTX 4090, 64 GB RAM, internal NVMe
Switching to an internal NVMe drive and doubling the RAM transformed the experience. Time to first token dropped to around 17 seconds, roughly 8x faster. Generation speed reached approximately 0.8 tokens per second.
The profiler showed I/O wait at 48% of total time and compute at 52%. With more RAM available, Colibri could cache a larger expert working set, and the NVMe's higher bandwidth meant much faster streaming for cache misses. The system was spending roughly equal time thinking and fetching, rather than almost entirely fetching.
What the hardware bottleneck means in practice
For anyone considering running Colibri, the practical takeaways from these tests are clear:
The single most important factor is disk speed. An internal NVMe is the minimum meaningful configuration. External drives, even fast ones, create a bandwidth ceiling that makes the experience painful regardless of your CPU or GPU.
RAM capacity directly determines expert cache size, which determines how much I/O the engine has to do per session. Below 25 GB the model won't run; at 32 GB it's uncomfortable; at 64 GB it becomes genuinely usable; beyond that each additional GB improves the warm cache hit rate.
GPU VRAM matters for the hottest experts at the top of the hierarchy but is not a hard requirement. Colibri runs without a GPU, using CPU computation for all matrix operations.
The engine has day-0 support in vLLM, SGLang, and llama.cpp-compatible deployments for teams wanting to integrate it into existing inference infrastructure. The GitHub repository includes setup documentation and the Hugging Face model page at mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp has the pre-converted weights.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.