How to Run a 35-Billion-Parameter AI Model on Your iPhone
Running a 35-billion-parameter language model on a device with 8 GB of RAM sounds impossible. At full precision, a model that size would need around 70 GB of memory. Even at 4-bit quantization, it would still take roughly 17.5 GB, more than twice the total memory available on an iPhone 15 Pro.
And that 8 GB is not dedicated to the model. It is shared between iOS, the GPU, and every other running process.
What makes this possible is not one trick, but four techniques working together.
The first is the model’s Mixture-of-Experts architecture, which means only about 3 billion parameters are active at any given time. The second is on-demand expert streaming, where the model loads experts from SSD into GPU memory only when they are needed. The third is the iOS page cache, which effectively acts as another memory tier between storage and RAM. Finally, tiered quantization compresses less frequently used experts more aggressively than the ones the model depends on most.
Together, these techniques bring the active RAM footprint down to roughly 1.4 GB, while the full model occupies about 13.4 GB on the SSD.
That changes the problem completely. Instead of trying to squeeze the entire 35B model into memory, the device only keeps the small portion it needs at any given moment.
This article explains how each of those techniques works and walks through how to run the model on your own device using the Flash-iOS app.
The model: Qwen3.5-35B-A3B
The "A3B" suffix in the model name is the key specification. It means that out of 35 billion total parameters, only approximately 3 billion are active for any given token. The architecture is Mixture-of-Experts with 40 layers, 256 expert sub-networks per layer, and a router that selects 8 of those 256 experts per token plus one shared expert that always activates.
The other 248 experts in each layer sit idle. For every token generated, the compute load resembles a 3B dense model, not a 35B one. This is what makes the streaming approach viable.
Why it can't just load into RAM
The memory arithmetic makes the problem clear.
At FP16 (2 bytes per parameter): 35B × 2 = 70 GB. At 4-bit quantization (0.5 bytes per parameter): 35B × 0.5 = 17.5 GB.
The iPhone 15 Pro has 8 GB RAM total. The model won't fit, even compressed.
Traditional inference engines try to fit as much of the model as possible into fast memory and fall back to CPU for the rest. This works poorly for MoE because which experts are needed changes with every token, so any static partitioning is wrong most of the time.
On-demand expert streaming
Rather than trying to load the model into RAM at all, Flash-iOS keeps only the always-resident components in RAM and streams the expert weights from SSD on demand.
What stays in RAM (1.4 GB total): - Token embeddings - Attention layers (one per transformer layer) - Router networks (one per layer) - Shared experts (one per layer, always active)
What lives on SSD (13.4 GB): - All individual routed expert weights, split into separate files by expert
The inference loop for each token: the attention layer runs in GPU memory, the router selects 8 experts, the app reads those 8 expert files from SSD into GPU memory, the GPU computes on them alongside the shared expert, and the output flows to the next layer. This repeats 40 times per token: 8 experts × 40 layers = 320 file reads from SSD per token generated.
The iOS page cache as a free memory tier
The 320 file reads per token would be prohibitively slow if each one went to flash storage. The key insight is that MoE models have strong locality of reference: certain experts are activated far more frequently than others across any given conversation. iOS's page cache automatically retains recently accessed file data in unused RAM.
After the first few tokens, the cache fills with the frequently activated experts. Subsequent reads for those experts hit the cache rather than the SSD, which is orders of magnitude faster. The page cache behavior is automatic and requires no application code; the OS handles it transparently. This is why the first token is slow and subsequent tokens get significantly faster as the cache warms up.
Tiered quantization
The final optimization addresses the 19.5 GB → 13.4 GB size reduction. Not all experts deserve the same compression treatment. Hot experts (the ones the model activates most frequently) are kept at 4-bit quantization to preserve output quality. Cold experts (rarely activated ones) are compressed to 2-bit quantization, halving their size with minimal impact on overall accuracy since they're rarely used.
The smaller total footprint means more of the expert files fit in the page cache at once, directly improving the cache hit rate and therefore the sustained tokens-per-second figure.
Setup instructions
Requirements: iPhone 15 Pro or newer (8 GB RAM minimum), a Mac, Xcode, a paid Apple Developer account ($99/year), Git, and 14+ GB free on both the Mac and the iPhone.
Clone the repository
Apply the pread fix
The public repository has a bug in the expert validation code. Without this fix, 2-bit experts are silently skipped, which means the tiered quantization model produces incorrect output.
Open Flash-iOS/metal_infer/infer.m and find the async_pread_wait function. Inside the for loop, find:
Replace it with:
This validates each expert's read against its own expected size rather than a hardcoded 4-bit size, which allows 2-bit experts to pass validation correctly.
Build in Xcode
Open Flash-iOS.xcodeproj, connect your iPhone, select it as the build target, configure your Apple Developer account under Signing & Capabilities, and set the build configuration to Release under Product → Scheme → Edit Scheme → Run. Build with ⌘R.
Download and transfer the model
Download the pre-packaged model from dan-woods/Qwen3.5-35B-A3B-tq-2-4-prepacked-bf16 on Hugging Face. The largest file is approximately 13.4 GB.
Transfer it to the iPhone via Xcode: Window → Devices and Simulators → select your iPhone → Flash-iOS app → drag the model file into the Documents section. Expect roughly 7 minutes over USB.
Run it
Open the Flash-iOS app on the iPhone, tap the model file, wait for the 1.4 GB resident components to load into RAM, and start typing. Expect 11 tokens per second on an iPhone 15 Pro. The device will get noticeably warm; this is expected.
What to expect
At around 11 tokens per second, the model is usable but not especially fast. The first response in a conversation usually feels slower while the page cache warms up, and later responses tend to improve. Longer prompts also add noticeable latency because attention becomes more expensive as the context grows.
Battery use is another limitation. Streaming weights from SSD while driving the GPU and allocating memory puts sustained pressure on the device, so this is one of the heavier workloads you can run on a phone. It is better viewed as a technical demonstration of what on-device inference can already do than as a practical replacement for a cloud model you use all day.
The result is more impressive when you compare it with a desktop. On a Mac Mini M4 with 16 GB of unified memory, the same model with six active experts reaches about 11.5 tokens per second with a 2.5-second time to first token. That puts the iPhone surprisingly close to a budget desktop machine, despite working with much tighter memory and thermal limits.
The broader point is that the techniques used here are starting to move from custom engineering tricks into platform features. MoE sparse activation, SSD streaming, page-cache reuse, and tiered quantization are increasingly becoming part of the standard on-device inference stack.
iOS 2026, for example, introduced llmCache in Core ML to manage model-weight paging between NVMe storage and DRAM automatically. What requires careful manual tuning today could become a normal API pattern for running much larger models on mobile hardware over the next few years.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.