Running a Local LLM on a Six-Year-Old Apple Watch

The hardware comparison looks modest at first. The Apple Watch Series 6, released in 2020, has 1 GB of RAM and a dual-core 1.8 GHz processor. A Raspberry Pi 1 from 2012 has a single-core 700 MHz processor and 512 MB of RAM. On paper, that might suggest a noticeable but not dramatic gap.

In practice, the difference is much larger. On the same model, the Apple Watch generates text at about 15 tokens per second, while the Raspberry Pi 1 reaches only about 0.3 tokens per second. That is roughly a 50x performance gap, far beyond the kind of 5x improvement a rough spec-sheet comparison might lead you to expect.

That gap helps illustrate an important point. Modern inference performance is not determined by clock speed and RAM alone. System-on-chip design, tighter memory integration, and instruction set support all have a major impact on how efficiently a device can run local AI workloads.

The result is that a smartwatch can outperform an older single-board computer by a much wider margin than the raw hardware specs seem to imply.

This article explains what it takes to run a local LLM on watchOS, why Apple's Core ML is not the right fit for this particular setup, how the arm64_32 architecture works in practice, and how to build and deploy the WatchLLM app on your own device.

Why Core ML doesn't work for this model

The natural starting point for on-device ML on Apple platforms is Core ML. You convert a model to .mlpackage format and hand it to the OS, which routes computation to CPU, GPU, or Neural Engine automatically. The limitation is that Core ML needs to recognize every operation in the model's computation graph.

The model used here is Falcon-H1 90M, a 90-million-parameter model that combines two architectures: standard Transformer self-attention and Mamba-2 State-Space Models. The combination is efficient: attention handles broad contextual relationships, while Mamba-2 processes sequences with linear rather than quadratic scaling by maintaining a compressed running state.

The problem is the "scan" operation at the heart of Mamba-2.

The scan processes each token sequentially, updating a fixed-size state block based on the previous state and the new token. Core ML has predefined building blocks for convolutions, dense layers, and attention, but not for State-Space Models. When the Core ML converter encounters the scan operation, it fails. The architecture is also dynamically shaped: the scan length depends on conversation length, and Core ML requires a static computation graph defined at compile time. Both problems are fundamental and not workaroundable.

The solution is llama.cpp, a pure C/C++ inference library with minimal dependencies, hand-optimized for ARM CPUs, and capable of running models in GGUF quantized format. It has no equivalent constraints on model architecture and no requirement for static graphs.

The arm64_32 architecture

watchOS uses an architecture called arm64_32. The name sounds like a 32-bit system, which would mean losing access to modern vector extensions and being stuck with 32-bit arithmetic. This is a misreading of what the name means.

arm64_32 breaks into two components:

arm64: The processor uses the full ARMv8 64-bit instruction set. All 64-bit arithmetic, the NEON vector unit for batched floating-point operations, and every modern ARM instruction are available. Nothing about the computational capability is 32-bit.

_32: Only the pointer size is 32-bit. Pointers are the variables that store memory addresses. A 64-bit pointer can address 16 exabytes of memory space. A 32-bit pointer can address 4 GB. Since the Apple Watch has 1 GB of RAM, there's no practical reason to waste 4 bytes per pointer when 2 bytes would cover the entire address space. Apple made this tradeoff deliberately to reduce memory overhead on a RAM-constrained device.

The practical implication: 32-bit pointers mean the watch can't address more than 4 GB of memory, which isn't a constraint for a device with 1 GB. Computationally, the watch is a full 64-bit ARM processor with NEON. The performance difference from pointer size is negligible.

llama.cpp doesn't officially support watchOS. Its build scripts cover iOS, macOS, tvOS, and visionOS, but not watchOS. The fix is a one-line source guard in llama.cpp that tells the compiler to recognize the arm64_32 target as a 64-bit ARM system for the purpose of enabling NEON optimizations.

The WatchLLM project repository includes the pre-compiled libllama.a static library with this patch already applied. You don't need to compile llama.cpp from source.

Building the app

Clone the repository:

git clone https://github.com/andrisgauracs/Apple-Watch-Edge-AI

Open WatchLLM.xcodeproj in Xcode.

The project structure:

• ContentView.swift: SwiftUI interface
• LlamaCppEngine.swift: Swift wrapper for the llama.cpp C API, handles model loading and inference
• LLMRunner.swift: Observable object managing app state, prompts, and output
• WatchLLM-Bridging-Header.h: Exposes C functions from llama.cpp to Swift
• vendor/llamacpp/: Pre-compiled libllama.a and headers
• models/: Where GGUF model files go (the Falcon-H1 model is included)
• tools/: Build scripts and the tool call configuration file

Connect your Apple Watch (paired with your Mac via iPhone), select it as the build target in Xcode, configure your Apple Developer account under Signing & Capabilities, set build configuration to Release for performance, and build with ⌘R.

Tool calls

The app supports tool calls: when the model determines it needs external data, it fetches from a configured API endpoint before generating the response. This is configured in tools/Tools.example:

[
  {
    "name": "Weather",
    "triggers": ["weather", "temperature", "forecast"],
    "url": "https://api.open-meteo.com/v1/forecast?...",
    "jsonPath": "current_weather.temperature",
    "factTemplate": "The current temperature is {value}°C."
  }
]

Rename the file to Tools.json to enable it. Each tool definition specifies trigger keywords that cause the model to attempt a tool call, the API endpoint to hit, a JSON path to extract the relevant value from the response, and a template for presenting the value back to the user. Add your own API endpoints by following the same structure.

Performance

On an Apple Watch Series 6 with Falcon-H1 90M:

Short factual queries: The model triggers the Wikipedia tool for "What is the capital of France?" and returns the answer at 24.8 tokens/second. At this rate, responses feel instantaneous.

SmoL M2 (135M parameters) on the same query: 14.7 tokens/second. Noticeably slower but still comfortable for interactive use.

Long-form generation ("Explain the depth-first search algorithm"): Starts at around 20 tokens/second and degrades to under 10 tokens/second as the response lengthens. The KV cache for long conversations consumes an increasing fraction of the 1 GB RAM, causing memory management overhead that slows down inference. This is the watch's binding constraint: computation is fast enough, but RAM runs out for long sessions.

For short, interactive queries and tool-augmented factual lookups, the experience is genuinely good. For multi-paragraph explanations or extended conversations, you'll notice the slowdown.

The 90M and 135M models are genuinely small. They can answer factual questions, follow instructions, and make tool calls reliably, but they won't match a 7B or 13B model on complex reasoning or nuanced writing. The appropriate expectation is a capable, fast, private assistant for short tasks, not a frontier AI.

Got an article suggestion? Let us know

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.