LiteRT.js: Google's Runtime for Near-Native AI Inference in the Browser

Running machine learning inference in the browser has always meant accepting a performance compromise. TensorFlow.js, the main tool for browser-based ML, runs on JavaScript kernels that can't fully exploit the hardware underneath them.

Google's new LiteRT.js changes that by bringing the same C++ inference runtime that powers ML on Android and iOS into the browser via WebAssembly. The result is meaningful performance gains, native hardware acceleration across CPU, GPU, and NPU, and access to the full .tflite model ecosystem without any new conversions.

What LiteRT.js is

LiteRT (formerly TensorFlow Lite) is Google's on-device inference runtime, deployed across billions of Android, iOS, and embedded devices. LiteRT.js is a JavaScript binding for that same runtime, compiled to WebAssembly and wrapped in a clean JS/TypeScript API.

The key distinction from TensorFlow.js is architectural. TensorFlow.js implements its inference operators as JavaScript kernels, which limits how deeply it can interact with CPU and GPU hardware. LiteRT.js doesn't reimplement the runtime in JavaScript at all. It compiles the existing, battle-tested C++ engine to Wasm and exposes it through the @litertjs/core package. When you run a model with LiteRT.js, you're running the actual native runtime engine, just compiled to a target the browser can execute securely.

Installation:

npm install @litertjs/core

The official documentation covers setup and model loading in detail.

Why TensorFlow.js hits a ceiling

The architectural gap matters because of what JavaScript can and can't do with hardware.

JavaScript kernels have limited access to low-level CPU and GPU features. They can't use advanced instruction sets, fine-grained memory management, or multi-threading as efficiently as a native C++ program. For light tasks this is acceptable, but for the kinds of computations ML models require, the gap is significant.

How LiteRT.js achieves near-native speed

The process is straightforward. Google takes the same optimized C++ runtime used on Android and iOS, compiles it to a WebAssembly binary, and loads it in the browser. LiteRT.js handles the JavaScript wrapper that exposes this to developers. The model operations run inside the Wasm module, not in JavaScript, which removes the abstraction layer that limits TensorFlow.js.

The backend architecture

LiteRT.js has a three-tier hardware acceleration stack that targets the most capable hardware available on each user's device.

CPU: XNNPACK

The CPU backend uses XNNPACK, Google's optimized library for floating-point neural network inference. It supports multi-threading across CPU cores and leverages WebAssembly SIMD instructions, which let the CPU process multiple data points in a single operation. XNNPACK is the universal fallback: it runs on any modern browser regardless of GPU or NPU support, so your application works everywhere.

GPU: ML Drift and WebGPU

The GPU backend is where the biggest performance gains come from. LiteRT.js uses ML Drift, Google's GPU acceleration layer, running on top of the WebGPU API. Unlike the older WebGL approach, WebGPU gives more direct control over the GPU and allows native GPU kernels (shaders) to handle tensor computations directly. The JavaScript layer is no longer in the critical path for inference work.

NPU: WebNN

The most experimental tier is NPU support via the WebNN API. NPUs are dedicated AI accelerators present in many modern phones and some laptops with Apple Silicon or Snapdragon chips. WebNN is currently only available in recent Chrome and Edge builds, but it represents the most power-efficient path for inference on supported hardware.

Benchmark results

Google benchmarked LiteRT.js against other web inference runtimes on a 2024 MacBook Pro with an M4 chip.

Across CPU and GPU inference on classical computer vision and audio models, LiteRT.js delivers up to 3x the performance of competing web runtimes.

The gains are more dramatic when looking at task-specific speedups from moving work off the CPU.

On WebGPU versus CPU, speedups range from 5.6x for automatic speech recognition to 30.8x for image segmentation. On WebNN versus CPU, the range is 6.9x for ASR to 67.2x for text encoding. Individual results will vary based on the user's GPU hardware, thermal conditions, and browser driver quality, but the direction is consistent.

Model compatibility and workflow integration

LiteRT.js runs .tflite models directly. If you already have models in that format, there's no conversion step. Models from Kaggle and Hugging Face in the LiteRT format are supported out of the box.

For PyTorch models, Google provides litert-torch, a converter that produces a compatible .tflite file in a single step. For size and performance optimization, the AI Edge Quantizer lets you configure quantization per model layer, reducing file size and improving inference speed while preserving accuracy.

For teams already using TensorFlow.js, the @litertjs/tfjs-interop package allows LiteRT.js to slot into existing pipelines as a drop-in replacement for TFJS Graph Models.

Building a real-time 3D motion capture app

To see what this performance difference actually means in practice, consider a real-time motion capture application that uses a webcam to track body movements and animate a 3D character. The entire application runs client-side with no backend.

Loading and compiling the model

The first step is initializing the Wasm runtime and loading a pose estimation model, in this case BlazePose in .tflite format.

import { loadLiteRt, loadAndCompile } from '@litertjs/core';

// Initialize LiteRT.js Wasm files (served from your web server)
await loadLiteRt('/path/to/wasm/');

// Load and compile the model with WebGPU acceleration
const model = await loadAndCompile('/models/pose_landmark_full.tflite', {
    accelerator: 'webgpu', // falls back to 'wasm' if WebGPU is unavailable
});

loadAndCompile fetches the model, compiles it for the specified accelerator, and returns a ready-to-use model object. If webgpu isn't available in the user's browser, you can catch the error and recompile with wasm as a fallback.

The inference loop

Inside a requestAnimationFrame loop, each frame from the webcam is drawn to a canvas, preprocessed to the model's expected input size, and passed to model.run(). The model returns 33 body landmarks, each with x, y, and z coordinates representing joints like shoulders, elbows, wrists, and hips.

Driving the 3D character

With landmarks on each frame, a Three.js scene can map those points to the corresponding bones in a rigged 3D character. The vector between landmark pairs, such as left shoulder and left elbow, gives the rotation for the corresponding bone. Updated on every frame, this produces a continuous, real-time animation driven by actual body movement.

The performance difference

The application includes a backend toggle to switch between CPU and WebGPU at runtime. The difference is immediately visible:

On CPU (Wasm), the application runs at around 38 FPS with inference taking roughly 23 ms. There's a small but perceptible lag between movement and the character's response.

On WebGPU, the application runs at 120+ FPS on capable displays, with inference time dropping to about 8 ms. The animation is smooth and the lag is imperceptible.

That roughly 3x real-world improvement matches what the benchmarks predicted and makes the difference between a demo and a usable application.

What's coming: LiteRT-LM.js

Google has announced LiteRT-LM.js, an upcoming package that will add browser support for running full large language models. That opens the door to generative AI applications that run entirely offline in the browser, with no API calls and no data leaving the user's device.

When to use LiteRT.js

LiteRT.js is a strong fit for any project that needs to run .tflite models in the browser and wants to maximize performance. The main things to account for before committing to it:

Model compatibility: Not every operation is supported on every backend. WebGPU has the most constraints; XNNPACK (CPU) has the broadest coverage. Before deploying, run your model through the @litertjs/model-tester package to confirm it works on your target backends.

Fallback handling: LiteRT.js doesn't automatically split a model across backends. If a model has one unsupported operation on WebGPU, the entire graph falls back to CPU. The standard pattern is to attempt WebGPU and recompile on wasm if it throws.

Model size: WebAssembly memory is capped at 2 GB, which limits the size of models you can load. Very large models may need quantization before they're viable in the browser.

Preprocessing: LiteRT.js handles inference only. Input preparation, resizing, normalization, tokenization, and output decoding are your responsibility. For complete model pipelines with pre- and post-processing included, Google's MediaPipe suite is worth looking at alongside LiteRT.js.

For projects already in the .tflite ecosystem or looking to bring native-class performance to browser-based ML, LiteRT.js is the most direct path there.

Got an article suggestion? Let us know

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.