Ollama: How to Run Any Open-Source LLM Locally with Your Existing Tools

Most AI coding tools and agents are built around a specific API, which means you're limited to whatever models that provider decides to offer. Ollama solves that by running a local LLM server that mimics the APIs those tools already speak. You point your tool at localhost instead of a cloud endpoint, and suddenly you can run any model from the Ollama model library without changing your workflow. This article covers what Ollama is, how to install and use it, how to integrate it with tools like Claude Code, and how it compares to alternatives like vLLM and LM Studio.

What Ollama does

Ollama is a headless LLM server that handles downloading, running, and managing open-source models locally. It presents a unified API endpoint, compatible with both the Anthropic and OpenAI API formats, so your existing tools can talk to it without modification. The only change required is updating the base URL in your tool's configuration.

Whether you're routing requests to a model running on your local GPU or configuring Ollama to forward traffic to a hosted model, the interface your tools see stays the same. You can also swap models at will without touching your tool's configuration beyond specifying the model name.

The model library

The Ollama library gives you access to a large catalog of open-source models including Llama, Gemma, Mistral, Qwen, DeepSeek, and many others. New models typically appear within days of their public release.

Running models locally means no API costs during development, no data leaving your machine, and the ability to work offline once a model is downloaded. For sensitive codebases or private documents, the privacy benefit alone is often worth the setup.

Installation

Ollama is available for macOS, Linux, and Windows. On macOS and Linux, install via the official script:

curl -fsSL https://ollama.com/install.sh | sh

This downloads the binary and sets up Ollama as a background service. After installation, the server runs on http://localhost:11434 by default. Full installation instructions for all platforms are on ollama.com.

Basic CLI usage

Launching the Ollama CLI opens a launcher menu showing integrated tools and agents you can start directly:

ollama

To download and run a specific model, use ollama run. If the model isn't already downloaded, Ollama fetches it automatically:

ollama run gemma

Once the download completes, you're dropped into an interactive chat session in the terminal. Type /bye to exit. This is useful for quick testing before wiring a model into a larger tool.

Integrating Ollama with Claude Code

Claude Code is an example of a tool that's straightforward to redirect to a local Ollama server. By default it talks to Anthropic's API. A few environment variables are enough to change that.

Add the following to your shell configuration file (~/.zshrc, ~/.bashrc, or ~/.bash_profile):

export ANTHROPIC_BASE_URL="http://localhost:11434"
export ANTHROPIC_API_KEY="ollama"
export ANTHROPIC_AUTH_TOKEN="ollama"

ANTHROPIC_BASE_URL redirects API calls from api.anthropic.com to your local Ollama server. The key and token variables need to be non-empty strings; Ollama doesn't validate them for local models, so any placeholder value works.

Apply the changes with:

source ~/.zshrc

Then launch Claude Code with a model flag to specify which local model to use:

claude --model gemma

The interface looks and behaves identically to the standard Claude Code experience. If you ask the model "who are you?", it may identify itself as Claude Code since the tool's system prompt instructs it to do so regardless of the underlying model. The harness and persona stay the same; only the model doing the reasoning changes.

Multimodal image input

Models with vision capabilities can accept image input directly in the ollama run command. Gemma is one example:

ollama run gemma:latest "What's in this image? /path/to/your/image.png"

The model returns a text description of the image contents. This is useful for tasks that require visual understanding without building a separate pipeline.

The HTTP API

Ollama exposes an OpenAI-compatible HTTP API while it's running. You can call it directly with any HTTP client:

curl http://localhost:11434/api/generate -d '{"model": "gemma", "prompt": "Why is the sky blue?"}'

The API includes /api/chat for conversational interactions and /api/embeddings for generating vector embeddings. Full documentation is available in the Ollama GitHub repository.

Docker deployment

For shared or production setups, Ollama can run inside a Docker container. The official documentation covers CPU-only configurations as well as GPU-accelerated setups for NVIDIA and AMD hardware. This is useful for providing a self-hosted LLM service to a team without requiring each developer to manage their own local installation.

Hardware and performance

Performance depends primarily on your GPU's VRAM. A rough guide based on available memory:

Under 24 GB of VRAM supports smaller to medium models with around a 4k token context window. 24 to 48 GB opens up larger models or context windows up to 32k tokens. 48 GB or more unlocks context windows of 256k tokens or larger, which makes long-form tasks and large codebase analysis practical.

For Apple Silicon users, Ollama's MLX backend takes advantage of the unified memory architecture and GPU Neural Accelerators in M-series chips, resulting in significant improvements in time-to-first-token and generation speed compared to running models on CPU alone.

Ollama vs. vLLM and LM Studio

Ollama vs. vLLM: Ollama is optimized for local development. It prioritizes ease of use and a simple CLI for downloading and running models on a development machine. vLLM is a library designed for high-throughput hosted serving, with advanced optimizations like PagedAttention and quantization that make it significantly faster under concurrent load. The practical split: use Ollama on your development machine, use vLLM when deploying a model as a production API service.

Ollama vs. LM Studio: Ollama is CLI-first, lightweight, and scriptable. LM Studio is GUI-first, with a desktop application for browsing, downloading, and chatting with models in a visual interface. Both are valid choices; the difference is mostly workflow preference. Ollama now includes an official desktop GUI for users who want a graphical interface without switching tools entirely.

When to use Ollama

Ollama is a good fit when you want to run open-source models locally with the least friction, integrate them with existing tools that speak the OpenAI or Anthropic API format, or eliminate API costs during development and testing. It's also the right choice if you need to keep data on-device for privacy reasons.

If you need a production-grade serving layer with high throughput and advanced quantization options, vLLM is the better tool. If you prefer a graphical interface for managing models, LM Studio is worth evaluating alongside Ollama.

Got an article suggestion? Let us know

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.