Running Open Models Locally with Ollama

Ollama downloads and runs open models (Llama, Mistral, Qwen, Gemma...) on your laptop with one command. Good for learning, offline work and private data. Install from ollama.com, then:

ollama pull llama3.2          # ~2 GB, 3B parameters, 4-bit
ollama run llama3.2 "Explain a context window in one sentence."

A context window is the maximum amount of text, measured in tokens, that a language model can read and consider at one time when generating a response.

Ollama serves an HTTP API on localhost:11434. Note that open models still accept temperature and similar options:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Say hello in Hindi"}],
  "stream": false,
  "options": {"temperature": 0.2}
}'

{
  "model": "llama3.2",
  "message": {"role": "assistant", "content": "नमस्ते! (Namaste!)"},
  "done": true,
  "prompt_eval_count": 31,
  "eval_count": 9
}

import json, urllib.request

body = {"model": "llama3.2", "stream": False,
        "messages": [{"role": "user", "content": "Say hello in Hindi"}]}
req = urllib.request.Request("http://localhost:11434/api/chat",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req) as r:
    print(json.load(r)["message"]["content"])

• Small local models are much weaker than frontier API models at reasoning and tool use.
• Speed depends on your hardware; a laptop CPU gives a few tokens per second.
• For production self-hosting, teams use servers like vLLM or TGI on GPU machines.