Tools & AgentsPractitioner
Running Models Locally
How to run AI on your own machine — private, free, and offline.
Running a model locally means inference happens entirely on your hardware — no data leaves your machine, no API costs, no rate limits.
Tools (2026):
- Ollama — one-command install; OpenAI-compatible REST API, supports Llama 4, Gemma 4, Mistral Small 4
- LM Studio — GUI desktop app; browse HuggingFace, download, chat, and serve models
- Jan.ai — privacy-first local chat with built-in OpenAI-compatible server
Hardware requirements (Q4 quantisation):
| Model size | RAM / VRAM | Example models |
|---|---|---|
| 3–7B | 6–8 GB | Gemma 4 E4B, Mistral Small 4, Llama 4 Scout |
| 13–27B | 12–16 GB | Devstral Small 24B, Gemma 4 26B A4B |
| 70B+ | 40–80 GB VRAM | Llama 4 Maverick, Mistral Large 3 |
Apple Silicon (M1–M4) excels: unified memory means a 16 GB MacBook Air runs Llama 4 Scout or Mistral Small 4 at 40+ tokens/s with full privacy. Gemma 4 E4B targets phones (Pixel, Snapdragon) at near-zero latency.
In plain terms
Like running your own web server instead of using a cloud host — more control and privacy, but you manage the hardware.