Quantization

Quantization reduces the numerical precision of model weights from 32-bit or 16-bit floating point to 8-bit integers (INT8) or 4-bit integers (INT4). A 7B parameter model at full 16-bit precision requires ~14 GB of memory. At 4-bit, it drops to ~4 GB — fitting on a consumer GPU or M-series Mac.

Common formats:

• GGUF/GGML — the standard for running quantized models with Ollama, LM Studio, and llama.cpp
• AWQ — Activation-aware Weight Quantization, better quality than naive INT4
• GPTQ — GPU-friendly post-training quantization
• QLoRA — combines 4-bit quantization with LoRA for memory-efficient fine-tuning

Quality tradeoff: 8-bit quantization has minimal quality loss. 4-bit trades ~1–3% on benchmarks for 4× memory reduction. 3-bit and 2-bit become noticeably degraded. Most practitioners use Q4_K_M or Q5_K_M as the sweet spot.

Practical rule: 1 billion parameters ≈ 0.5–0.6 GB at 4-bit quantization. A 7B model at Q4 = ~4 GB VRAM. A 70B model = ~40 GB.