Inference
The moment a trained model actually runs and generates output.
Inference is the process of running a trained model to produce output from new input. It's distinct from training, which is the expensive, one-time process of teaching the model.
When you send a message to Claude or GPT, you're triggering inference. The model processes your tokens through its billions of parameters and generates a response, one token at a time.
Key inference concepts:
Latency — time to first token (TTFT). Users notice anything over ~500ms.
Throughput — tokens per second. GPT-4o runs at ~100 tok/s; older models at 20–40 tok/s. Matters for streaming long responses.
Batch inference — processing many requests at once, more efficient, used for non-real-time jobs (the Claude Batch API cuts costs by ~50%).
On-device inference — running small models locally (Phi-3 Mini, Llama 3.2 1B) — no API call, private, lower latency, limited capability.
Inference cost — billed per input + output token. Frontier models cost $3–$75 per million tokens; efficient models cost $0.10–$0.40.
Hardware: GPUs (NVIDIA H100/H200, AMD MI300X) dominate inference workloads. Google uses custom TPUs. Apple Silicon enables on-device inference via the Neural Engine.
In plain terms
Training is writing a textbook over years. Inference is a student using that textbook to answer a question in seconds.