AI Cost Optimization Playbook
Cut your AI API spend by 60–90% without sacrificing quality — techniques that actually work.
Where the costs actually come from
API costs = (input tokens + output tokens) × price/M. For most applications, input tokens dominate — system prompts, conversation history, and retrieved documents are repeated on every call. A 2000-token system prompt called 1 million times per month costs more than the entire infrastructure for many startups. Measure your token split (input vs. output) before optimizing — it determines which techniques will have the most impact.
Prompt caching (biggest lever)
Anthropic and OpenAI both offer prompt caching: if the beginning of your prompt is identical across calls (system prompt, static documents, few-shot examples), the cached portion is charged at 10% of normal input cost on subsequent calls. For a 2000-token system prompt repeated 1M times/month: uncached = $2 per M input tokens × 2000 tokens × 1M calls = $4,000/month. Cached = $0.40/month for the system prompt portion. This is the single highest-impact optimization for production systems.
Model routing
Not every request needs your most capable model. Classify incoming requests by complexity — simple factual questions, classification tasks, and format transformations can go to a small model (Haiku, GPT-4o-mini) at 10–20× lower cost. Complex reasoning, synthesis, and creative tasks go to your frontier model. A 70/30 split between small and frontier models can cut overall costs by 60% with minimal quality loss. Implement routing as a fast, cheap classifier call before the main call — the meta-cost is tiny.
Context window hygiene
Conversation history grows with every turn — by turn 10, you may be sending 8000 tokens of history to reconstruct a context the model could summarize in 200 tokens. Implement conversation summarization: periodically have the model compress older turns into a summary that replaces the raw history. For RAG pipelines, retrieve fewer chunks with higher precision rather than padding the context with marginally relevant passages. Remove boilerplate from prompts — every redundant instruction costs money.
Batch API and async processing
For tasks that do not need real-time response (document processing, nightly analysis, offline enrichment), use batch APIs. Anthropic's Message Batches API and OpenAI's Batch API both charge 50% of normal pricing for async jobs that complete within 24 hours. This halves the cost of all offline workloads. Pair with caching and model routing and you can run the same batch workload for 20–30% of the naive real-time cost.
More Intermediate guides
Build Your First AI Agent
From a single LLM call to an autonomous multi-step agent — the practical path.
RAG Architecture: Ground AI in Your Data
Build retrieval-augmented generation pipelines that give models access to your documents.
Fine-Tuning vs. Prompting
When a well-crafted prompt is enough — and when you genuinely need a custom-trained model.