Fine-Tuning vs. Prompting

When a well-crafted prompt is enough — and when you genuinely need a custom-trained model.

Start with prompting

Almost every use case should start with a well-crafted prompt. Modern frontier models have seen vast amounts of domain knowledge — with the right system prompt, few-shot examples, and output format instructions, they handle the majority of tasks better than a fine-tuned smaller model from two years ago. Fine-tuning is a significant investment: it requires labelled data, compute, evaluation infrastructure, and ongoing maintenance as models are updated. Make sure prompting genuinely fails before going further.

If you can achieve 90%+ of the quality you need with a good prompt, that is almost always the better choice.

When to fine-tune

Fine-tuning makes sense when: (1) You need a very specific output style, tone, or format that you cannot achieve reliably with prompting. (2) You have a narrow, repetitive task where consistent behavior matters more than general capability (e.g., entity extraction from a specific document type). (3) Context window cost is prohibitive — you would otherwise need to insert large instruction sets on every call. (4) Latency matters and you need to use a smaller base model. Fine-tuning is not a silver bullet for improving general capability — it makes the model better at one specific thing.

LoRA vs. full fine-tuning

Full fine-tuning updates all model weights — it is the most flexible but requires significant GPU memory and time. For most practitioners, LoRA (Low-Rank Adaptation) is the right approach: it freezes the base model and trains only a small adapter layer (~1% of parameters). This reduces training cost by 10–100× and produces models that are nearly as capable as full fine-tunes for most tasks. QLoRA adds 4-bit quantization, further reducing memory requirements — you can fine-tune a 7B model on a single consumer GPU.

Data requirements

For LoRA fine-tuning, 500–2000 high-quality examples is typically enough for a well-defined task. Quality trumps quantity: 200 carefully crafted, diverse examples beat 5000 noisy ones. Your data needs input-output pairs that exactly represent the task you want the model to do. If you cannot define the task precisely enough to create 200 consistent examples, the task may be too vague for fine-tuning. Common mistake: using model-generated examples to fine-tune the same model family — this amplifies existing biases rather than correcting them.

Providers and cost

For fine-tuning with managed APIs: OpenAI offers GPT-4o mini fine-tuning via API. Google offers Gemini 1.5 Flash fine-tuning on Vertex AI. For open-source models (Llama, Mistral, Qwen), use Together AI, Replicate, or Hugging Face AutoTrain for managed training, or run locally with Axolotl or LLaMA-Factory. A typical LoRA fine-tuning run on a 7B model costs $5–50 depending on dataset size and provider. Running the fine-tuned model typically costs the same as the base model.

Concepts: Fine-Tuning Concepts: Prompt Engineering Resources: Model Training Tools