Multimodal AI: Images, Documents, and Vision
How to effectively use AI vision — from screenshots and PDFs to complex document analysis.
What vision models actually see
When you send an image to a vision model, it is encoded into visual tokens — a grid of patches across the image, each represented as a vector. The model processes these alongside your text tokens. For a 1024×1024 image, Claude uses around 1,600 visual tokens; GPT-4o uses a similar approach. This means: larger images cost more tokens, and very small text in images may be difficult to read if the resolution is low. Resize and crop to focus the model on the relevant region before sending.
The five most valuable vision use cases
Document extraction: extract structured data from invoices, receipts, forms, and scanned contracts. This replaces expensive OCR + parsing pipelines. Screenshot debugging: paste a screenshot of an error, UI bug, or unexpected output and ask for diagnosis. Chart and graph analysis: describe a chart's trends, extract data points, identify anomalies. Visual QA for products: check product images for defects, consistency, or compliance against a spec. Presentation and slide review: provide feedback on a deck's visual clarity, layout, and information hierarchy.
PDF and document handling
All major frontier models accept PDFs directly via API. Anthropic, OpenAI, and Google each handle them differently under the hood: some render each page as an image, others use built-in text extraction. For text-heavy PDFs (reports, contracts, research papers), the text extraction path is more reliable and cheaper. For image-heavy PDFs (scanned documents, forms), image rendering is essential. When precision matters — legal contracts, financial documents — always verify extractions against the source rather than trusting output blindly.
Effective prompting for vision tasks
Be specific about what you want the model to look at. 'What does this chart show?' is weaker than 'What is the year-over-year revenue growth rate shown in the bar chart, and which year had the highest growth?' For extraction tasks, specify the output format: 'Extract all line items from this invoice as JSON with fields: description, quantity, unit_price, total.' For analysis tasks, provide context the model cannot see: 'This is a screenshot from our internal dashboard. The red values indicate alerts.'
When vision fails — and what to do
Vision models struggle with: small, dense text in low-resolution images; complex tables with merged cells; handwritten text (though Claude and Gemini have improved significantly); images with many similar elements that require counting or precise comparison. Workarounds: increase image resolution before sending; crop to the specific region of interest; for complex tables, convert to text via a dedicated OCR step first; for counting tasks, use structured detection prompts rather than open-ended 'how many...' questions.