ProtocolsBeginner
Streaming
Receiving an AI's response token by token as it's generated, not all at once.
Streaming sends each token to the client as soon as the model generates it, rather than waiting for the full response to complete. This is why ChatGPT and Claude "type" text progressively rather than showing a blank screen then dumping the answer.
Technical mechanism: Server-Sent Events (SSE) or WebSockets. The server pushes a delta (one or a few tokens) as each is generated. The client appends it to the UI in real time.
When to use:
- Always in user-facing apps: Streaming is dramatically better UX for responses over ~100 tokens. A 20-second wait feels instant when you see output after 200ms.
- Not needed for batch processing: If you're processing documents offline and don't need real-time display, non-streaming is simpler.
Practical notes:
- Structured output (JSON) works poorly with streaming — you need the full response to parse JSON. Either stream and parse at the end, or don't stream when you need structured output.
- Token counts arrive in the final delta for billing purposes.
- Most SDKs handle streaming with a simple
stream=Trueparameter and async iteration over the response.
In plain terms
A live sports ticker vs. a post-match summary. The ticker streams each score as it happens; the summary appears once the game ends.