Why we stream token-by-token instead of waiting for the final answer

Waiting 20–30 seconds for a multi-step agent task to finish, then dumping a wall of text on the user all at once, feels broken — even when the underlying reasoning was fast and entirely correct. People calibrate trust in a system by watching it work, not by staring at a spinner and hoping. UnderOcean streams every token as it's generated, plus structured events for each reasoning step, over Server-Sent Events (SSE) — a deliberately boring, deliberately reliable choice.

The problem with waiting for the whole answer

A multi-step agent task — call a tool, read the result, decide whether another call is needed, call the model again, finally produce an answer — can easily take longer than a single completion call. If nothing is visible until the very end, a user watching a blank screen for that whole duration has no way to tell "this is working normally" from "this is stuck," and every extra second erodes confidence in the product regardless of whether the agent is actually doing something reasonable.

What gets streamed, and as what

UnderOcean's execution contract is intentionally simple: a client starts an execution, then opens one EventSource connection and receives a small, fixed set of event types as the agent runs:

POST /api/v1/executions             → { exec_id, status: "pending" }
GET  /api/v1/executions/{id}/stream → EventSource SSE

event types: token · step · tool_call · observation · done · error

token events stream the model's output as it's generated, so the final answer appears with the familiar "typing" effect users already expect from modern chat interfaces. Alongside that, step, tool_call, and observation events surface what the agent is doing while the final answer is still being formed — which tool it's calling, what came back, and which reasoning step it's on — so the UI can show "searching the knowledge base" or "calling the pricing API" instead of a generic loading indicator with no information behind it.

Why SSE and not WebSockets

SSE is a deliberately boring choice, and that's the point for this use case. It runs over plain HTTP — no separate connection upgrade, no dedicated WebSocket infrastructure to provision, monitor, or scale independently. It's one-directional by design, which matches the shape of this problem exactly: the client doesn't need to send messages back over the same connection mid-stream, it just needs to receive a sequence of events until done or error closes the run. Browsers' built-in EventSource handles reconnection semantics automatically, and because it's just HTTP, it degrades gracefully through the same infrastructure — proxies, load balancers, logging — that already handles every other API call, instead of requiring special-cased handling for a different protocol.

One engine, every entry point

This isn't a one-off feature bolted onto a single screen. The same generator function that streams an execution's events backs the in-app flow test panel, the Consumer API's POST /v1/agents/{id}/execute when called with stream: true, and public flow-share execution pages — so a public, anonymous-facing share link gets the same live reasoning visibility as an internal test panel, not a stripped-down version. That consistency is deliberate: transparency about what an agent is doing shouldn't depend on which surface happens to be rendering the execution.

What this buys beyond "feels faster"

Streaming isn't purely a perception trick — it changes what's actually possible to build on top of an execution. A UI can react to a tool_call event the instant it arrives (showing a "searching your documents" indicator tied to the actual tool being invoked, not a generic one), and a monitoring or logging system can react to error events in real time rather than waiting for a request to fully complete and then inspecting a final payload. None of that is available if the only signal a client gets is a single response at the very end.

FAQ

Does streaming add overhead compared to a single blocking response? No — the same tokens and events are produced either way; streaming just delivers them as they're generated instead of buffering everything until the end.

What happens if the connection drops mid-stream? EventSource's built-in reconnection behavior handles transient drops; the execution itself continues server-side regardless of whether a particular client connection is currently attached to it.

Can a client ignore the intermediate step/tool_call/observation events and just render the final token stream? Yes — a minimal client only needs to handle token and done/error; the intermediate events are additive information for clients that want to render richer reasoning visibility, not a requirement for basic functionality.

Is this the same streaming mechanism used for a flow, or only standalone agents? The same one — an agent_node or llm_node inside a flow streams through the identical event-type contract as a standalone agent execution.