Managing context windows at scale without losing conversation history

Every model's context window is finite, and any agent that runs long enough — hundreds of support turns, dozens of documents read over a multi-day research task — will eventually exceed it. Truncating old messages is the obvious fix and the wrong one: it silently drops information the agent may still need, like a customer's account details from turn three, a decision made twenty turns ago, or a constraint the user stated once at the very start of the conversation. UnderOcean instead manages context with three cooperating layers, each responsible for a different kind of information surviving past the point where raw messages would fall out of window.

Why naive truncation fails

The simplest context strategy — keep the last N messages, drop the rest — treats every message as equally disposable once it ages out. In practice, conversations aren't uniform: most turns really are safe to forget, but a handful carry information the agent needs for the rest of the conversation (an account ID, a stated preference, a decision that later turns depend on). Naive truncation can't tell those apart, so it either keeps a wastefully large window to avoid losing anything, or drops something important and produces a response that contradicts what the user already told it.

Layer 1 — the rolling running summary

The first layer keeps the narrative of the conversation bounded regardless of length. Instead of keeping every raw message, older turns are periodically condensed into a compact running summary — updated every turn or on-demand depending on configuration — that preserves what happened and why, without preserving every word used to say it. This is the layer that keeps an agent coherent about context — what's already been discussed, what's already been tried — as a conversation grows far past what would fit in a single prompt.

Layer 2 — structured fact extraction

A running summary is good at narrative, but bad at guaranteeing that a specific fact survives summarization untouched — a phone number, an order ID, an exact preference the user stated. The second layer runs as a periodic background job (a Celery task, not an inline blocking step) that extracts durable, structured facts out of the raw conversation into an explicit profile: names, identifiers, preferences, decisions. These extracted facts sit alongside the running summary rather than depending on it, so a detail doesn't quietly degrade into "something about an order" after enough rounds of re-summarization.

Layer 3 — long-term memory across sessions

The first two layers manage a single conversation. The third is what lets an agent recall relevant facts and summaries from a previous session entirely — so a user who talked to a support agent last week doesn't have to repeat context that was already established, and a research agent picking up a task days later can recall what it already found without re-reading everything from scratch.

RAG fills a different gap: external knowledge, not conversation memory

It's worth being precise about what these three layers are not solving: retrieving relevant documents from a knowledge base. That's a separate concern, handled by hybrid search (dense + BM25) over a project's knowledge base, populated into retrieved_docs either by an agent's own on-demand retrieval decision or by a flow's explicit rag_node. Context management keeps a conversation's own history usable at scale; RAG brings in information the conversation never contained in the first place. Both feed the same agent state, but they solve different problems and would fail in different ways if conflated.

What the agent actually sees

All of this comes together in a single state object threaded through the agent's reasoning loop:

class AgentState(dict):
    messages: Annotated[list[AnyMessage], add_messages]
    context: dict[str, RunningSummary]      # Layer 1 — rolling summary
    extracted_profile: dict                 # Layer 2 — extracted facts
    recalled_memories: list[str]            # Layer 3 — long-term memory
    retrieved_docs: list                    # RAG results, a separate concern
    inference_steps: list                   # streamed to the UI

The practical result is an agent that "remembers" what matters — durable facts, prior decisions, relevant history from earlier sessions — without needing to keep every raw message in context forever, and without the unpredictable failure mode of naive truncation silently dropping something important.

FAQ

Does summarization mean the agent loses precision over a long conversation? Precision-critical facts are the job of the extracted-profile layer specifically, so they don't depend on surviving repeated summarization the way free-text narrative does.

Is fact extraction synchronous, adding latency to every turn? No — it runs as a background Celery task, not an inline step in the request path, so it doesn't add latency to the turn that triggered it.

How is this different from a vector database storing chat history? A vector store retrieving similar past messages is a form of RAG over conversation history; the three layers here are specifically about keeping a single ongoing agent's working context correct and bounded, including facts that a similarity search over raw messages might not surface as the most relevant hit.

Does long-term memory recall happen automatically, or does the agent request it? It's populated into agent state as part of the execution setup for agents configured to use it — the same way retrieved documents or the running summary are available to the agent's reasoning without the agent needing to explicitly ask for them turn by turn.