Conversation compression

Long conversations break models. Once the buffer exceeds the model's context window the request fails outright; well before that, quality degrades as the model spends attention on stale content. The cascade exists so the chat handler can hand the model a clean, compact buffer on every turn, regardless of how long the underlying conversation has been running.

The order is intentional. The cheapest layer that fixes the problem runs first. If the buffer is bloated by a single huge tool result, layer 1 fixes it without touching anything else. If the buffer is long but mostly recent, layer 2 fixes it by dropping old messages. Only when the conversation is genuinely too rich to keep in full does the cascade reach layer 4 and spend a model call to write a structured summary.

The function takes a shallow clone of the input so it never mutates the caller's array. The layersApplied array tracks which layers ran for the current call; the return value exposes it so the caller can log compression behaviour without re-deriving it. The maxTokenBudget shown here is a default the caller can override — in practice the chat handler passes a budget appropriate to the tier serving the conversation, which is why the effective ceiling scales with the tier in use rather than being one fixed number in production.

Tool results are the biggest individual contributors to context bloat. A scrape of a content-heavy page can return tens of thousands of characters; a list of leads from a CRM can be hundreds of kilobytes; a generated document can dwarf the rest of the conversation by an order of magnitude. Layer 1 catches those by spotting messages that look like raw tool output (assistant role, starts with a JSON token or a code fence, or simply over 4,000 characters) and clipping them to the configured threshold.

The truncation appends an explicit marker so the model knows the content was cut. This matters: a silently truncated tool result reads as if the tool returned exactly that much, which can mislead the next turn into incorrect assumptions about completeness.

Layer 2 keeps the most recent recentKeep messages and drops everything before them. The default is 10. The assumption is that the most recent interactions carry the most contextual weight for the next turn; older content can be reconstructed through retrieval against persistent memory or, if needed, by reading the chat history directly.

The drop is lossy and intentional. Anything that mattered from the dropped range should have been written to persistent memory by an earlier turn. Skills designed for long sessions write memory aggressively for this reason.

Layer 3 protects the last four messages, then iterates over the rest in chunks of groupSize (default 6). Each chunk becomes a single synthetic assistant message that lists the role and the first hundred characters of each member. The result keeps the timeline visible (the order of the bullets reflects the original order of the messages) without paying for full content storage.

No model call is involved. The summaries are mechanical and deterministic. That is the point: layer 3 has to be cheap enough to run frequently without affecting turn latency or cost.

Layer 4 calls compressConversation, which splits the buffer into recent messages to keep verbatim and older messages to compress, then sends the older portion to a model on the Allegro tier with a structured system prompt. The model produces a bullet list of decisions, facts, names, numbers, and constraints. The result replaces the older portion of the buffer and the recent messages stay attached.

If the compression call fails for any reason, the cascade does not abort. It falls back to keeping only the last four messages, which guarantees a buffer the next turn can read even in degraded conditions.

The prompt is the contract between the cascade and the model. The model is told to retain every load-bearing fact and to drop everything that is decoration. The output is forced into bullet points so the downstream turn sees a structured block that fits comfortably above the recent messages.

Structured replay is an append-only, faithful reconstruction of exactly what happened in a conversation — including tool calls and their results — rather than a paraphrased summary. The stable parts of the prompt are kept byte-identical from one turn to the next, so a long conversation's shared prefix is cheap to reprocess instead of being rebuilt from scratch every turn.

Structured replay is not on for every conversation type. Where the history cascade documented above is the general-purpose mechanism, structured replay is a more specific reconstruction used where a faithful, tool-call-accurate record of the conversation matters more than a compact summary of it.

Working-set trimming keeps the most recent chunk of tool output in full detail and distills everything older into a one-line outcome. Unlike the cascade's layer 1 truncation, which only fires once the whole buffer is over budget, working-set trimming runs automatically on every tool round regardless of overall buffer size — it is a standing discipline on tool output specifically, not a budget-triggered layer.

The distillation is recoverable: if the model actually needs the detail from an older tool result again, it can re-fetch it rather than being permanently cut off from it, which is a different trade-off than the cascade's layer 1 truncation described above.

The whole-history digest is a hard ceiling backstop: if a conversation still risks overflowing after everything else, the oldest middle portion of the conversation collapses to one line per turn, while the very first turn and the most recent turns stay verbatim. It exists for the case none of the other mechanisms are built to fully absorb — a conversation that is long enough, and dense enough throughout, that even cascading compression and working-set trimming leave it near the ceiling.

The same mechanism powers the manual "compact this conversation" action available to a user — triggering it runs the identical whole-history collapse on demand rather than waiting for the ceiling to be reached automatically.