Memory

Hybrid retrieval

Memory retrieval is genuinely hybrid: a single retrieval pass combines meaning-based search (do the stored memory and the query mean the same thing, even with different words) with keyword search (does the stored memory contain the same terms) over the same corpus, merges the two into one ranked list, and then a lightweight selection step narrows that list down to the handful most relevant to the current turn. This replaces an older design where a model judge and a pure similarity search were two separate, alternately-chosen paths — today they run together as one path, by default.

Updated today

TL;DR

The default recall path in src/lib/agentMemory.ts queries the user's stored memories two ways at once — by meaning (vector similarity) and by keyword (literal term matching) — merges the two rankings into a single ordered candidate list, then runs a lightweight selection pass to pick the few entries most relevant to the query. If no embedding is available the merge runs on keyword matches alone; if the selection pass is unavailable the merged ranking is used as-is; if the whole path is unavailable retrieval falls back further to an older, model-judged manifest path and finally to a plain recency window. This hybrid path is what seeds context automatically at the start of a turn. An on-demand memory-search tool available mid-turn still runs the older manifest-and-judge path on its own.

Overview

At a glance

Module: src/lib/agentMemory.ts
Default entry point: hybridRecall(query, options)
Signal 1: Meaning-based (vector) similarity across the full corpus
Signal 2: Keyword / literal-term matching across the same corpus
Combine step: Both rankings are merged into one ordered candidate list
Selection step: A lightweight pass narrows the merged candidates to the handful most relevant to the turn
Legacy paths (still present): A manifest-plus-judge path and a similarity-only path, used as fallbacks and by the on-demand tool

Hybrid retrieval exists because neither signal alone is enough. Meaning-based search finds entries that share intent with the query even when the wording is completely different, but on its own it can miss an exact term the user actually cares about, especially once the corpus grows large. Keyword search catches exact identifiers, names, and jargon but misses paraphrases and indirect questions. Running both in the same call and merging the results covers both shapes of query at once, instead of asking the system to guess in advance which one a given query needs.

How hybrid retrieval works

One retrieval call, two signals, one merged ranking.

Signal	How it matches	Catches
Meaning-based	Compares the query's embedding against the embeddings of stored memories	Paraphrases, indirect phrasing, conceptually related entries the exact words don't cover
Keyword	Matches literal terms and phrases against the stored memory text	Exact identifiers, names, jargon, and tags the user typed verbatim

Both signals run against the same stored memories in the same call — there is no branch where the system picks one signal instead of the other. The two rankings are merged into a single ordered list of candidates, and only then does a separate, lightweight step narrow that list down to what actually gets attached to the turn.

The meaning-based signal

Vector similarity over the full memory corpus, not just a recency window.

The query is embedded and compared against the stored embedding for every candidate memory. This is the signal that answers questions like "what did we decide about the Acme deal" even when the stored note never uses the word "decide" or "Acme deal" verbatim — it matches on meaning, not on shared vocabulary.

Unlike the older per-turn recall path, the meaning-based signal in hybrid retrieval is not limited to a small window of the most recent rows — it runs across the user's full stored corpus, which matters once a user has accumulated more memories than fit comfortably in a manifest.

The keyword signal

Literal term matching, running alongside the meaning-based signal.

The same query is also matched against the stored memory text for literal overlap — the terms, names, and tags the user actually typed. This is the signal that answers a query like "find anything tagged pricing that mentions enterprise": the value here is exact recall on vocabulary, which meaning-based matching alone can under-rank if the phrasing is unusual.

If no embedding could be produced for the query — for example, the embedding step is unavailable — the keyword signal still works on its own, so retrieval degrades to keyword-only rather than failing outright.

Narrowing the candidates

A lightweight step turns a merged ranking into the handful worth attaching to the turn.

Once the meaning-based and keyword rankings are merged into one ordered candidate list, a lightweight selection step reviews the top candidates and picks the few that are actually worth attaching to the current turn — by default a small number of entries out of roughly a dozen merged candidates. This step exists because a merged ranking is still just a ranking; it takes a judgment call to decide how many of the top entries are genuinely relevant versus merely nearby.

When the candidate set is already small enough, the selection step is skipped and the merged ranking is used directly. When the selection step itself is unavailable, the system falls back to using the merged ranking order as-is rather than returning nothing.

Why hybrid

Both signals exist because both shapes of query are common.

A user who asks "what did we say about the Acme deal last week" wants the system to find the right memory even if their exact phrasing does not appear in the stored text — the meaning-based signal handles that. A user who asks "find anything tagged with 'pricing' that mentions enterprise" wants high recall on lexical overlap — the keyword signal handles that. Running both signals together, on every automatic recall, means the system does not have to guess which shape of query it is looking at before it searches.

Note

Retrieval quality also depends on what is in the corpus in the first place, not just how it is searched. Separate from any single retrieval call, an automatic nightly memory review runs on its own schedule — it reviews, merges, and prunes stored memories, keeping what is useful and retiring what isn't, so the corpus retrieval runs against stays healthy as it grows instead of just getting noisier over time.

The embedding model

One shared embedding space spans stored memories and uploaded files, so a query against either shares semantics.

Property	Value
Used for	Stored memories and uploaded-file passages, so both are searchable in the same embedding space
Distance metric	Cosine similarity
Combined with	A keyword/literal-match signal over the same stored text, as described above

Warning

The embedding dimensionality is baked into the storage layer. Switching to a materially different embedding model requires a migration that re-creates the vector column and re-embeds every existing row — not a casual change.

Tuning knobs

What you can change without touching storage.

Knob	Default	Effect
Candidate count	12 merged candidates	Larger gives the selection step more to choose from at the cost of a larger merge; smaller narrows the search sooner.
Selection cap	3 entries	How many of the merged candidates the selection step is allowed to keep for the turn.
Legacy manifest size	80 rows	Still used by the older manifest-and-judge fallback path and by the on-demand memory-search tool.
Legacy similarity threshold	0.5	Still used by the older similarity-only fallback path; higher returns fewer, more precise matches.

Fallback behaviour

Each layer degrades to the next rather than returning nothing.

01
No embedding available
The meaning-based signal is skipped and the merge runs on the keyword signal alone.
02
Selection step unavailable
The merged, ranked candidate list is used directly instead of being narrowed further.
03
Hybrid path unavailable
Retrieval falls back to the older manifest-and-judge path, which reads a compact preview of the user's most recent memories and picks the relevant ones from that window.
04
Nothing else available
Retrieval falls back to a plain recency window — the most recent memories, unranked by relevance. The caller (usually a skill prompt) is responsible for handling absence gracefully.

Calling it from tools

The automatic per-turn recall and the on-demand tool call don't currently run the same path.

Caller	Path used	Use case
Automatic per-turn context seeding	Hybrid recall (meaning + keyword, merged, then narrowed)	Runs at the start of a turn to attach relevant memories without the model asking for them
`search_memories`	Older manifest-and-judge path	Skill explicitly asks for a memory lookup mid-turn
`brain_search_files`	Similarity search over uploaded-file passages	Skill wants to find passages inside uploaded documents

Tip

A skill that already has the context block prepended at turn start should not call the memory-search tool as a first move. The context block is built from the automatic hybrid path; calling the tool again re-runs a different, narrower path for no benefit. Use the tool when the model realises it needs more than what the context block already includes.

File map

src/lib

agentMemory.tsaddMemory, hybridRecall, recallMemories, searchMemories, recentMemories

brain-tools.tsbrain_search_files executor over uploaded-file passages

skills

memory.tsconversation compression — see the compression page

src/app/api/chat

v2/route.tscalls hybridRecall for automatic per-turn context seeding, falling back to the older recall path on failure

supabase/migrations

202602080002_agent_memories.sqlagent_memories table + vector index

Glossary

Hybrid retrieval: A single retrieval call that combines a meaning-based (vector) signal and a keyword signal over the same stored memories, merges the two, then narrows the merged list with a lightweight selection step. The default recall path today.
Meaning-based signal: Vector similarity between the query's embedding and each stored memory's embedding. Matches on intent, not shared vocabulary.
Keyword signal: Literal term/phrase matching between the query and the stored memory text. Matches on shared vocabulary, including exact names and tags.
Selection step: The lightweight pass that narrows a merged, ranked candidate list down to the handful actually attached to the current turn.
Legacy manifest-and-judge path: The older recall design: a compact preview of the user's most recent memories is assembled and a model picks the relevant ids from it. Still used as a fallback and by the on-demand memory-search tool.
Legacy similarity-only path: The older design where retrieval was pure vector similarity against a threshold, with no keyword signal. Still present as a narrower fallback.
Recency fallback: The last-resort path retrieval falls to when nothing else is available. Returns the most recent memories, unranked by relevance.