Measuring LLM quality: evals and RAG, part 3 of 3

AI Agent Memory: When It Beats Plain Search

AI models remember nothing between calls. How agent memory really works: plain search over past chats, extracted facts, stale-fact updates, ranking, and when each one wins.

10 min · 2026-11-02 · Watch on YouTube · Subscribe

After this video

You can choose between sending the whole chat, plain search over past rounds, and extracted facts with an update step, and test your agent's memory.

The short version

The model remembers nothing between calls, so agent memory is your code: save something now, put it back in a later prompt. Plain search saves every round (one user message plus one reply) and finds the closest ones by embedding. Extracted facts ask a model to pull out what's worth keeping, which keeps prompts small but can drop details. Either way, memories go stale: when the team moves from Yarn to Bun, the old fact has to be deleted or outranked by date. Short history? Send the whole chat. Long history with detailed questions? Plain search over rounds. Preferences that change? Extracted facts with an update step. Then test it with a planted fact that changes.

The model forgets everything

Every request to the model starts blank. A chat feels continuous only because your app sends the history again, so memory across chats is a save step plus a lookup before the next prompt.

Monday and Tuesday. Monday: 'we use Yarn, not npm'. Tuesday, new chat: the assistant runs npm install, because nothing saved Monday's chat.

Plain search over rounds

Save each round (one user message plus one reply) with its embedding, and pull the closest rounds for the new message. LongMemEval found rounds a better size to store than whole sessions, and squeezing them into short facts lost details.

Crisis averted. Tuesday's 'set up the project' finds Monday's round about Yarn, it goes into the prompt, and the agent runs yarn install.

Extracted facts

After each exchange a model pulls out the facts worth keeping (Mem0 does this). Prompts stay small, about 1,800 tokens per question in Mem0's tests against about 26,000 for the whole chat, at the price of extra model calls and lost detail.

One line. From Monday's chat it keeps: package manager is Yarn.

Stale facts

A memory that only adds ends up holding two facts that disagree, and the old one can rank first. Compare each new fact with similar old ones and pick ADD, UPDATE, DELETE or nothing; with plain search, date every round and tell the model newer beats older.

Yarn to Bun. 'We're on Bun now' contradicts 'uses Yarn', so the old memory is deleted and Bun is saved.

What goes back in

Score memories on relevance, recency and importance (the Generative Agents paper), scale the three to the same range, add them up, and keep a short list: in LongMemEval GPT-4o fell from 87% with only the right chats to 61% with the whole history.

Which one wins, and the test

In Mem0's tests the whole chat scored highest (about 73%) but its slowest answers took about 17 s, against about 1.4 s for extracted facts. Pick by history length and question type, then test the five LongMemEval question kinds.

The test. Plant Yarn in chat one, Bun in chat five, ask in chat ten. If it says Yarn, the update step is broken.

The code

Plain search: save a round, recall the closest
async function saveRound(u: string, r: string) {
  const text = u + "\n" + r;
  await db.add(text, await embed(text));
}

async function recall(msg: string) {
  const q = await embed(msg);
  return db.nearest(q, 2);
}

db and embed stand for your vector store and embedding call. Stamp each round with its date too, so newer can beat older.

The update step for extracted facts
type Op =
  | { op: "ADD"; text: string }
  | { op: "UPDATE"; id: string; text: string }
  | { op: "DELETE"; id: string }
  | { op: "NOOP" };

// show the model the new fact and its closest old memories,
// ask for one Op, then apply it
async function remember(fact: string) {
  const similar = await db.nearest(await embed(fact), 5);
  const op: Op = await chooseOp(fact, similar);
  await apply(op);
}

The Mem0 idea: a new fact that contradicts an old one deletes it (and still gets saved).

The one test to write first
await chat(1, "We use Yarn, not npm.");
await chat(5, "We moved to Bun.");
const answer = await chat(10, "Which package manager do we use?");
if (!/bun/i.test(answer)) throw new Error("update step broken");

A knowledge-update question, one of LongMemEval's five kinds.

Check yourself

  1. Why does a new chat with your coding assistant forget what you said yesterday?

    Each model request is stateless; only what your app saves and puts back in the prompt carries over.

  2. A user said 'we use Yarn' and later 'we moved to Bun'. What must the memory do so the agent uses Bun?

    Compare the new fact with similar old ones and delete or update the contradicted one (and/or date memories so newer wins).

  3. How would you test that your agent's memory handles a changed fact?

    Plant the old fact in an early chat (we use Yarn), the new one later (we moved to Bun), then ask in a later chat; answering Yarn means the update step is broken.

Sources

Every claim in this video was checked against these pages. Open one to see the claims it backs and the exact words they rest on.

  • platform.openai.com/…/conversation-state 2 claims
    • Model API text generation requests are independent and stateless; multi-turn chat works by resending prior messages.
      while each text generation request is independent and stateless, you can still implement multi-turn conversations by providing additional messages as parameters to your text generation request.
    • Some APIs can keep the conversation for you (OpenAI previous_response_id) so the app does not resend it; the model still reads it each turn.
      another way to manage conversation state is to share context across generated responses with the previous_response_id parameter.
  • docs.claude.com/…/memory-tool 4 claims
    • With the memory tool, memory lives entirely in your application; the model only requests memory operations.
      memory lives entirely in your application.
    • Claude's memory tool: Claude stores what it learns in files under /memories and reads them back in later conversations; it is client-side, your app runs the operations against storage you control.
      as it works, claude stores what it learns in files under /memories and reads them back in later conversations to continue earlier work. because the memory tool is client-side, claude only requests memory operations.
    • Memory tool docs: periodically delete memory files that haven't been accessed in a long time.
      periodically delete memory files that haven't been accessed in a long time.
    • Memory tool docs: add validation that strips sensitive data before writing.
      for stronger guarantees, add validation that strips sensitive data before your handler writes the file.
  • arxiv.org/…/2410.10813v2 10 claims
    • LongMemEval decomposes sessions into rounds: one user message followed by one assistant response.
      each session can be further decomposed into rounds: one user message followed by one assistant response.
    • LongMemEval: round is a better granularity than whole sessions for storing history; compressing into individual user facts harms overall performance due to information loss but improves multi-session reasoning.
      instead of sessions, round is the more optimal granularity for storing and utilizing the interactive history. while further compression into individual user facts harms overall performance due to information loss, it improves the multi-session reasoning accuracy.
    • LongMemEval tests five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention.
      five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
    • Knowledge updates: the ability to recognise changes in the user's personal information and update knowledge over time.
      knowledge updates (ku): ability to recognize the changes in the user's personal information and update the knowledge of the user dynamically over time.
    • Time-aware indexing and query expansion (timestamps on facts, narrowed search range) improved memory recall for temporal reasoning by 6.8-11.3% with a strong model.
      we propose a simple indexing and query expansion strategy to explicitly associate timestamps with facts and narrow down the search range, improving the memory recall for temporal reasoning by 6.8% ∼ \sim 11.3% when a strong llm is employed for query expansion.
    • In LongMemEval, GPT-4o scored 0.870 with only the evidence sessions (oracle) vs 0.606 reading the full ~115k-token history (30.3% drop).
      gpt-4o - 0.870 0.606 30.3%
    • LongMemEval_S has about 115k tokens per problem.
      longmemeval s with approximately 115k tokens per problem
    • Expanding memory keys with extracted user facts improves recall@k by 9.4% and QA accuracy by 5.4%, over a flat index with values as keys.
      further expanding the keys with extracted user facts improves both memory recall (9.4% higher recall@ k k ) and downstream question answering (5.4% higher accuracy).
    • Long-context models show 30-60% drops on LongMemEval_S compared to the evidence-only setting.
      long-context llms show a 30% ∼ \sim 60% performance drop on longmemeval s
    • Compressing sessions into summaries or user facts can lose information, hurting detailed questions.
      compressing sessions into summaries or user-specific facts, as seen in prior work such as zhong et al. (2024) and du et al. (2024) , can lead to information loss,
  • arxiv.org/…/2304.03442v2 6 claims
    • Relevance can be computed as cosine similarity between embedding vectors of a memory and the query.
      we calculate relevance as the cosine similarity between the memory's embedding vector and the query memory's embedding vector.
    • Generative Agents' retrieval combines relevance, recency and importance.
      a memory retrieval model combines relevance, recency, and importance to surface the records needed to inform the agent's moment-to-moment behavior.
    • Recency is an exponential decay since the memory was last retrieved.
      we treat recency as an exponential decay function over the number of sandbox game hours since the memory was last retrieved.
    • Importance is rated by the model on a 1-10 scale, 1 mundane (e.g. brushing teeth), 10 extremely poignant.
      on the scale of 1 to 10, where 1 is purely mundane (e.g., brushing teeth, making bed) and 10 is extremely poignant
    • Importance score is generated at the time the memory object is created.
      the importance score is generated at the time the memory object is create
    • Final score is a weighted sum of normalised recency, importance and relevance (all weights 1); top-ranked memories that fit in the context window go in the prompt.
      the top-ranked memories that fit within the language model's context window are included in the prompt.
  • arxiv.org/…/2504.19413v1 9 claims
    • Mem0 has an extraction phase that processes new messages (a message pair) and historical context to create candidate memories.
      the extraction phase processes messages and historical context to create new memories.
    • Mem0 extraction starts on each new message pair, typically a user message and an assistant response.
      this pair typically consists of a user message and an assistant response, capturing a complete interaction unit.
    • In Mem0's LOCOMO tests, Mem0 used about 1,764 memory tokens per question versus about 26,031 for full context, and p95 total latency 1.440 s vs 17.117 s; J scores 66.88% vs 72.90%.
      full-context 26031 - - 9.870 17.117 72.90
    • Mem0 row: 1764 tokens, p95 total 1.440 s, J 66.88%.
      mem0 1764 0.148 0.200 0.708 1.440 66.88
    • Full-context (~26,000 tokens) still achieves the highest J score (~73%) but p95 latency around 17 seconds because the model reads the whole conversation every query.
      despite these improvements, a full-context method that ingests a chunk of roughly 26,000 tokens still achieves the highest j score (approximately 73%). however, as shown in figure 4(b) , it also incurs a very high total p95 latency-around 17 seconds-since the model must read the entire conversation on every query.
    • Mem0 update operations: ADD new, UPDATE with complementary info, DELETE when contradicted, NOOP when no change.
      add for creation of new memories when no semantically equivalent memory exists; update for augmentation of existing memories with complementary information; delete for removal of memories contradicted by new information; and noop when the candidate fact requires no modification to the knowledge base.
    • Mem0's update phase compares extracted memories against similar existing ones and applies an operation via a tool call.
      the update phase evaluates these extracted memories against similar existing ones, applying appropriate operations through a tool call mechanism.
    • In Mem0's tests the best RAG setup (k=2, 256-token chunks) scored J 60.97%.
      256 0.255 0.699 0.802 1.907 60.97
    • Mem0 vs full context: Mem0 reduces p95 latency to about 1.44 s, a 92% reduction.
      lower p95 latencies of around 1.44 seconds (a 92% reduction)
  • arxiv.org/…/2309.02427v3 1 claim
    • CoALA: semantic memory stores facts about the world; episodic memory stores sequences of the agent's past behaviours (past events).
      semantic memory stores facts about the world ( lindes and laird, 2016 ) , while episodic memory stores sequences of the agent's past behaviors

Spotted a mistake? Say so in the video's comments. Fixes are listed on corrections.