Skip to content

Search articles, repositories, topics and pages.

Agents

Understanding Memory in AI Agents

A rigorous guide to agent memory as a governed state subsystem: what is stored, how it is updated and retrieved, and how to evaluate it safely.

TIAN NEWS AI

Published Updated 11 min read

A selective agent-memory system retrieves a small set of versioned records into the current context while preserving temporal and provenance information

Updated

The taxonomy, write and retrieval architecture, evaluation guidance and product examples were checked against peer-reviewed research and current Mem0 and LangGraph documentation on October 1, 2026.

Tested with

CoALA · LoCoMo · LongMemEval · LongMemEval-V2 preprint · Mem0 · LangGraph

An AI agent does not remember in the human sense. A language model call receives an input context and produces an output; continuity across calls is created by the surrounding application. That application may retain conversation state, checkpoints, facts, events, instructions or external records, then select some of them for a later model call.

A useful engineering definition is therefore:

Agent memory is a selectively managed state subsystem that writes, maintains, retrieves and governs information beyond a single model call.

This definition is deliberately broader than a vector database and narrower than “everything the agent can access.” It includes the policies that determine what becomes a memory, how a newer observation affects an older one, which records may cross a user or tenant boundary, and how retrieved evidence enters a finite context window.

Key takeaway

Memory is not a transcript archive and it is not the model's context window. It is a governed process for preserving and reintroducing state when that state is relevant, authorized and sufficiently reliable.

What memory is—and what it is not

Several mechanisms are routinely called “memory,” although they have different failure modes:

MechanismWhat it containsPersistencePrimary failure mode
Model contextInstructions, messages, tool results and retrieved evidence supplied to one model callOne call unless the application resends itOverflow, distraction or missing evidence
Thread stateMessages, task variables, tool outputs and checkpoints associated with a conversation or workflowUsually durable for the threadStale or inconsistent state after retries and concurrent writes
Long-term memorySelected facts, events, preferences, examples or procedures intended for reuseAcross sessionsIncorrect extraction, conflict, leakage or irrelevant retrieval
External knowledge baseDocuments and records maintained independently of the agentDefined by the source systemStaleness, access-control mismatch or weak retrieval
Parametric knowledgePatterns encoded in model weights during training or adaptationModel-version lifetimeUnknown provenance, outdated knowledge or hallucination

Only the first four are directly manageable by an application at runtime. Loading a stored fact into a prompt does not update the model's weights, and saying that an agent “learned” a preference usually means that an external record was written and later retrieved.

LangGraph's current documentation illustrates another important distinction: short-term memory can be durable thread-scoped state maintained through a checkpointer, even though only a selected portion is loaded into the model context. “Short term” describes scope and use, not necessarily volatile storage.

A taxonomy with independent axes

Human-memory terms—working, semantic, episodic and procedural—are useful analogies, and the CoALA framework uses them to organize language-agent architectures. They are not a universal storage schema. A production design should classify memory along independent axes rather than place every concept in one flat list.

Three independent classification bands show memory scope and lifetime, representation, and retrieval method
Scope, representation and retrieval answer different design questions. A user-scoped fact, for example, may be retrieved by an exact key, metadata filter or semantic similarity. Image generated by AI.

1. Scope and lifetime

Scope answers who or what the record belongs to, while lifetime answers how long it remains valid.

  • A turn-level observation may expire after the next action.
  • Thread state may last for one workflow and survive process restarts.
  • A user preference may persist across sessions but must not leak to another user.
  • Organization knowledge may be shared only with authorized members.
  • A task checkpoint may be durable even though it is irrelevant after task completion.

Scope must be enforced before similarity ranking. A highly similar record from another tenant is still an unauthorized record.

2. Representation

Representation describes what the record means:

  • Semantic facts represent propositions such as a preference, identity attribute or project decision.
  • Episodic records represent an event or interaction situated in time, often with participants and outcome.
  • Procedural knowledge represents how to perform a task: instructions, examples, workflows or policies.
  • Task state represents the current status of a process, including completed steps, pending actions and tool outputs.

These categories can overlap. A support interaction is an episode; its resolved account preference may become a semantic fact; the successful resolution path may later be curated into procedural guidance. The transformation between them requires an explicit policy.

3. Retrieval mechanism

Retrieval is not synonymous with vector search. Useful systems combine mechanisms according to the query:

MechanismStrongest useTypical weakness
Exact key or lookupIDs, account facts, canonical settingsRequires a known schema or key
Metadata filterTenant, user, project, source, statusDoes not rank relevance by itself
Keyword searchNames, error codes, quoted phrasesMisses paraphrases
Semantic similarityConceptual matches and paraphrasesCan return plausible but contextually wrong items
Temporal retrievalCurrent state, recency and “when” questionsRequires reliable time metadata and update semantics
Entity or graph traversalRelationships among people, projects and resourcesAdds extraction and consistency complexity
Task/dependency lookupWorkflow state and prerequisite stepsUsually application-specific

A retrieval policy can use strict authorization and metadata filters first, generate candidates through several signals, rerank them, and then assemble a context under a token budget. Mem0's current managed documentation, for example, describes semantic, keyword, entity and temporal signals; its open-source behavior depends on the configured vector store and optional reranker, and does not provide the same managed graph-memory behavior.

Memory has a write path and a read path

The common diagram “retrieve, prompt, answer, write a summary” is too universal. Some turns need no long-term read. Many should produce no memory write. Writes may occur synchronously, in a background job, after user confirmation or only at a workflow checkpoint.

The write path: formation, not accumulation

A defensible write path separates observation from commitment:

A governed write pipeline observes an interaction, proposes candidate memories, validates them, resolves duplicates or conflicts and commits a versioned record, with an optional background path
A memory write is a policy decision. Validation and conflict resolution can run in the request path or asynchronously; not every interaction should create a durable record. Image generated by AI.
  1. Observe. Receive messages, tool results, user edits or environment events without assuming they should be retained.
  2. Propose. Extract candidate facts, events, task transitions or reusable procedures.
  3. Validate. Check authorization, sensitivity, usefulness, source reliability, confidence and retention policy.
  4. Resolve. Compare candidates with existing records; deduplicate, supersede, merge, preserve as history or reject.
  5. Commit. Store a versioned record with provenance, temporal fields and an audit trail.

The timing is a product decision. LangGraph documents both hot-path writes, which make a new memory immediately available but add latency and task complexity, and background writes, which separate memory formation from the response path but introduce freshness and scheduling trade-offs.

A useful record schema makes those decisions inspectable:

memory-record.ts, ts
type MemoryRecord = {
  id: string;
  namespace: { tenantId: string; userId?: string; agentId?: string };
  kind: "fact" | "event" | "procedure" | "task-state";
  content: string;
  source: { type: "user" | "tool" | "system"; reference: string };
  observedAt: string;
  effectiveFrom?: string;
  expiresAt?: string;
  supersedes?: string;
  confidence?: number;
  sensitivity: "public" | "internal" | "personal" | "restricted";
};

This is an illustrative contract, not a universal standard. The important properties are explicit ownership, provenance, time, versioning and sensitivity—not these exact field names.

Updates require temporal semantics

“The user lives in Austin” and “the user moved to Seattle” need more than similarity search. The system must distinguish:

  • observation time: when the application learned the statement;
  • effective time: when the statement became true;
  • expiration: when the record should no longer be used;
  • historical validity: whether an old fact remains useful for past-tense questions;
  • supersession: which record replaces another for current-state queries.

Deleting the older record loses history; keeping both without temporal semantics can make the current answer ambiguous. The correct policy depends on the product: an itinerary, a medical record and a music preference should not share the same update rules.

Mem0's current default extraction path is additive: new facts do not silently rewrite older ones. Its documentation directs applications to use explicit update or delete operations when correction or removal is required. That behavior is one implementation choice, not a substitute for application-level temporal rules.

The read path: scoped evidence selection

The read path should first establish authorization and scope, then gather and rank evidence:

A query and scope feed exact, semantic, entity and temporal retrieval signals before reranking and context assembly, while write, retrieval and end-to-end quality are evaluated separately
Hybrid retrieval is only the upper half of the problem. Write quality, retrieval quality and downstream behavior require separate measurements. Image generated by AI.
  1. Derive the query, task state and temporal intent.
  2. Apply tenant, user, agent, consent and purpose filters.
  3. Generate candidates through the mechanisms appropriate to the question.
  4. Rerank using relevance, recency, reliability, diversity and conflict signals.
  5. Assemble a bounded context that preserves source and time metadata.
  6. Require the model to distinguish evidence, uncertainty and absence.

The context assembler must not convert a low-confidence extracted memory into an unquestioned system instruction. Records should retain their status and provenance so that the model—or application logic—can prefer authoritative sources, surface a conflict or abstain.

Retrieved does not mean true

A memory may faithfully preserve a user's earlier statement and still be outdated, malicious, incorrectly extracted or irrelevant to the present purpose. Retrieval establishes availability, not truth or authorization.

Storage is an implementation detail, not the architecture

Different access patterns justify different stores. A relational database may be the source of truth for facts and metadata; a vector index may accelerate semantic candidate generation; an event log may preserve history; a graph may support relationship queries; object storage may retain larger artifacts.

Mem0's documented architecture illustrates this separation. Its managed system uses SQL for facts and metadata, a vector database for embeddings and an entity store for entity-oriented retrieval. The application still chooses which results to include in the model prompt. Managed Mem0 and Mem0 OSS also differ in available retrieval behavior, so claims about one should not be generalized to the other.

mem0

mem0ai/mem0

A memory layer that extracts, stores and retrieves user-specific context for agents

The Memory Layer for AI Agents - Drop-in memory infrastructure for AI agents and apps. Context that persists. Built for production.

  • Python
  • agentic-memory
  • agentic-memory-system

69,188 stars8,230 forksSnapshot Nov 18, 2026

The correct question is not “Which memory database should we use?” It is “Which record types, consistency rules and queries must the system support?” The store follows from that contract.

Evaluation must isolate the failing stage

An end-to-end answer can fail because the right event was never written, because retrieval missed it, because stale evidence outranked the current fact, or because the model ignored correct evidence. One aggregate accuracy score cannot identify those causes.

Write quality

Measure whether the memory formation process creates the right records:

  • extraction precision and recall against annotated candidates;
  • duplicate and contradiction rate;
  • proportion of sensitive or disallowed content incorrectly retained;
  • provenance completeness;
  • latency and cost of synchronous or background formation.

Retrieval quality

Evaluate the evidence set before generation:

  • recall at a fixed context budget;
  • precision or nDCG for ranked candidates;
  • current-fact accuracy and stale-record rate;
  • scope-violation rate, which should be zero for protected boundaries;
  • evidence diversity, conflict detection and retrieval latency.

End-to-end quality

Measure what the agent does with memory:

  • task or answer accuracy;
  • citation or evidence attribution correctness;
  • temporal and multi-session reasoning;
  • correct handling of knowledge updates;
  • abstention when the required information is absent;
  • user-visible correction and deletion behavior.

LongMemEval, published at ICLR 2025, contains 500 questions testing information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention. Its authors report a 30% accuracy drop for evaluated commercial assistants and long-context models in sustained-interaction settings, and analyze memory as indexing, retrieval and reading stages. The result demonstrates difficulty under that benchmark; it is not a universal estimate for every production agent.

LoCoMo, published at ACL 2024, evaluates long-term conversations averaging 600 turns and 16,000 tokens over as many as 32 sessions. Its experiments found that long-context and retrieval-augmented approaches improved some results but still struggled with long-range temporal and causal relationships relative to human performance.

As of this revision, LongMemEval-V2 is an arXiv preprint, not a peer-reviewed conference result. It extends evaluation from conversational history to multimodal web-agent trajectories and explicitly measures both answer accuracy and query latency. Its framing is useful because operational memory must retrieve adequate evidence within a bounded time, but its reported leaderboard numbers should be treated as preliminary research results.

Safety, privacy and governance are part of memory quality

Persistent memory creates risks that a stateless prompt does not:

  • Cross-user or cross-tenant leakage. Missing namespace filters can expose another user's data.
  • Memory poisoning. Untrusted text can plant instructions or false facts that influence later sessions.
  • Sensitive-data retention. Raw transcripts may contain credentials, health information, identifiers or third-party data.
  • Purpose drift. Information collected for one workflow may be reused in another without a valid product or legal basis.
  • Deletion failure. Removing the source record may leave embeddings, caches, summaries, backups or derived memories intact.
  • Invisible inference. A model may extract a sensitive or incorrect attribute that the user never intended to persist.

Minimum controls include server-side authorization on every read and write, tenant-scoped indexes, field-level sensitivity labels, encryption, retention limits, audit logs, poisoning-resistant separation between data and instructions, and a user-visible correction and deletion path. Deletion should propagate to derived indexes and caches, not merely hide a row in a dashboard.

Mem0's current guidance explicitly recommends scoping searches by identifiers such as user_id, agent_id or run_id and avoiding secrets, raw credentials and unredacted sensitive data. These are necessary controls, not a complete privacy or security program.

A practical design sequence

Before selecting a memory framework, write down the contract:

  1. Enumerate decisions. Which future decisions improve when past information is available?
  2. Define record types. Separate facts, events, procedures and task state.
  3. Set authority. Identify who may create, confirm, update and delete each type.
  4. Model time. Define observation, effective, expiration and supersession semantics.
  5. Enforce scope. Make tenant and user boundaries structural, not optional query hints.
  6. Choose retrieval. Match exact, filtered, semantic, temporal or graph retrieval to each query class.
  7. Budget context. Specify how much evidence can enter a model call and how conflicts are represented.
  8. Evaluate stages. Test write, retrieval and end-to-end behavior independently.
  9. Design recovery. Support correction, deletion, re-indexing and audit before launch.

The engineering test

A memory system is trustworthy only when it can explain what was stored, why it was stored, who may retrieve it, which newer record changed its meaning, and how its contribution to an answer was evaluated.

Conclusion

Agent memory is best understood as governed state management around a model, not as a human-like faculty and not as a vector-search feature. Its design spans independent choices about scope, representation, write policy, temporal consistency, retrieval, context assembly, security and evaluation.

The most important architectural move is separation: stored memory from model context, current state from history, semantic content from retrieval technique, and retrieval success from downstream answer quality. Once those boundaries are explicit, storage technology becomes a replaceable implementation choice rather than the definition of memory itself.

References

  1. Sumers et al., “Cognitive Architectures for Language Agents,” Transactions on Machine Learning Research, 2024 — a modular framework for organizing language-agent memory, action and decision-making.
  2. Maharana et al., “Evaluating Very Long-Term Conversational Memory of LLM Agents,” ACL 2024 — the LoCoMo dataset and evaluation of long-range conversational memory.
  3. Wu et al., “LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory,” ICLR 2025 — a benchmark for extraction, multi-session and temporal reasoning, updates and abstention.
  4. Wu et al., “LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues,” arXiv preprint, 2026 — a current, non-peer-reviewed benchmark for memory over long web-agent trajectories.
  5. Mem0, “How Mem0 Works” and “Add Memory” — current implementation behavior, retrieval signals, store roles and managed-versus-OSS distinctions.
  6. LangGraph, “Memory overview” — current documentation for thread-scoped state, long-term stores and hot-path versus background writes.

Updated

Topics

Repositories mentioned

  • mem0

    A memory layer that extracts, stores and retrieves user-specific context for agents

    69,188 starsPython

Related reading

  • Understanding Context Engineering in AI Agents

    Why the context you assemble matters more than the prompt you write, and practical techniques to keep it relevant and within budget.

    3 min read

  • Survival Rules for Working with Claude Code

    An evidence-graded field guide to using Claude Code safely and reproducibly: constrain the task, protect the context window, verify every change and automate only inside explicit boundaries.

    10 min read

  • LLM NPC Dialogue: State, Latency, Safety and Cost

    An architecture for adding generated dialogue to a game without giving up deterministic state, responsive interaction or narrative control.

    4 min read

  • Building an MCP Server from Scratch

    A practical guide to building a small Model Context Protocol server in TypeScript, from the first tool to testing it in a real client.

    2 min read