I keep seeing the same pattern in agent architecture discussions. Someone builds a retrieval-augmented generation pipeline, calls it agent memory, and then wonders why their agent still loses track of what happened three turns ago. The gap between "RAG works" and "agent memory works" is real, and it is not a implementation detail. It is a fundamental architectural mismatch.
RAG solves retrieval. Retrieval is one piece of what an agent needs to remember.
The three-layer problem
Agent memory is not one system. It is three distinct systems that happen to share a name.
Working memory is what the agent is actively reasoning about right now. This lives in the context window. It is fast, ephemeral, and subject to eviction as soon as the context fills. Most agents lose working memory entirely on every new session.
Episodic memory is what the agent learned in past interactions that is still relevant. "The user prefers concise responses." "This API endpoint requires an API key in the header." These are not document facts. These are observations the agent made and should retain. RAG does not handle episodic memory well because episodic facts are not embedded documents. They are attributed fragments of experience.
Semantic memory is the world knowledge the language model was trained on, plus any persistent facts added through retrieval. This is where RAG actually works. You query a vector database, you get relevant documents, you inject them into context. The problem is that semantic memory is the easiest layer to build and the least interesting one for agents.
I wrote about how these layers interact in my post on memory-hierarchy-in-ai-systems. The short version is that most agents over-invest in semantic memory (RAG) and under-invest in the other two layers.
Why RAG breaks for episodic recall
The core issue with using RAG for episodic memory is that episodic facts do not exist as documents. They exist as attributed observations inside conversations or inside tool execution logs. The embedding and retrieval model that works for finding relevant paragraphs in a technical manual fails completely when asked "what did the user tell me about their preferences in the last session?"
The embedding model has no concept of attribution. It retrieves based on semantic similarity. If you ask "what are the user's preferences" and the last conversation contained "I prefer short responses and JSON format," the retrieval query and the stored content have low semantic overlap even though they are directly related. This is the asymmetry problem I documented in asymmetric-retrieval-agent-memory. RAG retrieval is optimized for query-document relevance. Agent recall is optimized for attribution-document relevance. These are not the same optimization target.
A secondary issue is that episodic facts are time-sensitive in ways that document facts are not. "The user prefers concise responses" might be true now but was not true three months ago. Vector retrieval has no native concept of recency bias or temporal decay. You can layer recency onto RAG, but it is a retrofit, not a feature.
The cross-session persistence gap
RAG pipelines are typically session-scoped. You query the vector store, you get results, the session ends, the results evaporate. For agents that need to remember across sessions, this is a fundamental limitation. The retrieval system does not know what happened in previous sessions unless you explicitly index session artifacts, and even then, the indexing strategy matters enormously.
I tried several approaches to cross-session persistence. The simplest one that actually worked was a structured observation log. Every time the agent made an inference about the user or the environment, it wrote a short attributed fact to a separate store. Not a vector store. A key-value store with a timestamp and attribution source. The retrieval was a direct lookup by semantic category, not a similarity search.
This is architecturally different from RAG. RAG is retrieval by content similarity. What I needed was retrieval by semantic type. "Give me all facts about this user's communication preferences." The vector store could not answer that efficiently because the query "communication preferences" has low similarity to "the user likes short responses."
Fine-tuning versus retrieval for memory
The other approach people reach for is fine-tuning. If the agent needs to remember facts, maybe you fine-tune the model to encode those facts. I explored this in rag-vs-fine-tuning and the conclusion is not clean.
Fine-tuning works for injecting broad behavioral patterns. You can fine-tune a model to be more concise, more formal, more aligned with a specific writing style. Those are general capabilities. Fine-tuning does not work well for injecting specific attributed facts that change frequently. The cost of re-fine-tuning every time a user preference changes makes it impractical for episodic memory. Fine-tuning is also opaque. When the agent acts on a fine-tuned behavior, you cannot audit which training example caused the behavior or when it was last updated.
Retrieval (RAG) works the opposite way. It is transparent, auditable, and cheap to update. You add a document, the next retrieval query finds it. But it cannot capture episodic facts that are not embedded in documents. The two approaches solve different problems. Using RAG for episodic memory is the category error.
What actually works
The architecture that works for agent memory has three components that are rarely built as one system.
The first is a working context manager. This is not retrieval. This is active context maintenance. What facts from the retrieval layer are currently relevant? What observations from this session should survive the next context window reset? I find myself rebuilding working context from scratch on every major interaction because the context window is always the bottleneck.
The second is an episodic store. This is not a vector store. It is a structured log of attributed observations. The schema matters more than the retrieval algorithm. If you cannot answer "what does the agent believe about the user's goals right now?" with a direct lookup, your episodic store is not structured correctly.
The third is semantic retrieval (RAG proper). This handles world knowledge, documentation, code snippets, and any information that is document-shaped. This is where vector search and embeddings earn their place.
Most agents I have reviewed have the third layer and not the first two. That is why they forget, hallucinate past facts, and require users to re-explain basic context in every new session.
The practical starting point
If you are building agent memory today, start with the episodic store. You do not need a vector database for this. You need a clean schema for observations and a way to retrieve them by semantic type rather than by embedding similarity.
A simple observation schema looks like this: subject, predicate, attribution, timestamp. "User.prefers = concise_responses, source = 2026-04-29_session_3, confidence = high." The retrieval is a filter over subject and predicate, not a similarity search over embeddings.
Once the episodic store exists and is retrieving correctly, the semantic layer (RAG) handles everything that fits the document retrieval model. The working context manager keeps the most recent observations available in the active context window.
Separating these three concerns avoids the trap of overloading RAG to do work it was not designed for. RAG is a powerful tool. It is just not a complete memory solution for agents.