Which approach wins depends on what problem you are actually solving. Fine-tuning and RAG do not compete on the same axis, and conflating them leads to expensive mistakes that take months to surface.
The Fundamental Difference
Fine-tuning modifies the model's weights. You train the model on examples and it changes how the model reasons on inputs it has never seen.
The knowledge gets encoded in the weights, not in any external store. Think of it as changing how someone thinks rather than handing them a reference card.
RAG does not touch the model at all. You keep the model frozen and retrieve relevant context at inference time, so the knowledge lives in your vector store or document index and the model sees it only when you pass it in.
That reference card stays in your hand, and you can swap it out between one question and the next.
For agent memory, this distinction matters more than in most applications. An agent operating in memory needs to retain three things simultaneously: factual knowledge about the domain, procedural knowledge about how to handle tasks, and episodic knowledge about what happened in specific sessions.
Each of these has different update patterns and different failure costs. The practical map of episodic, semantic, and working memory in agents is useful background here, because the layer boundaries determine which memory problems are solvable with RAG and which require changing the model's weights.
Factual knowledge about your product changes constantly: a pricing update, a new API endpoint, a renamed plan tier. Fine-tuning on those facts means retraining every time one of them moves.
RAG means your index updates and the agent sees the new pricing on the very next query.
Procedural knowledge about how to handle a task stays far more stable. When you want the agent to follow a specific reasoning pattern or run a refund through legal review before it confirms anything, that is a behavior you want baked into the model, and that is where fine-tuning earns its cost.
Hardest of the three is episodic knowledge. What happened in this conversation, what preferences the user has expressed across sessions, what came of that troubleshooting step three weeks ago.
None of it is knowledge you would fine-tune on even if you could, because it belongs to one user or one interaction, not to the domain.
When RAG Is the Obvious Choice
The clearest signal that RAG is right: your knowledge base changes faster than you can retrain.
A customer support agent that needs current product features, last night's outage status, and today's pricing is a RAG problem, full stop. Retraining every time any of those move is not feasible.
RAG can expose an updated index on the next query. Fine-tuning requires another training and deployment cycle before the model reflects the change.
Standing up retrieval is not free either. You need an embedding model, a vector index, a retrieval pipeline, and reranking logic that keeps the relevant chunk at the top.
Compare retrieval overhead with the tool-call and inference traces from the target workflow. Define retrieval recall and evidence-use checks before building the pipeline so the evaluation measures the failure the product needs to prevent.
Verifiable sources make the second case for RAG. When an agent cites a refund policy, it should cite the actual clause, with a link a support lead can click.
RAG makes that possible because the retrieved context sits right there in the prompt. Fine-tuning buries the same knowledge in the weights, and you cannot point to where any given sentence came from.
For a bank or an insurer, citing the source document is not a nice-to-have, it is the thing a compliance review will ask for first.
RAG versus memory now treats agent-query asymmetry as part of the retrieval boundary. If the expected records do not reach the prompt, the model cannot ground the answer in them.
When Fine-Tuning Earns Its Cost
Fine-tuning has a narrower case than the vendors selling it would have you believe.
Reach for it when the model's behavior is wrong, not when information is missing. A base model that ignores your output format, misreads your domain, or fumbles your edge cases needs to be retrained on examples of the correct behavior.
Adding more documents to a prompt cannot fix how a model defaults to reasoning, because that lives in the weights, not the context.
For example, an agent may call a charge endpoint before creating the customer record. That is a behavior problem, so training examples of the correct sequence may help where retrieving another document would not.
Fine-tuning cost varies by provider, model, and dataset, and changing facts force another training run. A retrieval index can be updated without retraining the model.
There is a subtler cost. Fine-tuning on a narrow dataset risks catastrophic forgetting, where the model sheds capabilities from its pretraining because the fine-tuning set was too thin or the run too long.
The Memory Hierarchy Changes the Trade-Off
Looking at agent memory through the memory hierarchy lens, the fine-tuning versus RAG decision maps cleanly onto different layers.
Working memory is episodic and short-lived, and it is never a fine-tuning problem. No one retrains a model on what a user said two turns ago.
A conversation window or session store handles it, and the context window itself becomes your working memory, managed with retrieval and eviction rather than weights.
Semantic memory holds the stable domain knowledge, and that is where fine-tuning has its most legitimate use. The model's grasp of your domain, the reasoning steps it applies, the format it produces.
When any of those are wrong at the level of how the model thinks, fine-tuning is the fix.
Semantic memory also carries facts that move, though. Your product gains a feature, your pricing changes its structure, a plan gets renamed.
Those facts belong in RAG, not in the weights, because they turn over too fast to bake in.
The practical consequence is that you almost never pick one or the other for the whole memory system. You need both.
The real question is which one carries the bulk of the memory load, and the answer is almost always RAG, with fine-tuning added selectively for the reasoning patterns retrieval cannot teach.
What I Actually See in Production
A practical default is RAG for changing knowledge, with fine-tuning reserved for narrow behavior calibration.
The retrieval pipeline carries current product information, the user's stored preferences, and session history pulled from a vector store. When the agent needs to know something, it looks it up.
Fine-tuning, in those same systems, carries behavior. The model has been trained to run a refund through a specific escalation protocol, to format every response as the structured object the frontend expects, to check inventory before it promises a ship date.
Knowledge about how to operate is stable, and it has no business living in a retrieval index.
Adding more documents to RAG will not reliably correct behavior learned in model weights.
The agent does read them, sometimes, and still misbehaves, because the broken behavior sits in the weights and no amount of context reaches it. Imagine handing a driver who keeps turning the wrong way down one-way streets a thicker map.
The map was never the problem.
Rarer but worse is the reverse: teams that fine-tune on domain facts and then cannot change a price without a retraining run. The agent turns brittle and expensive to keep current.
The Decision Framework
Here is how I think through it now.
Is the knowledge changing frequently? Yes means RAG.
No means fine-tuning is worth considering.
Is the problem behavioral (how the model reasons) or factual (what the model knows)? Behavioral means fine-tuning.
Factual means RAG.
Do you need to cite sources? Yes means RAG.
The model weights do not give you provenance.
What is your retraining budget? If you cannot afford to retrain monthly, RAG is probably the foundation.
If retrieval latency breaks the service-level objective, test a different retrieval path or architecture rather than assuming a universal overhead.
Compare retrieval time with the actual tool-call and inference traces from the target workflow.
For most agent memory use cases in 2026, the honest answer is RAG as the foundation, with fine-tuning reserved for the reasoning patterns that refuse to stick through prompt engineering. Anthropic's contextual retrieval is worth reviewing before you finalize the architecture, because contextual embeddings lift retrieval accuracy enough to change whether you need fine-tuning for some reasoning tasks at all.
Build the retrieval pipeline first, then add fine-tuning once you have real behavioral failures that retrieval cannot reach.
Agent memory adds one constraint the general case does not: episodic and working memory are retrieval-and-state problems, not training-data problems. That pushes RAG and explicit state management to the primary layer even in systems where fine-tuning earns a real role.