I ran into this decision three times in the past year. Each time the team had already committed to a particular approach before I got there. The first team had fine-tuned a 7B model on their internal documentation and spent months trying to figure out why the agent still hallucinated. The second team had built an elaborate RAG pipeline and could not understand why the agent could not follow their custom reasoning patterns. The third team got it right by accident.
The right answer depends on what problem you are actually solving. Fine-tuning and RAG do not compete on the same axis. Conflating them leads to expensive mistakes.
The Fundamental Difference
Fine-tuning modifies the model's weights. You train the model on examples and it changes how the model reasoning behaves on unseen inputs. The knowledge is encoded in the weights, not in any external store.
RAG does not touch the model. You keep the model fixed and retrieve relevant context at inference time. The knowledge lives in your vector store or document index, and the model sees it only when you pass it in.
For agent memory, this distinction matters more than in most applications. An agent operating in memory needs to retain three things simultaneously: factual knowledge about the domain, procedural knowledge about how to handle tasks, and episodic knowledge about what happened in specific sessions. Each of these has different update patterns and different failure costs.
Factual knowledge about your product changes constantly. A pricing update, a new API endpoint, a changed workflow. Fine-tuning on this factual knowledge means retraining every time something changes. RAG means your index updates and the agent sees the new information next query.
Procedural knowledge about how to handle things is more stable. If you want the agent to follow a specific reasoning pattern or escalation flow, that is a behavior you want baked into the model. That is where fine-tuning earns its cost.
Episodic knowledge is the hardest. What happened in this conversation, what preferences has the user expressed across sessions, what was the outcome of that troubleshooting step three weeks ago. This is not knowledge you would fine-tune on even if you could, because it is specific to this user or this interaction.
When RAG Is the Obvious Choice
The clearest signal that RAG is right: your knowledge base changes faster than you can retrain.
A customer support agent that needs to know about current product features, recent outages, and today's pricing is a RAG problem. Retraining every time any of those change is not feasible. The latency between a product update and the agent knowing about it matters. With RAG, that latency is zero. With fine-tuning, you are looking at hours at minimum and probably days.
The retrieval infrastructure cost is real. You need an embedding model, a vector index, a retrieval pipeline, and reranking logic that keeps relevant results at the top. I have benchmarked this. Retrieval latency adds 40-120ms on top of inference for a typical setup. If your agent is latency-sensitive, that matters. If your agent is already doing multiple tool calls that take seconds, it probably does not.
The other case for RAG: you need verifiable sources. When an agent cites policy, it should cite the actual document. RAG makes this possible because the retrieved context is in the prompt. Fine-tuning makes the knowledge implicit in the model's weights, and you cannot point to where it came from. For regulated industries, this is not a nice-to-have.
I wrote about the retrieval accuracy problem in my post on asymmetric retrieval in agent memory. The short version: retrieval is harder than it looks, and bad retrieval will undermine your RAG pipeline faster than bad fine-tuning undermines your model.
When Fine-Tuning Earns Its Cost
The case for fine-tuning is narrower than the vendors making money from it would suggest.
You need fine-tuning when the model behavior itself is wrong, not when the information is missing. If your base model does not follow your output format, does not reason about your domain correctly, or does not handle your edge cases, then retraining the model on examples of correct behavior is the solution. This is a weights problem, not a context problem.
A practical example. We had an agent that consistently failed on multi-step API workflows. The model did not know the right sequence of operations, not because it lacked information, but because its prior training had taught it the wrong default sequence. Fine-tuning on 200 examples of correct workflows fixed this. RAG would not have helped because the information about correct sequences was not in any document we could retrieve.
The cost is the other factor. A full fine-tuning run on a 7B model costs between $100-500 depending on the provider and the dataset size. That is per run. If your domain knowledge changes monthly, you are doing this monthly. Compare that to hosting a vector index on a small instance for $20/month.
The other cost is subtler. Fine-tuning a model on a small dataset risks catastrophic forgetting. The model can lose capabilities from its pretraining if the fine-tuning set is too narrow or the training run is too long. I saw this happen. A team fine-tuned a model on their documentation and the agent lost its ability to do basic arithmetic. The documentation did not include arithmetic examples, and the training run was too aggressive. Guardrail tests caught it, but barely.
The Memory Hierarchy Changes the Calculus
When you look at agent memory through the memory hierarchy lens, the fine-tuning versus RAG decision maps to different layers.
Working memory is episodic and short-lived. This is never a fine-tuning problem. No one fine-tunes a model on what happened in this conversation. RAG with a conversation window or session store handles this. The context window is your working memory, and you manage it with retrieval and eviction, not with weights.
Semantic memory is the stable domain knowledge. This is where fine-tuning has the most legitimate use case. The model's understanding of your domain, the reasoning patterns it applies, the formats it produces. If this is wrong, you fix it with fine-tuning.
But semantic memory also includes things that change. Your product has features. Your pricing works a certain way. These are semantic facts that belong in RAG, not in weights, because they change too often.
The practical consequence: you almost never choose one or the other for your entire memory system. You need both. The question is which one handles the majority of your memory load, and that is almost always RAG, with fine-tuning added selectively for reasoning patterns that RAG cannot teach.
What I Actually See in Production
The systems I have seen work well have RAG as the primary memory layer and fine-tuning doing specific calibration.
The RAG pipeline handles current product information, user-specific preferences stored in a profile, and session history retrieved from a vector store. When the agent needs to know something, it retrieves it.
The fine-tuning handles behavior. The model has been trained to follow a specific escalation protocol, to format outputs in a particular way, to apply certain reasoning steps before taking action. This is stable knowledge about how to operate, and it does not belong in a retrieval index.
The failure mode I see most is teams trying to solve behavioral problems with RAG. They keep adding more documents to the index hoping the agent will read them and behave correctly. The agent reads them, sometimes, and still does not behave correctly, because the behavioral problem is in the weights, not in the context.
The reverse failure is rarer but worse. Teams that fine-tune on domain facts and then cannot update those facts without retraining. The agent becomes brittle and expensive to maintain.
The Decision Framework
Here is how I think through it now.
Is the knowledge changing frequently? Yes means RAG. No means fine-tuning is worth considering.
Is the problem behavioral (how the model reasons) or factual (what the model knows)? Behavioral means fine-tuning. Factual means RAG.
Do you need to cite sources? Yes means RAG. The model weights do not give you provenance.
What is your retraining budget? If you cannot afford to retrain monthly, RAG is probably the foundation.
How critical is retrieval latency? If every millisecond matters and you cannot absorb 60ms of retrieval overhead, you need a different architecture. Most agentic workflows have tool call latencies of hundreds of milliseconds, so retrieval overhead is rarely the bottleneck.
The honest answer for most agent memory use cases in 2026: RAG as the foundation, fine-tuning only for the reasoning patterns that do not stick through prompt engineering. Build the retrieval pipeline first. Add fine-tuning when you have real behavioral failures that RAG cannot fix.
I covered the retrieval fundamentals in RAG vs fine-tuning from a more general perspective. The agent memory context adds the constraint that episodic and working memory are non-negotiable RAG problems, which shifts the calculus toward RAG as the primary layer even for systems where fine-tuning has a legitimate role.