Use retrieval to find relevant information, a context window to supply information for the current model call, and persistent memory when selected information should remain available across interactions. They solve related but different problems. An agent may need all three, but none removes the need to decide what information is permitted, current, and useful for the task.
This is an architectural decision framework, not a benchmark of current models. It uses an invented support-assistant example and makes no claim that a particular token limit or retrieval product is best.
Separate the three jobs
“Long context” usually refers to a model’s capacity to accept a relatively large input. Retrieval selects relevant material from a larger source. Persistent memory stores information that can be used again after the current interaction.
| Approach | Main question | Example | What it does not solve automatically |
|---|---|---|---|
| Larger context | What information should this call receive? | Include a short policy document and the current conversation | Permissions, document freshness, or reliable use of every supplied detail |
| Retrieval-augmented generation | Which source material is relevant now? | Find the current cancellation policy for a product | Whether the selected source is authorized or the answer follows it |
| Persistent agent memory | What should remain available later? | Remember an explicitly approved communication preference | Consent, correction, deletion, or the validity of an inferred preference |
The original retrieval-augmented generation research combines generation with access to retrieved information. Modern systems use many retrieval designs; that paper is background, not evidence for a universal vector-database requirement.
Work backward from one support task
Imagine an assistant asked whether a customer can cancel a subscription at the end of the current term. It needs the applicable policy, the customer’s current subscription state, and the actual question.
Start by determining which sources are authoritative. A policy document describes the rule; a billing system may supply current account state. A previous conversation may describe what someone believed, but it should not silently replace either source.
If the policy set is small and stable, supplying the relevant documents directly may be a reasonable first design to test. If the corpus is larger or varies by product and region, retrieval may help select the material. In both cases, access controls must determine which information the assistant may use.
There may be no reason to store a new memory after this request. “The customer asked about cancellation” is not equivalent to “the customer wants to cancel,” and neither necessarily belongs in a durable preference store.
Test a context design rather than assuming bigger is better
Measure whether the answer uses the relevant evidence, not just whether all documents fit in a request. The Lost in the Middle study demonstrates sensitivity to information position in the models and tasks it evaluated. It does not establish how every current model behaves.
A useful test set can include conflicting policy versions, irrelevant but plausible documents, missing account state, and a correct answer that requires admitting uncertainty. Change one aspect of the context assembly and compare the resulting decisions using the same criteria.
Record which source passages were supplied. If an answer is wrong, distinguish a failure to select the right document from a failure to use a document that was present. Those are different engineering problems.
Make memory a product decision
For every proposed memory, define the source, scope, purpose, and rules for updating or removing it. Decide whether the user can inspect and correct the information. Do not make an inferred statement more durable merely because it is easy to extract.
Separate temporary conversation summaries from durable user preferences and business records. They have different lifetimes and authority. A compressed conversation note should not become an unreviewed account-action record.
Anthropic’s context-engineering discussion describes the broader task of curating information available to an agent. The practical implication for this design is to manage context deliberately, including material supplied through tools or remembered state.
Compare the operating burden
A direct-context design needs document assembly, input limits, and updates. Retrieval adds indexing or search behavior, selection quality, access filtering, and source freshness. Persistent memory adds lifecycle questions, including how information is created, revised, and deleted.
Estimate those responsibilities for your application before comparing provider feature lists. Include the work of debugging a bad answer and explaining which information influenced it. This guide provides no current price estimates or assumed savings.
The next step is a small AI feature evaluation using the task you actually intend to support. Select the simplest design that meets the tested need, and add retrieval or memory when a specific requirement justifies the additional responsibility.