MemUse tests whether agents actually use their memories
By AgentRiot Editorial
An 8,000-episode coding study finds little change in task success, but tracing individual memory deliveries exposes useful guidance, ignored information and costly repetition.

Adding memory barely moved the coding agent's success rate. What changed was harder to see: which instructions it followed, which relevant facts it ignored, and how much repeated text it received along the way.
That is the problem behind MemUse, introduced in a November 1 preprint (opens in new tab) by Changran Xu, Zeju Li and Qiang Xu at the Chinese University of Hong Kong. Across four memory configurations, resolve rates stayed within 0.4 percentage points of a no-memory baseline. Recorded episode costs rose by 8.8% to 293.2%.
The paper's contribution is an attribution framework, rather than a new memory store. It asks what a delivered memory added to the agent's existing context, whether the agent acted on it, and what followed. That distinction matters when deciding whether to improve retrieval, change delivery, or remove a repeatedly misleading entry.
Following a memory from input to consequence
MemUse starts at the model input. A memory bridge annotates storage, search and delivery operations; a pass-through recorder captures requests and responses at the model-API boundary. This catches memory inserted by a host or proxy even when it is absent from the agent's saved conversation history. Placement proofs connect the annotated memory text to its actual location in a request.
The evaluator reconstructs the text the agent received (opens in new tab), including whether an entry was complete, clipped or of unknown exposure. It does not silently substitute the full stored record for a truncated preview. The comparison context is everything else visible in the consuming input, with the target delivery removed. Earlier actions and plans remain inspectable even when they have left that visible context.
Identical repeat deliveries are grouped into an injection span only while an earlier copy remains visible. Restating the same text therefore adds delivery volume without earning another assessment. Changed content, or re-entry after visibility is lost, starts a new span.
Each entry also gets a credit window. It begins when the span is first consumed and can end when a later delivery of the same item takes over attribution. A fuller read can take credit from its preview; an uninformative copy of still-visible content does not. Removing text from context does not automatically end credit, because the agent can act on something it read earlier. Consequences of actions inside the window are followed through the episode's end.
An agentic judge then assesses relevance, accuracy, novelty, influence and observed effect. Relevant information can be wrong; correct information can be redundant; a mention can stop short of action. A preview that only prompts retrieval of a fuller record is marked selected, with the fuller delivery assessed separately. Those distinctions prevent retrieval clicks from being counted as completed useful work.
Why matching an action is insufficient
The paper's retry-loop example (opens in new tab) makes the attribution rule concrete. An entry says that fetch() needs a backoff delay between retries. The agent receives it at step 6 and adds the delay at step 7.
If the agent had already planned that edit at step 4, the entry is redundant and receives no action attribution. If the agent had only asked whether a delay was needed, the entry is confirming, but its influence remains undetermined: the record cannot separate the competing explanations. If the earlier record only showed the code, without that proposal, the memory supplies the specific instruction and the edit can be attributed to it.
This is a worked rubric example, not an additional experiment. It illustrates why neither a matching patch nor an agent's general ability to infer the fix settles attribution.
The judge must use read-only evidence tools and supply action anchors. Programmatic checks (opens in new tab) enforce required reads, valid identifiers and credit-window boundaries. Claiming no use requires reading the whole window; determining an acted-on entry's effect requires reading from the window's start through the episode's end. Search snippets do not count as complete reads.
Accuracy also has an explicit evidence limit. An uncontradicted claim remains unverified, rather than being accepted as true. An improved effect needs a recorded check or downstream use connecting the memory-shaped action to progress; a generic test pass is insufficient. An unknown effect stays unknown even when the overall task succeeds.
Fixed rules convert accepted assessments into span verdicts, with harm taking precedence over benefit. The framework preserves uncertainty instead of forcing every delivery into helpful or useless.
What the coding study actually tested
The authors ran all 500 tasks from 12 Python repositories in SWE-bench Verified. The benchmark's official description (opens in new tab) identifies it as a human-filtered subset of SWE-bench, reviewed for clear descriptions, correct test patches and solvability.
The study setup (opens in new tab) used mini-SWE-agent with gpt-6-luna. Every configuration followed the same task sequence four times: 8,000 memory-equipped episodes plus 2,000 no-memory episodes. Within repositories, tasks followed pull-request creation order; repositories were interleaved. Writable stores started empty for each run and retained information across tasks.
The order was deliberately constructed to create transfer opportunities, placing source tasks before related tasks in other repositories for 55 of 66 identified pairs. It was not a random task shuffle or a natural production workload.
MemGovern provided a fixed external corpus of experience cards, searched and opened by the agent. Mem0 inserted repository-scoped task recall and allowed further memory operations. Its local-plus-global configuration, L+G, also permitted explicitly requested cross-repository scope, but the agent never requested it. TencentDB Agent Memory used a shared store and inserted persona, scene-index and skill-catalog information through its proxy, with fuller content available on request.
These are comparisons of memory configurations attached to one coding agent, not a ranking of different language models or proof about memory across all workloads.
The outcome table (opens in new tab) reports means and sample standard deviations across four runs. Differences use unrounded rates, explaining why they do not always equal subtraction of the displayed means.
| Configuration | Resolved, mean ± SD | Difference vs baseline, percentage points [95% interval] | Recorded USD/episode | Cost increase |
|---|---|---|---|---|
| MemGovern | 73.4% ± 0.6 | +0.40 [-0.70, 1.50] | $0.0049 | 8.8% |
| Mem0 L+G | 72.7% ± 0.7 | -0.25 [-1.55, 1.00] | $0.0062 | 37.7% |
| Mem0 L | 72.8% ± 0.6 | -0.20 [-1.50, 1.05] | $0.0062 | 38.1% |
| TencentDB | 72.7% ± 0.8 | -0.30 [-1.50, 0.85] | $0.0178 | 293.2% |
| No memory | 73.0% ± 1.5 | Reference | $0.0045 | Reference |
Every confidence interval includes zero. This study does not establish a resolve-rate improvement for any configuration; it also does not establish exact equivalence. Baseline runs ranged from 71.8% to 75.0%, wider than the differences between configuration means.
Where delivered information stopped becoming useful
The memory profiles (opens in new tab) show the bottleneck before final task outcomes. Depending on configuration, 71.5% to 98.6% of entries were irrelevant on average within a profiled span. These are equally weighted span averages, not percentages pooled over every delivered entry.
A separate entry-level breakdown (opens in new tab) finds that most relevant entries went unused, meaning no attributable action, selection or reference within their windows.
| Configuration | Mean relevant share per span | Unused among relevant entries | Mean acted-on share per span | Episodes containing benefit |
|---|---|---|---|---|
| MemGovern | 15.08% | 51.32% | 3.95% | 4.00% |
| Mem0 L+G | 1.42% | 79.68% | 0.40% | 3.65% |
| Mem0 L | 1.43% | 77.06% | 0.51% | 4.50% |
| TencentDB | 28.48% | 75.06% | 4.08% | 7.05% |
The columns deliberately use different denominators. Relevant and acted-on shares average within spans; unused rates condition on relevant assessed entries; benefit coverage divides by all 2,000 episodes per configuration.
Non-use was not simply duplication. Among unused relevant entries, roughly half were novel in Mem0 and MemGovern, and 89.52% were novel in TencentDB. TencentDB's broad relevance coverage also depended heavily on recurring persona content: excluding recurring items reduced episodes with a relevant entry from 99.8% to 14.3%.
Across 714 acted-on entries, the judge assessed 421 as beneficial, 16 as harmful and 64 as having no effect. The remaining 213, or 29.8%, had undetermined effects. More than half of first attributed actions occurred after the first consuming step, so evaluating only the immediate response would miss substantial recorded use.
Helpful work can still end in failure
In one reported TencentDB example (opens in new tab), guidance about a different Napoleon bug sent the agent into the wrong Sphinx component. A reproduction ruled that explanation out; the agent later fixed sphinx/util/docfields.py and passed the task. The final success concealed a memory-directed detour.
The paired MemGovern example runs the other way. A card guided a subset handler that changed the reported check from None to True, but an equality error remained unfixed and the task failed. That failure did not erase the local improvement.
Among episodes with completed harness results, 89 of 383 containing beneficial use failed, or 23.2%. Five of 16 containing harmful use succeeded, or 31.3%. These are overlapping possible categories: an episode can contain both benefit and harm. Three memory episodes lacked completed harness results and were excluded from this outcome analysis, though counted as non-resolved in resolve rates.
This also changes what is worth saving. The memory-history analysis (opens in new tab) traced 73 of 331 beneficial entries with known source-task outcomes to failed earlier episodes. Rejecting every memory from an unsuccessful run would discard some guidance that later helped.
Cost depends on what keeps getting delivered
The delivery-form breakdown (opens in new tab) suggests a narrower engineering target than replacing an entire memory backend.
| Delivery form | Share of its system's delivered tokens | Share of its system's beneficial entries |
|---|---|---|
| MemGovern full cards | 12.48% | 96.43% |
| Mem0 task recall, both configurations combined | 12.96% | 75.77% |
| TencentDB full skills | 1.99% | 86.01% |
Previews and indexes still served a purpose: they could direct selection of fuller material. But repeated deliveries accounted for about 78.2% of Mem0 deliveries and 88.4% of TencentDB deliveries. Copies of still-visible entries consumed 68.1% of Mem0's delivered tokens.
The dollar figures need their accounting limits (opens in new tab). They cover retained executions' recorded agent and memory-system usage at saved model rates, excluding the MemUse judge. Discarded attempts are omitted, and failed requests without reported usage contribute zero. Unreported service work and hosting are outside the accounting. These are not complete production bills or current vendor tariffs. Percentage increases use unrounded costs.
Delivered-text volume is a separate estimate based on placement characters divided by four, including wrappers and repetitions, rather than exact tokenizer counts. Judging itself required 77,329 model calls and 31.5 hours (opens in new tab) across the recorded evaluation segments. The small episode prices should not be mistaken for the price of running this audit.
What the evidence cannot settle
MemUse can operate without reference answers, but this study used gold patches. DeepSeek-V4.1-Flash-1017 received each reference repair, not the harness outcome. Its relevance and accuracy assessments were therefore reference-assisted; the experiment does not validate reference-free performance.
The validation appendix (opens in new tab) explicitly says agreement with human judgments was not measured. Read-coverage checks establish access to evidence, not semantic correctness. Notes and prose-only citations are not fully enforced by the structural checks. Repairing incomplete reads is useful discipline, but cannot substitute for human calibration.
MemGovern has another material limitation: retrieval-time benchmark overlap (opens in new tab). Delivered cards included guidance from benchmark problems' own discussions and repair attempts. Three identified cards contributed seven of its 84 beneficial assessments. Other overlaps included a repair matching the gold fix and superseded guidance that repeatedly harmed a task. The targeted audit did not measure prevalence across the full corpus, so its higher token efficiency cannot be read as clean evidence of generalization to unseen problems.
Nor are the configurations perfectly interchangeable. TencentDB used Chat Completions while the others used Responses, and its extraction timers were shortened: the idle trigger fell from 600 to 60 seconds, second-stage delay from 90 to 10, and minimum second-stage interval from 900 to 60. The fixed sequence shaped evolving memory histories; task-bootstrap intervals did not model dependencies between those histories. Above all, tracing a recorded action is observational attribution, not a counterfactual demonstration that removing memory would change success.
For builders, the useful next experiment is specific: record actual inputs, identify repeated resident text, and test whether slimmer, task-timed delivery preserves useful actions at lower cost. Retain the outcome baseline and inspect harmful items individually. LongMemEval's conversation-memory benchmark (opens in new tab) tests abilities such as knowledge updates and temporal reasoning; MemUse adds a different question for acting agents. Once a memory arrives, does it change an executed action in a way worth paying for?

