Does memory help coding agents work faster?
Summary
Coding agents start each session without memory of earlier ones. They repeat failed approaches and look up answers they already have. Published work on agent memory reports gains on benchmark tasks, and some reports show lower latency and fewer tokens when retrieval stays selective. Our own measurements do not yet show a speed-up. The experiment that would settle it is set out below.
What the published work reports
Figures are as reported in each paper's abstract or secondary summaries. We are checking them against the source tables.
| System | Memory form | Reported result |
|---|---|---|
| Reflexion (2023) | Verbal reflections on failed episodes | Better success on coding and decision tasks without weight updates |
| Voyager (2023) | Library of reusable, verified skills | Faster learning in Minecraft than a baseline starting from scratch |
| ExpeL (AAAI 2024) | Experience pool and extracted insights | HotpotQA: 39% success against 28% for ReAct. Insights alone (36%) beat past trajectories alone (31%) |
| Mem0 (2025) | Extracted facts with add, update and delete | LOCOMO: about 26% relative quality gain over OpenAI's memory; about 91% lower p95 latency and over 90% lower token cost than full-context prompting |
| Zep / Graphiti (2025) | Temporal knowledge graph | LongMemEval: up to 18.5% accuracy gain and about 90% lower latency |
Three findings carry over to coding work. Distilled lessons help more than replaying past runs. Retrieval must stay small, since the speed and cost gains come from fetching a few facts rather than the whole history. Facts go stale, so a memory needs a way to supersede them.
These are question-answering and simulation benchmarks, not repository tasks with tests. They show the mechanism works; they do not predict gains in coding.
What we have measured in Commonshell Pilot
Our harness compared the same scenario before and after a rework of research reuse:
| Run | Requests | Prompts | Requests per prompt | Context read | Lookups skipped |
|---|---|---|---|---|---|
| Before | 47 | 15 | 2.7 | 0.93M tokens | 0 |
| After | 45 | 16 | 2.8 | 0.98M tokens | 0 |
The two runs are the same within noise, and neither exercised the reuse path. Earlier end-to-end cases showed the intended behaviour: a proven research answer caused zero repeat lookups and cited its date and source. That is evidence the mechanism works, not that it saves time. We do not yet measure failed attempts avoided, which matters most.
The experiment we are running
The claim to test is: with recalled learnings, a coding agent completes the same tasks with fewer requests, fewer tokens and fewer repeated failures.
- Tasks. 20 to 30 repository tasks, each with a test that defines success, including held-out tasks from repositories the learnings never saw.
- Design. Each task runs with memory off and on, in random order, from the same commit, with the same model and settings.
- Outcomes. Task success first; then repeated failures; then requests and tokens per completed task; then time to completion.
- Reporting. Paired differences with confidence intervals. No speed-up is claimed unless the interval excludes zero. Every task is reported, including those where memory did not help.
We will publish the results here, including if they are negative.
Limitations
- Some published figures come from secondary summaries and have not been re-checked.
- Our baseline is one user's usage, not a population sample.
- Hook-based signals for reuse and grading can misclassify a turn.
References
- Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning.
- Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., Anandkumar, A. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models.
- Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., Huang, G. (2024). ExpeL: LLM Agents Are Experiential Learners. AAAI 2024. arXiv:2308.10144.
- Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., Bernstein, M. S. (2023). Generative Agents: Interactive Simulacra of Human Behavior.
- Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560.
- Chhikara, P., Khant, D., Aryan, S., Singh, T., Yadav, D. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413.
- Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., Chalef, D. (2025). Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956.