When memory goes stale: benchmarking mex against nine tools

Memory that is wrong is worse than no memory. An agent will act on an outdated fact with the same confidence as a true one, and nothing in the fact tells it the code has moved on. This report starts there: what each tool does with project knowledge after the code it describes has changed. It then measures the code graph underneath, which is what lets mex notice those changes at all.

We ran mex and nine other free, local tools on the same machine, against the same repositories and tasks, and scored every tool against answer keys that none of the tools produced. Project knowledge was tested by replaying 367 commits of the Hono web framework and checking, after the code had moved, which tools still served outdated facts as current. The code graph was tested on 848 real GitHub issues from SWE-Explore, on 11 repositories in 5 languages, and against 200 real commits.

Key findings

• Stale knowledge. Memory tools with no link to code served 90 to 100% of outdated facts as current. mex served 20.0%.
• The cost of warning. mex flags a fact when its linked code changes, including facts that are still true. Plain mex check returned only 56.9% of still-true facts cleanly, against 85 to 96% for tools that never warn. With mex sync, which uses an LLM to re-check flagged facts, that rose to 89.2% and balanced accuracy to 82.9%.
• Finding code. On the 656 SWE-Explore issues in languages mex supports, mex put code the fix needed in its top five results for 65.9% of issues. Keyword search (BM25) did so for 30.2%.
• Staying exact. After every one of 24 incremental updates, mex's graph was identical to its own clean rebuild. The next best tool matched in 21 of 24.
• Search. mex ranked the right symbol first more often than any tool tested (Hit@1 66.7%, MRR 0.710) on a small public task set. On multi-hop tasks, code-review-graph scored highest.
• Impact analysis. mex had the highest precision (11.4%) and the highest F1 (0.157), with lower recall than four of the five other graph tools.

Our first benchmark post was a pilot. It compared mex with plain file search on 12 tasks across two repositories, and with a real agent in the loop. It answered one question: does mex hand an agent less noise. This round asks harder questions and holds mex to a stricter standard.

• A public benchmark. Code retrieval is scored on SWE-Explore [1], with the benchmark's own evaluator, not tasks we wrote.
• Other tools, not only a baseline. Five code graph tools and four memory tools, each installed and run by us, plus two baselines: keyword search and a plain AGENTS.md file.
• Answer keys no tool produced. Ground truth comes from the benchmark, from real commits, or from an audited fact set.
• A new question: staleness. Memory that is wrong is worse than no memory. We measured what each tool does with facts the code has since made false.

One rule decided which tools were in scope: we tested only what runs free and locally. Tools that need an LLM provider, a hosted or paid service, or several services were left out. They are listed in section 07. Every competitor was run in its native mode, without an LLM.

Project knowledge is the layer on top of the code: decisions, conventions, how a subsystem works. It goes stale when the code changes under it. The question is what a tool does with a fact that used to be true.

We replayed 367 commits of the Hono web framework. At checkpoint C5, the fact set held 132 facts: 102 still true and 30 made false by the code changes since they were written. We report three main metrics:

• Stale-served (lower is better): outdated facts returned with no warning.
• Fresh retention (higher is better): still-true facts returned cleanly, without a false warning.
• Balanced accuracy (higher is better): the average of fresh retention and (100 minus stale-served), so true and false facts count equally.

Table 5 also reports code recall@5: the percentage of the 94 code-linked facts found in the top five results when searching by their file and function.

We show mex two ways. mex check links each fact to the code it describes and flags the fact when that code changes, deterministically and with no LLM. mex check + mex sync adds a review step: mex sync runs your coding agent to re-check each flagged fact against the current code, and renews the facts that are still true.

Reading the result. Without a link to code, a tool has no way to know a fact went stale, so 90 to 100% of outdated facts were served as current. That is expected by design, and it is why those tools score high on fresh retention: they never warn. mex warns when the linked code changes. On its own that produces false alarms on 38.3% of returned true facts. With mex sync re-checking the flagged facts, false alarms fall to 2.2% and fresh retention rises to 89.2%, at the cost of the one wrongly renewed fact in note 3.

SWE-Explore gives a tool a repository and a real GitHub issue, and asks for a ranked list of the code regions that matter [1]. It contains 848 issues. We report the 656 written in languages mex supports: 546 in Python and 110 in JavaScript, TypeScript or Rust.

The metric is issue Hit@k: the share of issues where the top k results contain code the fix needed. We compare against keyword search (BM25) [2], the standard lexical baseline. None of the five other graph tools has a published SWE-Explore result, so this comparison is against keyword search only.

Reading the result. Finding the file is the easy part. The harder part is how much of what comes back is useful. On Python, 44% of the lines mex returned were relevant, against 10% for keyword search, measured without a line cap. Less of what mex returns is noise the agent has to read past.

A code graph is only useful if it tracks the code. Graph tools typically update incrementally after a commit instead of rebuilding from scratch. Incremental updates can drift: a renamed function keeps its old edges, a deleted file leaves nodes behind.

We replayed commits one at a time (16 single commits and 80 commits) plus one 80-commit jump. After each step, we compared the tool's incrementally updated graph with a clean rebuild by the same tool.

Reading the result. Agents query the graph between rebuilds, so whatever drift an update leaves behind is what they see. Matching a clean rebuild at every step means the graph an agent queries is the graph you would get by rebuilding from scratch.

For symbol search we used code-review-graph's public task set: 15 searches and 9 multi-hop tasks. It is a small set, so treat the differences as directional. We report mean reciprocal rank (MRR), which rewards putting the right answer near the top, and Hit@1, the share of searches where the first result is right.

Reading the result. An answer ranked fourth still costs the agent a step to work out which of several matches is right. A first-place answer removes that step. Multi-hop is the exception here: code-review-graph, using embeddings, scored 0.889, well above mex at 0.556.

Before changing code, an agent wants to know what else the change affects. We asked each tool for the impact set of a change, and checked it against 200 real commits: the files that actually changed together.

Reading the result. There is a real trade-off here. Four of the five other graph tools find more of the co-changed files, but at precision between 0.7% and 3.7%, so most of what they flag did not change. mex flags fewer files and is right more often. Recall is where mex has the most room to improve.

The numbers above come from different tests. The table below puts the graph tools side by side on what was measured, and on where their knowledge lives.

For the memory tools, the practical difference is what has to run, and whether the knowledge can be reviewed like code.

mex is the only tool tested that is git-native, linked to specific code, and checked against every code change with no LLM.

Not measured. supermemory, Zep, Graphiti, Letta and TencentDB-Agent-Memory need an LLM provider, a hosted or paid service, or several services, so they fall outside the free-and-local rule. codegraph, code-review-graph, codebase-memory-mcp, graphify and GitNexus have no real project-knowledge layer to test, so they appear only in the code graph sections.

Setup. Every tool was installed and run by us, on the same machine, against the same repositories and tasks. Competitors ran in their native modes without an LLM. Answer keys came from the benchmark (SWE-Explore), from repository history (co-changed files), or from the audited Hono fact set. The mex sync run needed a Windows launch workaround for the agent.

Scoring. SWE-Explore results use the benchmark's own evaluator. Update consistency compares each tool against its own clean rebuild. Search uses code-review-graph's published task set. Impact analysis is scored against files that changed together in 200 real commits.

Full method and raw data. The detailed method, the benchmark harness and the raw results are in BENCHMARKS.md in the mex repository.

Limits. We want these to be as visible as the results.

• SWE-Explore results cover the 656 of 848 issues in languages mex supports. We make no claim about the rest.
• The search set is small: 15 searches and 9 multi-hop tasks.
• Co-changed files are a proxy for impact, not a causal truth.
• The knowledge benchmark covers one repository, Hono. Its 132 facts were written with AI and its answer key was audited by AI. A maintainer spot check is pending.
• Competitors ran without an LLM. mex check + mex sync uses one, so that row is not a like-for-like comparison with their AI update paths, which we did not test.
• In the sync run, one outdated fact (h018) was wrongly renewed. It is counted in the results.

None of this supports claims like "most accurate" or "always correct", and we don't make them. The claim the data supports is narrower. On these tests, mex found the code a fix needed far more often than keyword search, ranked symbols best on a small search set, stayed identical to its own clean rebuild, and served far fewer stale facts than the memory tools we could run free and locally.

• Over-renewal in mex sync. The h018 error came from renewing an outdated fact that lists a complete set (the CORS default methods). Fixing it is on our roadmap.
• Retrieval recall. mex trails four of the five other graph tools on impact recall. We want to raise it without giving back precision.
• A maintainer-checked answer key. The spot check of the Hono fact set is pending. Until it is done, we disclose it.

1. Zhang, S. and colleagues. "SWE-Explore: Benchmarking How Coding Agents Explore Repositories." arXiv:2606.07297, 2026.
2. Robertson, S. and Zaragoza, H. "The Probabilistic Relevance Framework: BM25 and Beyond." Foundations and Trends in Information Retrieval, 2009.
3. mex. 90% less noise, up to 70% fewer tokens: the numbers behind mex. August 2026.