90% less noise, up to 70% fewer tokens: the numbers behind mex

Every few months, a model ships with a bigger context window, and it's tempting to treat that as a solved problem. If an agent can hold your entire repo in its head, why build anything smarter than "just give it everything"?

The gap is that a model doesn't read a million tokens the way a database reads an index. Every token is technically processed, but not every token gets equal weight in what the model actually does next. Put a relevant file next to a hundred irrelevant ones, and the model still has to find the signal inside the noise, every single call. That search isn't free. It costs tokens to process, it costs latency, and past a certain point it costs correctness, because the right answer can be sitting in the context window and still not be the thing the model acts on.

mex is our attempt to sidestep that problem rather than out-scale it. Instead of handing an agent more to sift through, it decides what a task actually needs and hands back only that. This post is the data behind that claim.

Researchers at Stanford, Berkeley, and Samaya AI found that a model's accuracy at using information in its context follows a U-shaped curve depending on where that information sits: best near the start or end, worst in the middle, and this held even for models built specifically for long context (Liu and colleagues, "Lost in the Middle," TACL 2024). NVIDIA's RULER benchmark pushed on this from a different angle, testing whether a model's claimed context length matches its effective one. It found that models score well on simple needle-in-a-haystack retrieval but degrade substantially on harder multi-hop and aggregation tasks well before reaching their advertised context length (Hsieh and colleagues, "RULER," 2024).

A 2026 benchmark built specifically around this question, ContextBench, evaluated four frontier models and five coding agents across 1,136 real issue-resolution tasks and found what it calls the "Bitter Lesson" of coding agents: more sophisticated agent scaffolding produces only marginal gains in context retrieval on its own, models consistently favor recall over precision, and there's a persistent, measurable gap between the context an agent explores and the context it actually uses to produce its answer (Li and colleagues, "ContextBench," 2026). That gap, between what gets explored and what gets used, is close to the distinction our own fallback numbers point at: a larger response doesn't help if the agent still has to search around it to actually use it.

RepoGraph built a repository-level code graph as a plug-in for AI coding agents and evaluated it on SWE-bench, the standard benchmark for real-world repo-level coding tasks, and found it substantially improved performance across four different agent architectures, producing a new state-of-the-art among open-source frameworks at the time (Ouyang and colleagues, "RepoGraph," ICLR 2025). It's a different implementation from ours, evaluated on a different benchmark, but it's the closest thing to independent confirmation of the core idea mex is built on: a structured, navigable representation of a codebase beats handing an agent more raw text.

Industry work is converging on the same conclusion from a different direction. SWE-Pruner trains a lightweight model specifically to prune a coding agent's context down to only the lines relevant to its current goal, rather than relying on a human or a fixed heuristic to decide what to keep. Sourcegraph made a related argument in their own 2026 write-up on context engineering for coding agents: retrieving a symbol's definition directly through code intelligence, rather than a ranked list of files that merely mention it, turns a diffuse search problem into a precise one, the same distinction our minimal mode's graph get mechanism is built around.

We ran two different types of test.

The first is a controlled, deterministic retrieval test. Given a task, how much does mex return compared to a reasonable baseline, and does it still find the right answer, every time, byte-identically. This isolates the retrieval logic itself from any variance an LLM might introduce. It tells you what mex's selection mechanism does on its own, before an agent ever gets a chance to act on it, get confused by it, or route around it.

The second is a real-agent test. We put an actual coding agent (Claude Sonnet, running headless through the claude -p CLI) in front of two setups: one with only ordinary file search tools (Read/Grep/Glob), and one using mex. Same tasks, same model, same prompts. This is the test that matters more in practice, because retrieval efficiency in isolation doesn't tell you whether an agent actually ends up doing less work or getting better answers. An agent could easily undo a compact retrieval result by grepping around it anyway, and a synthetic benchmark alone would never catch that.

Invoking the agent headless through the claude -p CLI, running one fresh session per task with no interactive back-and-forth, follows standard harness design principles established by agent benchmarks like Terminal-Bench and SWE-agent. In interactive multi-turn loops, an agent accumulates conversation history and context-compaction artifacts across runs that can unpredictably bias downstream tasks. Spawning a fresh, non-interactive subprocess for each task (pinned CLI version, single-shot invocation, no --resume) guarantees that every evaluation run is order-insensitive, isolated, and strictly reproducible.

We ran 6 retrieval tasks and compared mex's output size against a grep-top-3 baseline doing the same job, the number of tokens a well-configured agent would burn reading the top three grep hits for that task. Token counts are estimated as ceil(characters / 4), and the same baseline logic was reused unchanged from an earlier, separate benchmark, so this isn't a comparison invented after the fact to flatter the result. The subject graph itself, mex's own repository at the time, indexed 152 files into 1,801 nodes and 2,808 edges in about 7.5 seconds, against a harness corpus of 251 source files and roughly 730,000 tokens, so the baseline this is being measured against is not a toy.

That median wasn't the starting point, either. Before a deliberate rework of the retrieval surface, the same six tasks scored a median of only 1.34x against the same grep baseline, and one task, runDriftCheck, was actually worse than grep, returning 27,396 tokens across 32 separate facts for a single query, an over-expansion pathology, not a win. The rework that produced the 10x number specifically added a hard cap on how many nodes a single response could return and made source code opt-in rather than automatic, and runDriftCheck alone went from 0.26x (worse than grep) to 5.90x once those caps were in place. The 10x figure isn't a baseline that was always true, it's the result of a specific fix, which matters if you're trying to understand what actually earns the improvement rather than treating it as an ambient property of the tool.

A separate check ran the same tasks through graph query where-defined, asking the graph to locate the exact declaration of a symbol rather than a broader task-relevant scope. The found rate was 1.0, and every single result ranked first, not third, not "somewhere on the first page," first. That distinction matters for an agent specifically: a correct answer buried at rank 4 still costs the agent a disambiguation step, additional reasoning to decide which of several plausible matches is the right one. A rank-1 exact match removes that step entirely.

Why this test matters on its own, separate from anything downstream: it isolates the selection logic from everything else. It tells you that when mex decides what to retrieve, that decision is genuinely tighter than what naive search would surface, independent of how any particular model then chooses to use it. But a synthetic baseline can't tell you whether a real agent behaves any differently as a result, that's what the next test is for.

We ran 12 natural-language coding tasks across two codebases, six against mex's own repository and six against the Hono framework, through Claude Sonnet, once with plain file search and once with mex.

The six Hono tasks were:

1. How does an incoming Request enter a Hono application, get matched to routes, and dispatch through one or multiple handlers?
2. How does Hono's smart router choose the first router that supports all registered paths and then reuse it for later matches?
3. How are Hono middleware handlers composed so next advances exactly once while errors and not-found responses are handled?
4. How does request validation extract JSON, form, query, parameter, header, or cookie input and make the validated value available to later handlers?
5. How does a Hono Context construct a Response while combining prepared headers, explicit headers, cookies, and status values?
6. How does Hono build safe cache keys for QUERY requests using a bounded body digest, representation metadata, and Vary headers?

Two token figures are reported here on purpose, and they measure genuinely different things, not the same number rounded differently. "New tokens" is uncached input plus cache writes plus output, the tokens the run actually had to newly process for that specific task. "Total tokens processed" additionally includes cache reads, which is dominated by the shared system prompt and tool schemas repeated on every single call regardless of which arm is running. New tokens is the more honest signal of what the task itself cost, since cache-read volume is mostly constant overhead that neither arm controls, while total processed tokens is useful for understanding absolute infrastructure load but can make a small task-level difference look artificially larger or smaller depending on how much shared prefix happened to be cached that run.

The correctness split by repository is worth showing rather than collapsing into one number: on Hono, mex got 5 of 6 correct versus 3 of 6 for files-only, with a 7.4% reduction in total new tokens on that repo specifically. On mex's own codebase, both arms tied at 4 of 6 correct, but mex still cut new-token usage by 72.3% on those tasks specifically.

Retrieval quality backed this up independent of whether the final answer was graded correct: mex surfaced the right file on the first try in 11 of 12 tasks (6 of 6 on Hono, 5 of 6 on MEX), returned 22 of 23 required source snippets (14 of 14 on Hono, 8 of 9 on MEX), found every required directed code path on Hono where those were declared (6 of 6), and had complete graph evidence coverage across all 12 tasks. The mean number of distinct Scope queries per candidate run held at exactly 1.0 across the board, meaning the agent typically got what it needed from a single retrieval call rather than iterating or retrying.

In a separate 5-task test with the same real agent, mex's default mode (minimal) never once fell back to grep. mex's source mode, which returns the actual code directly instead of a compact summary, fell back to grep on 4 of the 5 tasks, because the code it returned didn't fully cover what the agent needed. The per-task breakdown makes the pattern explicit rather than just the summary number: on scope-select, minimal made 3 follow-up graph get calls and never fell back, while source made zero follow-up calls but fell back to grep twice on that single task. On impact-walk, neither mode fell back. On budget, compact-fact, and source-expand, minimal made 2 follow-up calls each with zero fallbacks, while source made zero follow-up calls and fell back once on each. The pattern holds task by task, not just in aggregate, which is a meaningfully stronger claim than a single summary ratio would be.

This matters more than it might look, because a fallback to grep is the exact failure mode the whole approach exists to eliminate: it means the agent, despite having mex available, went right back to unstructured search anyway. Zero fallbacks means the retrieval, in its default configuration, was self-sufficient for every task in that test. The trade-off is visible in cost and turn count, not just correctness: minimal averaged 4.4 turns per task at $0.20, while source averaged 3.0 turns at $0.17, both fully correct on all 5 tasks. Source is genuinely a little cheaper and a little faster when it works. The reason minimal ships as the default anyway is explained below, and it isn't because it wins on every metric, it's because the metric it wins on is the one that doesn't degrade unpredictably on repositories nobody has measured yet. The ~$0.03 cost difference between the two modes, worth noting explicitly, sits within the roughly 35% run-to-run cost noise this kind of prompt-cache-dominated measurement produces at this sample size, so it's not a difference we'd treat as decisive on its own even before considering the fallback behavior.

We rebuilt mex's graph on a TypeScript codebase (a sparse checkout of a real compiler's src/compiler subtree, not a synthetic toy) and on a synthetic multi-language repo spanning TypeScript, Python, and Rust, and checked what came back. Every rebuild produced byte-identical output, confirmed via normalized graph-content hashes across repeated rebuilds, and retrieval found the right answer 100% of the time on both suites, with the synthetic multi-language suite additionally scoring 0.9167 on Recall@5, 0.875 on Mean Reciprocal Rank, and 0.9095 on nDCG@10 across its full ranked-retrieval evaluation. Integrity gates on both suites found no extraction or storage loss, no duplicate or dangling graph rows, no full-text-search index drift, no invalid confidence values, and no suspicious edges linking production code to test code, the kind of structural corruption that wouldn't show up in a correctness score but would quietly poison results over time.

Determinism matters here specifically because mex is meant to be a persistent memory layer, not a cache. If the same query could return different results after an unrelated rebuild, grounding references inside the wiki would silently rot over time, an anchor pointing at a symbol that used to exist in a particular shape but now resolves to something subtly different. Byte-identical output means that risk doesn't exist by construction on the repositories tested, not just by observation on one lucky run.

We also measured indexing time and disk usage across five repositories, comparing our previous release (0.7.2) against the current one (0.7.3):

Storage dropped 36-46% across the board, with identical graph output. On a per-file basis, that's roughly 176 KB down to 104 KB for the smallest repo tested, and 220 KB down to 142 KB for the largest, so the saving holds up whether you look at total footprint or per-file density. The saving comes from a schema rewrite: the tables handling fingerprint lookups previously stored binary similarity sketches and hash bands as separate, loosely-packed rows, together accounting for 37-48% of every 0.7.2 store measured. The rewrite re-encodes them as binary MinHash sketches with integer band hashes and integer fingerprint references, folded into a single composite primary key in place of what used to be a separate lookup table entirely. That's a storage-layer change, not a change in what gets indexed, which is why node and edge counts were verified identical across both versions on every repo, and why the graph output itself stays byte-identical before and after.

With those fingerprint tables shrunk down to a minor line item, the largest table in the 0.7.3 store, on every repository measured, not just the Python-heavy ones where you might expect it, is now unresolved_refs, references the extractor found but couldn't yet resolve to a concrete declaration.

Indexing got faster on most repos too, up to 41% on the largest ones we tested, with one small repo (A) showing a slight regression, 9.9 seconds up to 11.8.

Note: the largest repo's figures come from an earlier, separately-run paired measurement rather than this specific sweep, since a repeat run of that same repo on 0.7.2 alone recorded 7.0 minutes rather than 448 seconds, a large enough swing that we chose to report the original paired numbers, both halves measured under one consistent harness, rather than mix instruments.

mex can respond to a task in one of two ways: minimal, which returns a compact summary of the relevant code (no source) plus a way to fetch it on demand if needed, or source, which returns the code directly in the first response. Minimal is the default, and the reasoning behind that default, rather than just the fact of it, is worth walking through.

The intuitive assumption is that giving an agent more upfront, actual code instead of a summary, should mean it needs to do less afterward. The per-task data above says the opposite, consistently, not just on average. When mex responds with minimal, the agent gets back a summary plus an explicit next step: a graph get <id> call it can make if it needs the underlying source. That next step is precise, pointing at exactly the thing the agent identified as relevant, so the agent almost never needs to reach outside the graph for anything. When mex responds with source instead, it hands over a fixed block of code, and if that code doesn't happen to fully answer the question, the agent has no structured path forward from there. Its only option is to fall back to grep, the exact unstructured search mex exists to replace, and the per-task breakdown shows this happening on four of five tasks, not as a rare edge case but as the typical outcome for that mode.

So the failure mode isn't "not enough information," it's "no path to more information." A smaller response with a clear way to expand it beats a larger response with no way to correct for what it missed. That's the actual mechanism behind the zero-fallback result above, not a coincidence of these five tasks specifically, and it's also why the decision to default to minimal was made even though source was marginally faster and marginally cheaper on this sample: correctness was already tied, the cost edge was smaller than the run-to-run noise, and minimal's self-sufficient behavior is the one more likely to hold up on the much wider range of repositories this hasn't been tested on yet. Source remains available for cases where a one-shot, answer-ready response is worth the fallback risk.

• Liu, N. F. and colleagues. "Lost in the Middle: How Language Models Use Long Contexts." TACL, 2024.
• Hsieh, C. and colleagues. "RULER: What's the Real Context Size of Your Long-Context Language Models?" NVIDIA, 2024.
• Ouyang, S. and colleagues. "RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph." ICLR, 2025.
• Li, H. and colleagues. "ContextBench: A Benchmark for Context Retrieval in Coding Agents." 2026.
• Wang, Y. and colleagues. "SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents." 2026.
• Sourcegraph. "Context Engineering: A Practical Guide for AI Agents." 2026.

Everything above is reproducible from the repo:

git clone https://github.com/mex-memory/mex.git
cd mex
npm ci
npm run build
npm run eval

Raw results, task definitions, and the full evaluation harness are in the evaluate/ directory.