Engram Studio · Technical report · October 2026
I built a memory system for LLM conversations on ACT-R, the cognitive architecture for human recall, then measured it hard enough to find out that the cognitive science was not what made it work. Here is what did, what didn’t, and the places the measurements contradicted my own design.
Retrieval loses to stuffing the whole conversation into context — by 1.9 accuracy points, on 12% of the tokens. But perfect retrieval beats stuffing, and that is the result the project turns on.
| Arm | Context tokens | Accuracy | Abstained | Accuracy when answering |
|---|---|---|---|---|
| Oracle Fragments | 89 | 67.1% | 14.3% | 78.3% |
| Full Context | 20,190 | 62.6% | 14.0% | 72.9% |
| Engram Engine | 2,466 | 60.7% | 15.3% | 71.7% |
| BM25 | 2,427 | 51.5% | 24.4% | 68.0% |
| Recency Window | 2,619 | 21.8% | 63.0% | 59.0% |
| Comparison | Difference | 95% CI | Significant |
|---|---|---|---|
| Engram Engine − Full Context | −0.019 | [−0.033, −0.005] | yes |
| Oracle − Full Context | +0.044 | [+0.016, +0.072] | yes |
| Engram Engine − BM25 | +0.093 | [+0.065, +0.117] | yes |
| Engram Engine − Recency Window | +0.389 | [+0.355, +0.418] | yes |
| Engram Engine − mem0 | +0.163 | [+0.121, +0.198] | yes |
The honest claim is efficiency, not quality. 97% of the accuracy for 12% of the tokens. I predicted stuffing would win before running this, and recorded the prediction; it won by less than I expected, because distraction costs it more than I thought. Its accuracy-when-answering is 72.9% against the oracle’s 78.3% while holding strictly more evidence.
Each is labelled with what kind of evidence it is. Two of them contradict decisions I had already written down and shipped; one is a comparison against the obvious alternative architecture.
I designed the system around episodes — semantically coherent spans of conversation — reasoning that a short question plus a long answer is one unit of meaning. I wrote that down as a design decision and built on it for weeks.
Then I swept the unit from a whole session down to a single turn, expecting a sweet spot in the middle: narrow units hold less text to match a question against, so they should get harder to find. There is no sweet spot. Precision and recall both improve monotonically all the way down. The reason is budget arithmetic, not anything subtle — a 2,000-token budget holds about 57 single turns but only 2.6 session-sized ones, and twenty times the attempts beats slightly better aim.
4–8× fewer tokens at matched recall. Larger than every other result here combined.
“Smaller chunks work better” replicates known information retrieval work and is not interesting on its own. What makes it interesting is the control. Running BM25 — a 1994 term-weighting function with nothing to tune — over the same units shows it barely cares about unit size, while every dense embedding model roughly halves.
So the effect belongs to the dense representation, not to the task getting harder. Pooling a long, multi-topic passage into one vector averages away the specific fact that answers the question. Per-term scoring with explicit length normalisation does not. Doubling the embedding dimensions makes it slightly worse, which rules out representational capacity as the explanation.
ACT-R strengthens a memory each time it is used. The obvious move is to strengthen whatever retrieval returned. It does not work, and it is not a tuning problem: I swept 24 combinations of strength and targeting, including an oracle that strengthened only the genuinely relevant memories using the answer key. The best achievable gain over simply not reinforcing was exactly zero.
The reason is a shape mismatch. Relevance is specific to a question, but base-level activation is one number per memory, used for every future question. Writing “this answered question 37” into it promotes that memory for the other 1,530 too. Since evidence is reused across questions, everything eventually gets promoted — which is the same as nothing being promoted, plus noise.
A perfect success signal is still worth nothing, so no attainable signal can help.
Spreading activation — the part of ACT-R that makes memory associative — never earned meaningful weight. I had built the associations from embedding-similarity clusters. Making that signal cue-driven tripled its ability to separate answering turns from the rest, and the weight tuner responded by setting its weight to zero and discarding it entirely.
Clusters are built from embedding similarity, so a sharper version was just a sharper copy of a signal the model already had. Rebuilt on co-occurrence in time — which is what ACT-R’s association actually means, and which the embedding cannot supply — the signal is nine times stronger, nearly independent of similarity, and the tuner keeps it in every fold.
The lesson generalises to any knowledge graph bolted onto a vector store: an edge built by semantic similarity adds almost nothing.
mem0 is the closest widely-used system, and it bets the other way: every turn goes through an LLM that extracts, rewrites and merges facts, and those facts are what gets retrieved. We keep the conversation verbatim and rank it.
On the same questions with the same judge, Engram Engine is 16.3 points more accurate (+0.163, CI [+0.121, +0.198]) on ten times the tokens — 2,466 against 253. That is not mem0 losing. It is a different point on the same frontier, and at a 253-token budget mem0 wins outright, because at that budget we retrieve almost nothing.
The real trade is where the cost falls. mem0 pays an LLM call per session on the write path — 30 minutes for this corpus, scaling with conversation length — against one local embedding per turn for us. In exchange its read path stays an order of magnitude cheaper forever. Which side wins depends on the read/write ratio of the workload, and nothing here measures that.
Write-time extraction also makes errors permanent: a wrong date baked into a stored fact cannot be recovered, where raw text stays fixable by a later read.
The pitch for retrieval is that it sends fewer tokens. But caching makes the tokens you don’t save about ten times cheaper, and stuffing a conversation is a perfect cache prefix — every turn is appended to an unchanged history. Retrieval reshuffles its payload every turn, so almost nothing stays cached.
At a one-hour cache lifetime, stuffing caches 89% of its prefix and wins on cost and recall. I built a core/tail layout to fix this; it became 97% more cache-efficient and still lost.
You cannot out-cache something that is 100% prefix. Retrieval’s cost advantage lives at short cache lifetimes and tight budgets — and nowhere else.
This is the part I would most like to have come out differently, so it goes near the top rather than in an appendix.
| Change | Gain |
|---|---|
| Fixing the retrieval unit | +0.365 |
| Swapping to a better off-the-shelf embedder | +0.079 |
| The whole ACT-R blend | +0.030 |
| Standard lexical fusion | +0.030 |
The ACT-R contribution is real, statistically significant, and survives cross-validation — base-level activation took positive weight in all ten folds, and so did temporal spreading once its association structure was rebuilt. It is also smaller than a one-line model swap. Three independent comparisons now say the cognitive architecture is a refinement rather than the main effect, and a write-up that led with the neuroscience would be misrepresenting its own numbers.
Every figure on this page has a command behind it. The scoring layer imports neither the store nor the embedder — it is plain array maths over a columnar frame — which is why swapping the retrieval unit, the single largest change in the project, was about twenty lines rather than a rewrite.
engram granularity --widths 1,2,4,8,16,32,session engram retune --width 1 --budgets 500,1000,2000 engram precision --width 1 --embedder-model BAAI/bge-small-en-v1.5 engram answer --width 1 --budget 2000
Language model responses are cached on disk by prompt hash at temperature zero, so a re-run costs nothing and the numbers do not drift between runs. That was not a convenience: the first full evaluation died partway through on a rate limit, and every completed call survived.