Research
Benchmarks & research foundations
MetaMemory is grounded in cognitive science and validated on production benchmarks. Here are the numbers, the methodology, and the research that informs our architecture.
Benchmark results
84.4%
LongMemEval-S (overall)
Overall accuracy across all 500 LongMemEval-S questions, vs. a GPT-4o full-context baseline of 60.6%.
89.0%
LongMemEval-S (task-averaged)
Accuracy averaged across the LongMemEval task types, including 96.7% abstention accuracy.
95.3%
Single-session accuracy
Single-session user questions on LongMemEval-S; 100% on single-session preference questions.
0.97
Session Recall@10
Fraction of LongMemEval questions where the evidence session is retrieved in the top 10.
0.75→1.00
Recall@10, learning on
Corporate simulation runner: the channel weight learner lifts recall@10 from 0.75 to 1.00 over 50 feedback iterations, while the static (learning-off) baseline stays flat.
0.64→0.88
MRR, learning on
Mean reciprocal rank over the same run. Ablating the channel weight learner removes the improvement entirely.
Evaluation methodology
LongMemEval evaluates long-term memory across single- and multi-session conversations. We ran the full 500-question LongMemEval-S set, ingesting 23,867 sessions (~247,000 conversation turns) into MetaMemory and answering each question from retrieved memories only. MetaMemory scores 84.4% overall and 89.0% task-averaged, vs. a GPT-4o full-context baseline of 60.6%. Per task type: 100% single-session preference, 95.3% single-session user, 92.9% single-session assistant, 87.5% knowledge update, 76.9% multi-session, 74.0% temporal reasoning, and 96.7% abstention accuracy.
The corporate simulation runnermeasures whether retrieval actually improves with use. We seed a simulated engineering-org knowledge base (projects, incidents, people, decisions — 1,000 memories) containing 48 adversarial “needle” memories engineered so that pure semantic search is pulled toward plausible distractors, then run 50 iterations of queries with the retrieval feedback loop closed. With learning on, the channel weight learner adapts the fusion weights across the retrieval channels and recall@10 climbs from 0.75 to 1.00 (MRR 0.64 → 0.88). With learning off, recall stays flat. Ablation runs attribute the gain to the channel weight learner: removing it removes the improvement.
Cognitive science foundations
Tulving's Memory Taxonomy
MetaMemory's architecture draws inspiration from Endel Tulving's taxonomy of long-term memory — semantic (facts), episodic (experiences), and procedural (skills). Our implementation maps these concepts to distinct retrieval signals — semantic embeddings, episodic session structure, temporal metadata, and emotional tags — optimized for AI agent retrieval rather than being a 1:1 replica of the cognitive model.
Multi-Armed Bandits for Retrieval
Adaptive strategy selection uses Thompson Sampling and Upper Confidence Bound (UCB) algorithms from the multi-armed bandit literature. These are well-studied Bayesian methods for balancing exploration and exploitation — applied here to learn which retrieval channel works best for each query type.
Memory Consolidation Theory
Inspired by the complementary learning systems (CLS) theory of hippocampal-neocortical memory transfer, MetaMemory's consolidation process mirrors what happens during sleep: related memories are merged, redundant information is compressed, and important connections are strengthened.
Reciprocal Rank Fusion
RRF is a well-established information retrieval technique for combining ranked results from heterogeneous sources. MetaMemory uses RRF to fuse results from its 5 specialized retrieval channels into a single ranking, avoiding the score calibration problems of raw score merging.
References
- [1]Tulving, E. (1972). Episodic and semantic memory. In Organization of Memory.
- [2]Tulving, E. (1985). Memory and consciousness. Canadian Psychology, 26(1).
- [3]McClelland, J.L., McNaughton, B.L., & O'Reilly, R.C. (1995). Why there are complementary learning systems in the hippocampus and neocortex.
- [4]Thompson, W.R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples.
- [5]Cormack, G.V., Clarke, C.L., & Buettcher, S. (2009). Reciprocal rank fusion outperforms condorcet and individual rank learning methods.
- [6]Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024). LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory.
Want to run your own benchmarks?
Start with the free tier and test MetaMemory against your own evaluation suite. No credit card required.