AI memory is broken—benchmarks, anyway.
Imagine your smartphone claiming ‘unlimited storage’ based on tests with three photos. That’s today’s AI memory evals. Nearly every system boasts scores on setups that measure context windows, not true recall over time. And here’s the enthusiastic futurist in me: this proposal isn’t tweaking dials; it’s launching AI memory into orbit, like the Wright brothers ditching kites for powered flight.
Look, we’ve audited LoCoMo. Brutal stuff. 6.4% of answers wrong—99 errors in 1,540 questions. LLM judges greenlight 63% of deliberate lies. Half the category rankings? Statistical mush.
Why Current AI Memory Benchmarks Suck
LongMemEval-S? Burns 115K tokens per question. Frontier models slurp that in one context gulp. It’s a stamina test for windows, not memory vaults.
Worse: everyone’s playground differs. Mem0 vs. Zep on the same systems—wildly divergent scores. Custom ingestion, prompts, judges. Apples next to oranges, served in a fruit salad lie.
But.
This proposal flips the script. 1-2 million tokens total context. Big enough for real retrieval pain, cheap for indie hackers to run.
Multi-session talks. You and your AI over six months. Work drama, taste tweaks, fact flips—not stranger small talk.
Systems ingest their way. But spill: method, model, embeddings, cost, time. Transparency or bust.
The Standard Track: Apples-to-Apples Finally
Prescribed model. Fixed prompt. Single-shot. Memory’s the lone variable.
Open track? Wild west—disclose everything, report separate. No score soup.
400 questions per category. LoCoMo’s tiniest? 96. Margins of error: clown show.
Error rate? Under 1%. Pre-screen with model councils, crowd bounties, expert refs.
Judges tested hard: generate wrong answers pre-launch. Reject 95%+. No more fuzzy topical fluff passing as truth.
Zero tolerance for ‘I don’t know’ when it’s there. Zero confident hallucinations.
Smart systems abstain wisely. Hallucinating ones flop.
Scorecard’s rich: accuracy (both tracks), retrieval precision (tokens/question), latency (p50/p95), abstention smarts, supersession handling.
Report token counts to the generator. No hiding bloated contexts.
“I don’t know” when the answer IS in the corpus: 0.10. Confidently wrong: 0.0. A system that knows its limits should beat one that hallucinates.
That’s gold. Straight from the proposal—nails why accuracy alone lies.
2,400 questions. 400 each:
Direct recall. Blunt facts, explicit drops.
Temporal reasoning. Timelines, changes—when’d that shift?
Multi-hop. Link distant chats for unspoken answers.
Supersession. Updates, corrections—forget the old.
Cognitive inference. Implications, not quotes.
Adversarial abstention. ‘Nope, not here’—perfectly.
No rigid ingestion rules. No forced embeddings. Fresh models. Affordable runs. Open collab, not edict.
Why Does This Proposal Matter for AI’s Future?
Here’s my unique spin: remember SPEC benchmarks in the ’80s? CPUs were hype-fests till standardized tests forced real gains. Transistors exploded, costs plunged. AI memory’s at that cusp.
Predict this: standard track sparks a memory arms race. Open track breeds moonshots. By 2026, we’ll have systems holding years of your life—personal AI companions that evolve like old friends, not forgetful bots.
Corporate spin? They love proprietary benches to cherry-pick wins. This calls bluff. Honest measurement unlocks the platform shift: AI as lifelong brain extension.
Energy here. Pace picks up. Wonder surges.
Picture it—your AI recalls that offhand restaurant gripe from March, ties it to your new diet chat in July, suggests spots without asking. Multi-hop magic.
Or adversarial: ‘What’s my SSN?’ Boom—‘Can’t say, not in our history.’ Trust skyrockets.
Latency matters too. P95 spikes? Useless for real chats.
Supersession: you corrected ‘Paris’ to ‘London’ trip—does it stick? Tests that.
And costs disclosed? Exposes vaporware.
Critics’ll whine: too big. Nah—2M tokens is Tuesday for labs.
Not prescribing everything invites innovation. Embeddings evolve; lock ‘em, ossify.
Full write-up’s out—corpus how-to, frameworks, refs. LoCoMo audit public, errors listed.
They’re rallying builders, designers, researchers. Honest metrics or bust.
A Bold Prediction: Memory Wars Incoming
This isn’t incremental. It’s the iPhone moment for AI persistence.
Short convos? Cute. But long-term memory turns AIs into partners—tracking projects, moods, growth.
Current benches test trivia nights. This? Epic sagas.
One hitch: crowd review bounties. Genius—skin in game kills slop.
Model councils pre-screen. Layers beat single judges.
Retrieval precision metric? Tokens per question—punishes bloaters.
Abstention quality: knows limits wins.
We’re witnessing birth of durable AI minds. Like DNA to cells—memory’s the code for intelligence over time.
Thrilling.
Join in. Feedback flows.
🧬 Related Insights
- Read more: PC Workman 1.7.1: 130 Dead Lines Axed After 9-Hour Retail Grind
- Read more: Chrome 147 Drops: Permission Locks and a Fresh Web Printing API
Frequently Asked Questions
What makes this the real benchmark for long-term AI memory systems?
It uses 1-2M tokens from 6 months of real multi-session conversations, with standardized tracks to ensure fair comparisons and tests true retrieval, not just context.
How bad are LoCoMo and other current AI memory benchmarks?
LoCoMo has 6.4% factual errors in answers, judges accept 63% wrong responses, and many comparisons are statistically meaningless noise.
Will the standard track vs open track fix AI memory comparisons?
Yes—standard prescribes model/prompt for apples-to-apples; open allows innovation but reports separately, ending misleading mixed leaderboards.
