Shared Brain Memory Benchmarks

HivemindOS is built to remember the useful parts of work without turning every run receipt, old decision, or vault note into prompt baggage. These benchmarks measure whether that design produces relevant recall, fast local retrieval, cleaner current truth, and smaller model context.

Results At A Glance

Top-1 means the expected memory ranked first. Top-3 means it appeared in the first three results. Mean reciprocal rank, or MRR, rewards putting the correct memory as high as possible.

LongMemEval Oracle Against Sibyl-Memory

Sibyl-Memory publishes a 95.6% LongMemEval score from LongMemEval Oracle: 500 questions, each with the past conversations that hold its answer. We ran the same 500 questions through the Superbrain as an agent asks it a question (hive-brain answer), with Claude reading the answer from what the Superbrain returned: Opus 4.6, the model behind Sibyl-Memory’s headline, and Sonnet 4.5, the model of its plugin run. Every answer, ours and Sibyl-Memory’s own published ones, was graded by LongMemEval’s official per-type judge prompts with gpt-4o-2024-08-06.

• Questions: all 500 of longmemeval_oracle.json from the cleaned LongMemEval release. Each question’s conversations are written into a fresh, empty Superbrain, then the question is asked with its own date as today.
• Reader: LongMemEval’s official reading prompt, unchanged; temperature 0; no extended thinking.
• Sibyl-Memory: its published answer files from Sibyl-Labs/memory-bench-kit, graded by the same judge. We did not re-run its system.
• Headline: excludes the 30 preference questions, as Sibyl-Memory’s does. Both figures are shown.

What the comparison shows:

• With the same reading model, HivemindOS scores higher. With Opus 4.6, on the same 470 questions it answered 16 that Sibyl-Memory missed, and Sibyl-Memory answered 10 that it missed (exact McNemar test, p = 0.33). With Sonnet 4.5 the counts were 21 and 10 (p = 0.07).
• Ahead, not proven beyond chance. Six questions out of 470 is within what a second run of either system could move. Updated facts and dates carry most of the lead; Sibyl-Memory’s Opus run counts across conversations slightly better.
• What moved it. The first full HivemindOS run scored 91.1% with Sonnet 4.5. Four changes to how the Superbrain answers a question took it to 95.1%. It finds conversations by the days a question names (“two weeks ago”, “last Saturday”) even when no word matches. It labels every session with how long before today it was. It hands over the conversations it found whole when they fit. It no longer repeats an excerpt of a conversation above its full text.

Limitations: the changes above were found by studying the misses on these same 500 questions, and there is no separate held-out set. They are general product behaviour rather than per-question rules, and they shipped for every user. One run per configuration. Every full run we did on this set is reported with the receipt: Sonnet 4.5 first version 91.1%, reported 95.1%, with the optional conversation ledgers 94.5%; Opus 4.6 reported 96.6%, with ledgers 96.0%. Sibyl-Memory’s own 95.6% comes from its string-matching scorer, which on step-by-step answers accepts almost any answer containing the right number anywhere; graded by the official judge, its Opus run scores 95.3%. Oracle hands each question only the conversations that hold its answer, so it tests reading and recall within them, not search through a large history.

The published receipt holds the exact text the Superbrain handed the reading model for every question, every answer and every verdict. Two free checks need only the public dataset: one confirms that every piece of conversation text the model saw comes word for word from that question’s own history, and one recomputes every number above. Scripts to grade again or read again on your own API key are included.

LongMemEval_S Against Sibyl-Memory’s Architecture

LongMemEval Oracle hands each question only the conversations that hold its answer. LongMemEval_S asks the same 500 questions with each answer buried among about 50 conversations (about 490,000 characters), which is the job a memory system is for. Sibyl-Memory has published no S result, so we ran its published architecture ourselves with its public runner: Claude Code reads the question’s whole chronological journal under Sibyl-Memory’s instructions. Its plugin product was not run, because the integration behind its plugin score is shared only with its beta testers.

• Both sides: Claude Opus 4.6 reading, LongMemEval’s official judge prompts with gpt-4o-2024-08-06.
• Sibyl-Memory: its runner unchanged except the per-question time limit (120 s to 600 s, because S journals are about ten times longer than Oracle’s), run with an empty home directory so no local settings reached it. The plan was published before it answered anything.
• HivemindOS: hive-brain answer as it ships, without model ordering or embeddings.

HivemindOS answered 26 questions Sibyl-Memory missed and Sibyl-Memory 16 that HivemindOS missed (p = 0.16): higher, not established beyond chance, while handing the model about a ninth of the text. Deviations are disclosed with the receipt: two copies of the run overlapped and answered 200 questions twice (the first answer counts), and requests refused for lack of API credit were re-sent.

LoCoMo Held-Out Test

The improvements behind the LongMemEval results were found on LongMemEval’s own questions. To test whether they only fit those questions, we pre-registered a comparison on LoCoMo, which none of them was developed on: 500 sampled questions, the memory code before the LongMemEval work against the code after it, Claude Opus 4.6 reading, Mem0’s published LoCoMo prompts and judge.

The newer code answered 13 questions the older missed and missed 8 it answered (p = 0.38). The changes did not overfit LongMemEval, and they are not shown to help on LoCoMo. The newer code hands the model about twice as much text there. A second, pre-registered variant (both speakers stored as the user) produced empty memory, because saved conversations need an assistant turn; it is published as a failed run. Receipt.

Public Long-Conversation Suites

HivemindOS also runs the public LoCoMo, LongMemEval, and BEAM suites through the product’s conversation archive, full-vault index, and final recall ranking. These are complete Top-50 retrieval runs, not hand-selected pilots.

Evidence-session recall asks whether the annotated source conversation appeared among the retrieved memories. It is a retrieval metric, not final-answer accuracy. BEAM does not publish equivalent evidence-session labels in this adapter, so its table rows report non-empty retrieval coverage and hit count without implying that every result is relevant.

The combined annotated-evidence check contains 2,036 eligible LoCoMo and LongMemEval questions. HivemindOS retrieved the labeled source session for 1,931 questions at Top-10 (94.84%) and 2,032 at Top-50 (99.80%).

Judge-Scored Answers

The answer phase used gpt-5.4-mini for both the answering model and judge through ChatGPT OAuth, with the same Top-50 recalled context produced by HivemindOS. Every question in all four suites completed.

LoCoMo and LongMemEval use binary judge outcomes, so score and pass rate match. BEAM averages compliance across each question’s rubric nuggets; its pass rate counts questions scoring at least 0.5. The adapter reports BEAM rubric compliance but does not yet publish the separate event-ordering Kendall tau-b metric.

The category results show where the current system is strongest and where it still needs work:

These model-judge results are a reproducible HivemindOS snapshot, not a direct competitor comparison. They predate the conversation-exchange recall used in the September head to head above, and use a different reader, judge and cutoff, so the two LongMemEval numbers are not comparable with each other. Scores produced with a different answering model, judge, prompt revision, or retrieval cutoff—including GPT-5/Top-200 configurations—are not apples-to-apples. ChatGPT OAuth did not expose token-usage counters, so the benchmark publishes latency and completion counts but does not estimate tokens or model cost.

The long-conversation run also validated the cache optimization that made BEAM 10M fit within the normal Node heap. Re-running 2,440 LoCoMo, BEAM 1M, and BEAM 10M questions preserved the same ranked context after excluding regenerated file timestamps, while median retrieval improved:

The public machine-readable result, including the full category breakdown and run metadata, is available as JSON.

Methodology is pinned to mem0ai/memory-benchmarks, snap-research/locomo, xiaowu0162/LongMemEval, and mohammadtavakoli78/BEAM. HivemindOS currently exposes at most 50 recalled memories, so this page does not compare its Top-50 results against another system’s Top-200 score.

Recall Quality Across Messy Questions

The live matrix did not only search exact titles. It generated sparse questions, added conversational noise, introduced typos, expressed natural intent, and applied type, project, and tag filters.

Across the generated retrieval portion, the aggregate was 90% Top-1, 98% Top-3, and 0.94 MRR. The main measured weakness is typo handling: it remains strong enough for 91% Top-3, but its 81% Top-1 result leaves clear room for improvement.

The same run added three behavior groups outside those generated variants:

• Unsupported compound questions abstained in 10/10 cases instead of injecting weakly related memory.
• Operational routing passed 40/40 checks: legacy receipts stayed out of normal durable recall and remained available when explicitly requested.
• Current, historical, and as-of chain queries reached 96% Top-1 and 100% Top-3.

The 1,000 calls completed at 86.38 sequential queries per second. Generated retrieval latency measured 12.27ms p50 and 16.28ms p95 during this run.

Real API Performance

The API benchmark goes through the authenticated dashboard HTTP route instead of calling the scorer directly. Its isolated temporary vault covers entity recall, aliases, current canonical heads, historical memory, retrieval-usage signals, explicit operational lookup, default operational isolation, and unsupported-query abstention.

Across 50 measured repetitions of eight cases—400 requests total after two warmup passes—the route achieved:

This is a same-machine product-path benchmark, not a hosted-service latency promise. Hardware, vault shape, filesystem load, sync activity, and development versus production builds can change absolute timing.

Hosted SuperAgent API memory

Cloud Superbrain was tested separately through the authenticated public SuperAgent API on August 27, 2026. Three fresh isolated accounts each received the same typed-memory behavior corpus and were measured for three repetitions after warmup.

The three corresponding fresh local behavior runs passed 8/8 cases and 6/6 ranked Top-1 cases. Their median trial p50 was 7.85ms and their median trial p95 was 11.63ms. On this corpus, hosted recall matched the tested local behavior but did not match local latency.

A separate scale diagnostic added 48 adversarial archive memories that reused coordinator, proof, and release terminology. The hosted corpus contained 57 memories and the local corpus contained 55 because the hosted test also carried two knowledge nodes for semantic-answer grading. Both paths kept every published behavior case correct. The single hosted run measured 431.67ms lexical p50, 1,674.67ms semantic p50, and 2,594.36ms answer p50; the repetition-matched local lexical run measured 14.48ms p50. This one scaled run is evidence that the distractors did not displace the expected memories, not a statistically powered scale or speed comparison.

The hosted quality cases cover entity and alias lookup, canonical evolution heads, explicit history, usage-signal ranking, operational-memory isolation, semantic paraphrases, grounded citations, and negative abstention. They do not upload or reproduce a private Obsidian vault, and they do not establish universal recall parity across arbitrary customer data. Hosted timings include internet transit, account isolation, metering, durable storage, and any paid semantic or answer work; local timings stay on the machine.

Past conversations on LongMemEval_S

On September 23, 2026, the hosted memory gained the shared brain’s recall of past conversations: the best-matching exchanges under each conversation’s date, ordered by the Turbo decision model for Memory Answer. It was measured before and after on LongMemEval_S through the hosted memory’s own search, recall, and answer code. Each question’s 50 or so past chats were saved to a fresh account as transcripts, and semantic recall was left off.

• Question by question, the new Memory Answer was right on 98 questions the old one missed, and the old one on 1.
• Retrieval and reader both mattered. With the old answer model, the new recall alone lifted correct answers from 16.7% to 45.3%. Given the evidence, that small model often replied that it had no information. Memory Answer now uses a larger model at the same price per token, which reached 81.9%. It reasons before answering, which added about three seconds to the reader’s median time in this test.
• The decision model helped in a smaller space. Ordered by it, the evidence fit in 12,000 characters for 81.1% of questions, against 74.8% in keyword order at the same size, and the reader did better with those 12,000 characters than with 24,000 in keyword order.

Limitations: answers were graded with LongMemEval’s official judge instructions but by gpt-4o-mini rather than its gpt-4o, so compare these rows with each other rather than with the head-to-head above. With semantic recall on, as in paid plans, recall can only gain candidates. LongMemEval measures memory of past conversations; typed memories are covered by the hosted behavior cases above.

Typed Memory At Scale

The reproducible scale benchmark creates isolated indexes at 100, 500, and 1,500 memories. Each size receives 200 exact, natural, sparse, and noisy queries with local embeddings disabled, so the result measures the local lexical and typed-memory path.

The synthetic corpus is intentionally controlled. Its perfect relevance shows that growing the index did not displace uniquely identifiable memories; it does not replace the noisier live-corpus benchmark above.

Searching The Whole Vault

Typed Agent Memory is the fast path for durable facts, preferences, instructions, decisions, and learnings. When that distilled layer is not enough, HivemindOS can search normal vault markdown through a generated lexical index and load only the ranked source notes.

The live eight-case benchmark covered project decisions, operations policy, control-plane documentation, shared skills, brain-service notes, secure references, imported sources, and intake notes.

The direct indexed stage was 15.39× faster without changing the expected first result in any case. The final runtime is deliberately heavier because it performs the product’s routing, candidate loading, and final ranking rather than returning raw index rows.

A separate large reference-vault test indexed 25,995 eligible notes from a 28,549-file vault in about 9.2 seconds and produced a 70.6 MB local JSONL index. In that five-query snapshot, median direct search improved from 2,285ms to 118ms, or 19.4×, with identical Top-1 results.

Less Prompt Baggage

Memory only saves money when agents retrieve focused context instead of repeatedly loading broad files and histories. The live token benchmark sent the same task to the same model with a broad baseline context pack and a targeted Hive context pack.

These are provider-reported usage counters from live OpenRouter calls using openai/gpt-4.1-mini. The broad baseline was intentionally truncated at 180,000 sent characters in both scenarios; the targeted Hive packs were not truncated. These tests measure context and token use, not a provider invoice or a statistically powered final-answer-quality comparison. The complete prompts, outputs, and methodology are documented in Token And Cost Savings.

The current deterministic local context-budget benchmark adds four scenarios—brain recall, software build, chatbot build, and workflow reuse. It estimated 2,906,702 baseline context tokens versus 22,764 targeted Hive context tokens, a 99.2% reduction. That is a reproducible text-budget estimate, not provider billing.

Cleaner Current Truth

Two architecture changes address different sources of memory pollution:

1. Routine receipts, retries, and completions go to a bounded operational journal rather than durable Agent Memory.
2. Durable records use canonical memory keys, while reviewed evolution preserves previous versions as history under one current head.

The live migration corpus made the effect measurable:

The current report-only health snapshot contains 101 active canonical heads and 21 superseded historical records. Forty-seven legacy operational records remain preserved for explicit access, but they no longer enter default durable recall. This before/after result combines architectural separation with reviewed consolidation; it should not be presented as an automatic cleanup percentage for every vault.

Pattern Mining, With A Gate

Pattern mining is useful only if repeated noise does not become permanent advice. The labeled 47-event fixture includes three positives—a recurring provider failure, a reusable content workflow, and a stable weekly routine—plus adversarial negatives such as retries of one task, test/E2E activity, insufficient support, unknown outcomes, and unrelated one-offs.

The miner returned exactly the three labeled proposals:

This passes the fixture gate, but it is not production precision. Pattern mining remains a dry run by default, and explicit enqueueing creates review proposals rather than memories, skills, or scheduled jobs. Broad autonomous promotion stays disabled until real reviewed events provide enough evidence.

Marketing-Safe Claims

These statements match the measured evidence:

In a 1,000-query live memory benchmark, HivemindOS ranked the expected memory first 90% of the time and in the top three 98% of the time. Exact current-title recall was 100%.

The indexed local memory path stayed under 30ms median at 1,500 synthetic memories while preserving perfect Top-1 across 200 exact, natural, sparse, and noisy queries.

Targeted Hive context reduced provider-reported total tokens by 87.5% on a complex build task and 99.2% on a focused memory-recall task.

In an eight-case live full-vault benchmark, the direct lexical index was 15.39× faster than the previous search path with the same expected first result in every case.

Operational receipts stayed out of default durable recall in 40/40 checks, while reviewed migration moved canonical conflicts and duplicate pressure to zero in the measured live corpus.

Across 2,036 annotated LoCoMo and LongMemEval questions, HivemindOS retrieved the labeled evidence session in the Top-50 99.80% of the time and in the Top-10 94.84% of the time.

Full local retrieval completed without an empty result for all 900 BEAM 1M and 10M questions; median retrieval was 121.9ms at 1M and 1.05s at 10M.

With GPT-5.4 Mini answering and judging the complete Top-50 runs through ChatGPT OAuth, HivemindOS scored 76.62% on LoCoMo and 53.40% on LongMemEval.

On the complete BEAM runs, HivemindOS averaged 41.12% rubric compliance at 1M context and 37.04% at 10M context, with 44.29% and 43.00% of questions meeting the 0.5 pass threshold.

On LongMemEval Oracle, all 500 questions, with LongMemEval’s official judge grading every answer, the HivemindOS Superbrain answered 96.6% correctly with Claude Opus 4.6 reading, against 95.3% for Sibyl-Memory’s Opus 4.6 run, and 95.1% with Claude Sonnet 4.5, against 92.8% for Sibyl-Memory’s Sonnet 4.5 plugin (excluding preference questions).

The Sibyl-Memory comparison is measured. Quote it with its setup: LongMemEval Oracle, all 500 questions, the same reading model on both sides, the official gpt-4o-2024-08-06 judge for both, and Sibyl-Memory’s own published answers. Say HivemindOS scored higher; do not say the difference is statistically significant, because it is not (p = 0.33 with Opus 4.6). The LongMemEval_S comparison is quoted the same way: all 500 S questions, Claude Opus 4.6 on both sides, Sibyl-Memory’s published architecture run by us with its public runner, 94.3% against 92.1% (p = 0.16), with HivemindOS reading about a ninth of the tokens; it is not a comparison with Sibyl-Memory’s plugin product. Do not turn these results into an unmeasured competitor comparison, a hosted latency SLA, a universal token-savings guarantee, or a claim that pattern proposals are production-perfect. The benchmarks deliberately publish their corpus, route, sample size, and limitations so the claims can stay useful as the product evolves.

Reproduce The Benchmarks

The local and fixture benchmarks do not require a paid model:

pnpm benchmark:agent-memory-live-recall -- --vault <vault> --calls 1000
pnpm benchmark:agent-memory-scale -- --calls 200
pnpm benchmark:agent-memory-upgrade -- --base-url <dashboard-url> --iterations 50 --warmup 2
pnpm benchmark:shared-brain-search -- --vault <vault> --limit 8
pnpm benchmark:agent-memory-pattern-mining
pnpm benchmark:context-savings

The authenticated API benchmark expects a running HivemindOS dashboard. The full-vault and live-memory benchmarks are read only. The scale, API behavior, and pattern benchmarks use isolated synthetic fixtures.

The standard public suites use the development-only harness documented under benchmarks/memory/. Datasets, temporary vaults, model clients, OAuth bridges, checkpoints, and raw outputs remain outside the production application and are not shipped to users.

The LongMemEval Oracle comparison uses the public longmemeval_oracle.json from the LongMemEval Hugging Face dataset. Building each question’s Superbrain needs no model; the reader and the judge call OpenRouter and cost about $20 for all 500 questions:

OPENROUTER_API_KEY=<key> node scripts/benchmark-longmemeval-arms.mjs retrieve --dataset <longmemeval_oracle.json> --out <dir> --arms product-answer-jev
OPENROUTER_API_KEY=<key> node scripts/benchmark-longmemeval-arms.mjs answer --out <dir> --arms product-answer-jev --reader anthropic/claude-sonnet-4.5 --judge openai/gpt-4o-2024-08-06 --reader-max-tokens 1500 --budget-usd 25
node scripts/benchmark-longmemeval-arms.mjs summarize --out <dir>

The published answers and a script that recomputes the scores are in hivemindos-memory-benchmarks.

The live token benchmark calls the selected provider and may incur provider charges:

./scripts/hive-env-run -- pnpm benchmark:e2e-token-savings -- --scenario brain-recall
./scripts/hive-env-run -- pnpm benchmark:e2e-token-savings -- --scenario chatbot-build

For the underlying memory architecture, see Brain, Vault, And Skills and Brain Services.