← Back to README

Benchmarks

Historical result tables in this document are not release evidence unless they include a reproducible command, fixture/version, commit, and environment. Use the offline gate below as the current repository verification contract.

Measured results: pre-change vs post-change (fixed judge)

Evidence classification: Measured, locally reproduced answer-judge scores published as eval-comparison/v1 artifacts under bench/reports/. Evidence identity: Every row below is read from a named report file that records the product commit, judge configuration, run timestamp, and question count. Evaluator classification: Scores are fixed-judge answer acceptability (fixed_judge_accuracy), not labelled stable-ID/span retrieval relevance. Comparability: The pre-change and post-change rows are comparable only with each other: same judge, same datasets, same question limit, same harness generation. No third-party system score appears on this page.

Evaluation protocol

Parameter	Value
Judge	qwen2.5:7b-instruct served by the local Ollama runtime
Configuration	OLLAMA_ENDPOINT, OLLAMA_JUDGE_MODEL, temperature=0, seed=42, format=json (protocol fixed-judge/ollama/v1)
Question limit	100 questions per benchmark per run (limit=100), same slice for every run compared here
Datasets	LOCOMO (CC BY-NC 4.0, five question types) and LongMemEval (ability slices as published by the runner: IE, MR)
Metric	fixed_judge_accuracy = accepted answers / evaluated questions, reported per slice and overall

Two harness properties make these numbers publishable at all:

• No-fabrication contract. Runners fail closed with common.BlockedError (exit code 2) when the judge endpoint is unreachable or a trustworthy score cannot be produced. A failed run publishes no score instead of a fallback or guessed score.
• Baseline chaining. common.EvalBaselineFromReport loads a prior report into Config.Baseline; each report then records baseline_status (not_recorded on first publication, recorded thereafter) and per-slice before / after / delta against min_delta gates. Chaining only compares against the report it is fed.

Measured comparison

Metric	Pre-change (product commit 3bf0f411)	Post-change	Δ absolute	Δ relative
LOCOMO overall	0.26 (26/100)	0.56 (56/100)	+0.30	+115%
LOCOMO single-hop	0.25	0.625	+0.375	+150%
LOCOMO multi-hop	0.081	0.324	+0.243	+300%
LOCOMO temporal	0.692	0.769	+0.077	—
LOCOMO open-domain	0.333	0.778	+0.445	—
LongMemEval overall	0.06 (6/100)	0.08 (8/100)	+0.02	—
LongMemEval IE	0.057	0.086	+0.029	—
LongMemEval MR	0.067	0.067	0.000	stable

Artifacts: pre-change bench/reports/prechange-baseline-locomo.json and bench/reports/prechange-baseline-longmemeval.json; post-change bench/reports/first-eval-locomo.json and bench/reports/first-eval-longmemeval.json, re-measured by bench/reports/second-eval-locomo.json and bench/reports/second-eval-longmemeval.json; current rows from bench/reports/third-eval-locomo.json (the LongMemEval columns are unchanged in bench/reports/third-eval-longmemeval.json).

The relative columns are computed from the published values: 0.30 / 0.26 = +115%, 0.375 / 0.25 = +150%, and 0.243 / 0.081 = +300%. Other rows are reported as absolute deltas.

Provenance of the pre-change baseline

The pre-change rows were measured against product commit 3bf0f411, the last main commit before the 2026-10-02 retrieval change wave, in a detached disposable worktree outside the main checkout.

• The eval-comparison/v1 harness postdates 3bf0f411, so the current bench/ harness plus go.mod / go.sum were copied into that worktree and executed against the pre-change product tree. The committed bench/ at 3bf0f411 cannot publish eval-comparison/v1, so reuse was mandatory.
• Compiling the reused harness against the pre-change API surface required exactly three adaptations, in the worktree copies only: drop the retrieval.AdaptiveSearchOptions.FusionScores literal in bench/locomo/runner.go and bench/longmemeval/runner.go, and drop the opts.QueryVector assignment in bench/locomo/runner.go. Those options do not exist at 3bf0f411. No main-checkout source file was modified.
• The harness-reuse decision and the three adaptations are recorded verbatim in the limitations array of both pre-change report files.

Reproducibility

• Post-change run 2 reproduced run 1 exactly: the canonical score payload (metric, overall, every slice, total_questions, correct) is byte-identical between first-eval-*.json and second-eval-*.json. Check it with:

diff <(jq -S .score bench/reports/first-eval-locomo.json) <(jq -S .score bench/reports/second-eval-locomo.json)
diff <(jq -S .score bench/reports/first-eval-longmemeval.json) <(jq -S .score bench/reports/second-eval-longmemeval.json)

• temperature=0 and seed=42 make the judge deterministic for a fixed input; determinism is not the same as accuracy, and the repetition above covers one machine and one judge only.
• Why the harness deltas read 0.000. The chained baseline inside the first- and second-pass post-change reports was first-eval, which already postdates every retrieval change, so its per-task rows compare 0.35 with 0.35 and report delta: 0. That chain cannot express pre-change lift; the comparison table above is computed across the separately published pre-change and post-change artifacts instead. The current third-eval-locomo.json chains against second-eval, so its per-task rows do report movement (0.35 → 0.56, delta: 0.21 on the overall slice).

Limitations

1. No vector row at baseline. The pre-change baseline and the embeddings-off post-change runs carry no rag-t07 vector row because no embedding model was available when the baseline ran. Embeddings-enabled runs are recorded separately in bench/reports/eval-embeddings-locomo.json (and bench/reports/eval-embeddings-longmemeval.json when published), with the embedding model in provenance; they have no pre-change counterpart and are therefore not part of the before/after table.
2. No competitor numbers. Scores from other systems are deliberately absent. Different judges, prompts, model backbones, question subsets, and metrics are not comparable, so importing them here would produce a false ranking.
3. Single local judge. One judge model on one local runtime: a single pre-change run versus three post-change passes (the first two byte-identical), n=100 per benchmark, no multi-seed or cross-machine variance analysis and no confidence intervals. Treat slice-level movements, especially multi-hop and open-domain, as indicative rather than settled.
4. Absolute level. The lift from 0.26 to 0.56 on LOCOMO (+0.30 absolute, +115% relative) and from 0.06 to 0.08 on LongMemEval is measured, but LOCOMO still misses 44 of 100 questions; in the current run open-domain (0.778) and temporal (0.769) are the strongest slices and multi-hop (0.324) the weakest.
5. Environment. The reports record the judge endpoint, model, seed, limit, and datasets, but no hardware profile; no latency, throughput, or resource claim is made from these runs.
6. Release claims. These tables stay answer-judge evidence under the release-claim rules above: they are not stable-ID/span retrieval relevance evidence and do not by themselves satisfy a release gate.

What the baseline proves

The committed baseline contracts prove that Cortex can validate a versioned, labelled retrieval corpus; preserve per-query ranked stable IDs; calculate retrieval-quality, correctness, latency, throughput, and resource fields; compare independent runs; and preregister immutable release gates. The corpus contract is cortex.retrieval-corpus/v1, the report contract is retrieval-evidence-report/v1, and each run must record its exact corpus, protocol, profile, build, and hardware versions. Universal correctness gates require zero isolation violations and an exact match with the authoritative eligible stable-ID set.

This is a methodology and current-path baseline, not evidence that a future retrieval candidate is better. A result supports a Cortex claim only when its report and referenced corpus are committed, complete, independently reproduced, and approved under a gate registered before candidate results are observed.

What the baseline does not prove

• It does not establish a new quality, latency, throughput, CPU, RSS, storage, or index threshold. Numeric thresholds remain unset until representative baseline runs, variance analysis, corpus/hardware approval, and reviewer sign-off exist.
• It does not turn external LOCOMO, DMR, or LongMemEval results into Cortex-reproduced performance.
• It does not make answer-token F1, ROUGE-L, or judge correctness equivalent to labelled stable-ID retrieval relevance. Those metrics evaluate answer text or answer acceptability; release retrieval evidence requires relevant episode or fact IDs, or immutable evidence spans.
• It does not prove portability, vector-provider parity, or production-scale performance. Those claims need their own versioned build and evidence matrix.

The legacy suite tables and commands below are preserved for continuity. Unless a row links a complete Cortex evidence report, treat it as historical or exploratory evidence, not as a release gate or a cross-system comparison.

Versioned retrieval protocol

Evidence field	Required record
Corpus	Schema and corpus version; immutable query IDs; profile/query classes; hard negatives; no-answer, temporal, isolation, privacy, classification, and lifecycle labels
Relevance	Relevant episode/fact stable IDs or half-open byte spans; missing labels make release evidence incomplete
Execution	Protocol and profile ID/version, build commit and dirty state, provider/model where used, and traceable current ranked outputs
Environment	Named hardware profile, OS, architecture, CPU, memory, and explicit measurement units
Report	Per-query and per-class results, metric definitions, limitations, and independently retained run IDs

Splits and evaluator

Splits are assigned by immutable query ID under a versioned strategy. The development/calibration split may select protocol details and gates; held-out decision evidence must not be reused after candidate results are observed. The evaluator is the committed Go implementation in bench/common: it validates stable-ID/span labels and computes deterministic aggregates. Optional answer judges belong only to the legacy answer-evaluation suites and must disclose the judge model, endpoint class, prompt/protocol version, and failure/abstention policy. They do not supply missing retrieval labels.

Metrics and uncertainty

Metric	Definition and interpretation
Recall@k	Fraction of labelled relevant stable IDs returned in the first k results
MRR	Reciprocal rank of the first labelled relevant stable ID
nDCG	Discounted cumulative gain normalized by the ideal ordering; supports graded relevance
Evidence recall	Fraction of labelled episode/fact IDs or evidence spans covered
No-answer/abstention	Whether the profile correctly returns no supported answer for labelled no-answer queries
Isolation violations	Count of unauthorized or otherwise ineligible returned IDs; any non-zero value blocks release
Filter correctness	Exact equality between returned eligibility and the authoritative filtered stable-ID set
Latency/throughput	p50/p95/p99 duration and completed queries per second, with units and sample size
Resources	CPU seconds, peak RSS bytes, corpus storage bytes, and retrieval-index bytes

Reports disclose sample size, per-class results, the uncertainty method and confidence level, dispersion/confidence intervals where applicable, and outliers. Independent runs are compared only when corpus, build, hardware, and protocol identities match. Deterministic fields must match exactly; measured fields use a preregistered tolerance derived from baseline variance, never a post-result tolerance.

Hardware and resources

Record a reproducible hardware envelope rather than a machine nickname: profile ID, OS/version, architecture, CPU model/count, available memory, and relevant provider/model versions. Reports must state whether CPU time, RSS, storage, index size, latency, and throughput were measured. Zero placeholders from a legacy adapter mean unmeasured, not free or instantaneous.

Release gates

The gate registry is versioned and immutable. Correctness gates (zero isolation leakage and exact filter eligibility) are universal and non-relaxable. Quality, latency, throughput, and resource gates are profile- and query-class-specific. They record metric direction, sample size, corpus/hardware versions, approval, and blocking policy before a candidate run. Changing a gate after results requires a new protocol version, written rationale, approval, and fresh held-out evaluation that does not reuse the decision evidence.

External and Cortex-reproduced evidence

Suite/evidence	Classification	Allowed claim
Versioned Cortex corpus + complete evidence report	Cortex-reproduced only when run by the documented protocol on the disclosed build/hardware	Retrieval and resource claims limited to that exact profile, corpus, protocol, and environment
LOCOMO adapter	External/legacy answer evidence; CC BY-NC 4.0 dataset	Preserve source, split, categories, upstream evidence locators, F1, and optional judge output; do not claim stable-ID retrieval relevance without Cortex labels
DMR / MSC-Self-Instruct adapter	External/legacy answer evidence; Apache-2.0 dataset	Preserve F1/ROUGE-L and optional judge semantics; the source answers do not supply Cortex relevant IDs/spans
LongMemEval adapter	External/legacy answer evidence; upstream distribution carries the applicable licence notice	Preserve ability categories, temporal locators, abstention, F1, and judge behavior; temporal locators are not Cortex relevance labels

The adapters are intentionally reporting-only and preserve the existing runners and scoring behavior. cortex_reproduction: false, incomplete retrieval labels, or a false release-eligibility flag blocks wording that presents an adapted result as Cortex retrieval performance.

Reproduce the baseline

Run from the repository root on a clean committed build. The first command is the executable documentation contract; the second validates deterministic baseline contracts and all preserved benchmark adapters without requiring a dataset download, embeddings, a judge, or an external service.

go test -v -count=1 ./bench -run TestRetrievalBaselineDocumentationContract
go test -v -count=1 ./bench/...

Dataset-scale, live-judge, embedding-provider, and performance runs are separate opt-in evidence. Record the exact command, dataset artifact/version and split, profile/provider/model versions, build commit/dirty state, hardware profile, report IDs, and output paths. Do not compare reports whose identity fields differ without a separately approved normalization protocol.

Licences and limitations

• Cortex code and generated Cortex evidence remain governed by this repository's licence; benchmark datasets retain their upstream licences and attribution.
• LOCOMO is documented as CC BY-NC 4.0, MSC-Self-Instruct as Apache-2.0, and LongMemEval with the licence notice shipped by the exact upstream distribution. Verify terms against the downloaded version before redistribution.
• Dataset downloads, embedding services, and live judges are not required for deterministic contract validation and may impose separate terms, cost, network, nondeterminism, or privacy constraints.
• The committed miniature corpus validates contracts and adversarial isolation; it is not representative evidence for production scale or domain coverage.
• Legacy suite scores remain useful for continuity, but their answer-oriented labels, evaluators, and splits are not interchangeable with the Cortex stable-ID/span relevance protocol.

The legacy Cortex suites below exercise three standard memory benchmarks. Their results are reproducible only when the exact dataset, split, evaluator, build, profile/provider, hardware, and protocol are disclosed; the tables alone do not constitute Cortex release evidence.

Results Summary

Evidence classification: Historical, unverified repository snapshot. Evidence identity: The retained rows do not identify a result artifact, run ID, build commit/dirty state, exact dataset artifact and split, hardware profile, or uncertainty. They are not reproducible Cortex release evidence. Evaluator classification: The numeric scores use legacy answer-token F1 (and, where stated, ROUGE-L or an optional answer judge), not labelled stable-ID/span retrieval relevance. Comparability: Values are preserved for historical continuity only. Do not use them for cross-provider, cross-system, performance, or causal conclusions.

LOCOMO (Long-Term Conversational Memory)

Dataset: 1,986 questions across 10 conversations, 5 question types. Source: snap-research/locomo (ACL 2024)

Mode	single-hop	multi-hop	temporal	Legacy ratio shown in snapshot
FTS5 only (baseline)	0.002	0.001	0.000	—
FTS5 + Ollama (nomic-embed-text)	0.025	0.016	0.026	12-16x
FTS5 + OpenAI (text-embedding-3-small)	0.026	0.016	0.037	13-37x

Note: These scores are raw answer-token F1 between retrieved context and gold answers. They do not measure answer-generation accuracy, do not contain Cortex stable-ID/span relevance labels, and do not substantiate an external system's accuracy. The ratios are retained data from the legacy snapshot, not verified retrieval-improvement claims.

DMR (Deep Memory Retrieval)

Dataset: 500 multi-session conversations from MSC-Self-Instruct. Source: MemGPT/MSC-Self-Instruct (arXiv:2310.08560)

Mode	Avg Score (F1 + ROUGE-L)
FTS5 only	0.000
FTS5 + Ollama vectors	Pending full run
FTS5 + OpenAI vectors	Pending full run

Embedding Provider Comparison

Tested on LOCOMO (50 questions subset):

Provider	Model	Dimensions	single-hop	multi-hop	temporal	Time	Cost
Ollama	nomic-embed-text	768	0.025	0.016	0.026	12 min	$0
OpenAI	text-embedding-3-small	1536	0.026	0.016	0.037	21 min	~$0.02
None (FTS5)	—	—	0.002	0.001	0.000	4 min	$0

Interpretation limit: The table preserves answer-token scores, elapsed times, and reported costs from an unidentified 50-question legacy run. Without the required evidence identity and repeated-run uncertainty, these rows do not support provider equivalence, temporal-reasoning conclusions, embedding-dimension causality, provider speed or network-cause claims, retrieval multipliers, or an absolute statement about FTS5 temporal capability.

Methodology

Pipeline

For each benchmark question:

1. Ingest — Parse dataset conversations into Cortex sessions + observations (stored in in-memory SQLite)
2. Embed — When vector search is enabled, generate embeddings for each observation via Ollama or OpenAI
3. Search — Run FTS5 keyword search + optional vector cosine similarity search
4. Fuse — Combine FTS5 and vector results using Reciprocal Rank Fusion (k=60)
5. Score — Compare top-5 retrieved results against gold answer using F1 token overlap
6. Judge — Optionally evaluate answer acceptability with the local Ollama runtime

Scoring

• F1 Token Overlap: Tokenize prediction and reference, compute precision/recall/F1 on token sets
• ROUGE-L: Longest Common Subsequence F1 between prediction and reference
• Ollama answer judge: The committed runtime is Ollama-only and defaults to qwen2.5:7b-instruct, temperature=0, and seed=42. Configure it with OLLAMA_ENDPOINT and OLLAMA_JUDGE_MODEL. This optional score is not retrieval evidence and never replaces stable-ID or evidence-span relevance labels.
• Correct threshold: F1 >= 0.3 for LOCOMO, (F1 + ROUGE-L) / 2 >= 0.3 for DMR

Reciprocal Rank Fusion (RRF)

When both FTS5 and vector results are available, they are combined using RRF:

RRF_score(doc) = Σ 1/(k + rank_i)  where k=60

Each ranking system (FTS5 by BM25, vectors by cosine similarity) contributes independently. Documents appearing in both rankings get a combined score higher than either alone.

Limitations

1. Retrieval-only evaluation — Cortex retrieves relevant memories but does not generate answers. End-to-end evaluation requires an LLM layer on top.
2. Sequential embedding — Each observation is embedded one at a time. The effect of batch embedding on runtime was not evaluated by this snapshot.
3. No graph boost — Knowledge graph neighbor expansion is not used in these benchmarks; its effect on multi-hop scores was not evaluated.
4. F1 is conservative — Token overlap penalizes retrieval systems that return contextually correct but lexically different content.

Running Benchmarks

Prerequisites

# Build with vector search support
go build -tags cortex_vectors ./cmd/cortex

# For local embeddings (recommended)
# Install Ollama: https://ollama.com
ollama pull nomic-embed-text

# Download datasets
cd bench
chmod +x download.sh
./download.sh

Run

# LOCOMO — FTS5 only (fast, no dependencies)
go test ./bench/locomo/ -run TestRunFullDataset -v -timeout 30m

# LOCOMO — with Ollama embeddings (requires Ollama running)
go test -tags cortex_vectors ./bench/locomo/ -run TestRunWithOllamaEmbeddings -v -timeout 30m

# LOCOMO — with OpenAI embeddings (requires API key)
CORTEX_EMBEDDING_API_KEY=sk-... go test -tags cortex_vectors ./bench/locomo/ -run TestRunWithOpenAIEmbeddings -v -timeout 30m

# DMR
go test ./bench/dmr/ -run TestRunFullDataset -v -timeout 30m

# All benchmarks (short mode — uses subsets)
go test ./bench/... -short -v -timeout 10m

Results

Results are saved as JSON in bench/results/:

ls bench/results/
# locomo.json              — Full LOCOMO (FTS5 only)
# locomo_ollama_50.json    — LOCOMO subset with Ollama
# locomo_openai_50.json    — LOCOMO subset with OpenAI
# dmr.json                 — Full DMR

SOTA RAG & Graph Retrieval Engines (2025/2026)

Cortex integrates a 4-tier state-of-the-art retrieval and graph propagation engine in 100% pure Go (Zero-CGO):

Engine / Algorithm	Origin & Reference	Latency (RAM)	Primary Use Case
HippoRAG (PPR)	NeurIPS 2024 (arXiv:2405.14831)	$< 2\text{ms}$	Multi-hop relational propagation across cognitive & AST knowledge graph edges
Adaptive-RAG	NAACL 2024 (arXiv:2403.14403)	$< 0.1\text{ms}$	Dynamic 4-tier query complexity routing (Direct, Hybrid, Multi-Hop, Architectural)
LightRAG	EMNLP 2024 (arXiv:2410.05779)	$< 1\text{ms}$	Hierarchical community modularity summaries ($C = \frac{2 E_{\text{int}}}{N(N-1)}$)
Corrective RAG (CRAG)	arXiv:2401.15884	$< 0.1\text{ms}$	Confidence scoring (high, medium, low) and hallucination noise suppression
ColBERT MaxSim	SIGIR / arXiv:2004.12832	$< 0.5\text{ms}$	Multi-token late-interaction re-ranking: $\text{MaxSim}(Q, D) = \sum_{q} \max_{d} (q \cdot d)$

---

Competitive Architecture Matrix

Capability	Cortex Server	MemPalace	Mem0	Zep
Runtime Architecture	Go Single Binary (Zero-CGO)	Python Flat-file	Python Service	Python / LangChain
Graph Propagation	HippoRAG PPR in Go ($< 2\text{ms}$)	No Graph	Basic NetworkX	Graphiti (Neo4j dependent)
Retrieval Routing	Adaptive-RAG (4 Tiers)	Keyword Grep	Fixed Vector	Vector Only
Noise Gating	CRAG Confidence Gate	None	None	LLM Reranker (Slow)
Multi-Tenancy & Auth	Postgres 16 RLS + Local SQLite	Single User	Cloud Service	SaaS Dependent
Realtime Monitoring	Next.js Control Room + Sigma.js	None	Web Dashboard	Cloud UI

---

Dataset Licenses

Dataset	License	Source
LOCOMO	CC BY-NC 4.0	snap-research/locomo
MSC-Self-Instruct (DMR)	Apache 2.0	MemGPT/MSC-Self-Instruct
LongMemEval	MIT	xiaowu0162/LongMemEval

References

• HippoRAG: "HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models" (NeurIPS 2024, arXiv:2405.14831)
• Adaptive-RAG: "Adaptive-RAG: Determining When to Retrieve via Dynamic Query Complexity" (NAACL 2024, arXiv:2403.14403)
• LightRAG: "LightRAG: Simple and Fast Knowledge Graph RAG" (EMNLP 2024, arXiv:2410.05779)
• CRAG: "Corrective Retrieval Augmented Generation" (arXiv:2401.15884)
• ColBERT: "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT" (SIGIR 2020, arXiv:2004.12832)
• LOCOMO: "Evaluating Very Long-Term Conversational Memory of LLM Agents" (ACL 2024, arXiv:2402.17753)
• DMR/MemGPT: "MemGPT: Towards LLMs as Operating Systems" (arXiv:2310.08560)
• LongMemEval: "Benchmarking Chat Assistants on Long-Term Interactive Memory" (ICLR 2025, arXiv:2410.10813)
• Engram: "Effective, Lightweight Memory Orchestration for Conversational Agents" (arXiv:2511.12960)
• Zep/Graphiti: "A Temporal Knowledge Graph Architecture for Agent Memory" (arXiv:2501.13956)