nishi code wiki / research / agent memory and Graph-RAG
Agent memory and Graph-RAG: the honest state
Researched 2026-07-30 (two deep research agents, ~340k tokens, primary sources only); transcribed to this library 2026-08-01. Corpus slug: reference-agent-memory-graphrag-sota-2026-07-30. Labels: RESEARCH paper/preprint · SHIPPING production · OURS measured in this ecosystem.
Question this brief answers. Should Nishi build cognee-class knowledge-graph memory? Verdict: NO as a first move. On independent evaluation, knowledge-graph memory does not beat a well-tuned embedder + cross-encoder reranker in 2026; graphs earn their cost on exactly five named query classes. The answer changed the build order: embedder → reranker → agent grep-tools → graph LAST, chosen by workload.
Forks off — the lineage
| parent | what this brief inherits |
|---|---|
reference-code-quality-eval-sota-2026-07-29 (corpus, STUB) | The self-authored-judge failure mode this field exhibits; matched-rate judging discipline. |
project-nishi-search-wand-impact-postings-2026-07-25 (corpus) | The search lane's own measured verdict: COVERAGE, not ranking, is our retrieval gap — consistent with rung 1 here. |
Fed: the Nishi retrieval build order (embedder/reranker rungs already held: nx_recall_dense, nx_recall_fuse); the decision NOT to adopt cognee; the debt-intelligence dense-arm experiments (which independently confirmed claim 7's pattern on our own ledger).
Claims — sourced, dated, labelled
| # | claim | source · date | label |
|---|---|---|---|
| 1 | Held-embedder baselines erase the graph advantage: Mem0 vs plain RAG = −1.2pp (p=1.0) at ~50× write cost (MemDelta); KG²RAG's graph edge on HotpotQA-distractor = +1.1 F1; plain BM25 60.5 vs Mem0 32.6 (MemoryAgentBench); Letta's own test: agent grep over files 74.0% vs Mem0 68.5%. | MemDelta · KG²RAG · MemoryAgentBench · Letta blog · 2025–26 | RESEARCH |
| 2 | Graph indexing does not scale: MS-GraphRAG physically indexed 1.8% of a 512k-doc corpus; LightRAG 0.5% (super-linear build, b=1.36 ⇒ ~102B tokens for the full set). HippoRAG 2 scales linearly and lands ~15 F1 BELOW BM25 at 132k docs. | scale studies cited in-corpus · 2025–26 | RESEARCH |
| 3 | Benchmark integrity in this field is broken: LoCoMo judge accepts 62.81% of WRONG answers (ceiling ~93.6%); 31–50% of HotpotQA answerable with NO retrieval; LightRAG's 72-28 pairwise win becomes 39.06% under position-bias correction; vendors score themselves with self-authored judges. | independent audits · 2025–26 | RESEARCH |
| 4 | Graphs genuinely win on 5 classes only: TEMPORAL/supersession 30.70→49.91 (arXiv 2502.11371, the largest honest margin); graph-topology operators 0/11→11/11 — dense retrieval cannot express them at all (arXiv 2606.06003); aggregation/comparison +16–36pp; entity-bridging multi-hop +9.5 F1 (⚠2Wiki is template-generated = near-circular); corpus-level sensemaking 72–83% (⚠vendor, LLM-judged). Everywhere else graphs lose or tie: single-hop −13.4%, time-sensitive updates −16.6%. | arXiv 2502.11371 · arXiv 2606.06003 · MultiHop-RAG · MSR · 2025–26 | RESEARCH |
| 5 | Cost decides it: plain vector ≈ $0.02 per corpus-Mtoken; MS-GraphRAG $3.0–3.7 (default chunking) to $18–34 (ℓ=150) — 100–1,700× vector cost. Index-token overhead: RAPTOR 1.1× · HippoRAG 3.5× · MS-GraphRAG 8.6× · LightRAG 9.0×. Query overhead 4–45× constrained, up to 377× (GraphRAG global = 331,375 tok/query). Nobody publishes p50/p95 — measure it yourself. | KET-RAG (KDD 2025, the one independent $-denominated study) · GraphRAG-Bench | RESEARCH |
| 6 | cognee: do not adopt; lift 4 patterns. Hard instructor dependency ⇒ local models fail outright (7 Ollama models, 0% success; their issue #3870 concedes it); default engine is a community fork of Kuzu (Apple-acquired, ARCHIVED 2025-10-10); entity resolution is exact-name matching; whole-graph materialization to rank triplets; their BEAM “SOTA” is in-sample by their own report; tuning swung HotpotQA EM 0.042→0.667 (~16×) ⇒ configuration dominates architecture. WORTH LIFTING (Apache-2.0): deterministic uuid5 identity for free UPSERT · Task/Pipeline decomposition · the 16-search-type taxonomy · content-hash incremental ingest. | cognee v1.4.0 repo + own benchmark report · 2026 | SHIPPING (audited) |
| 7 | Build order by measured leverage: (1) a strong embedder (~20 R@5 on HippoRAG 2 itself — enough to flip published conclusions); (2) a cross-encoder reranker (+17pp R@5, T2-RAGBench); (3) agent grep/filesystem tools (beat Mem0 in Letta's own test); (4) only then a graph, chosen by workload. Independently confirmed on our own ledger: the debt-clusterer's dense/PPMI arm was strictly harmful while precision-ordered lexical anchors + rulers won — the signal order mattered more than the architecture. | compiled from 1–5 · OURS: nx_debtcluster arms 2026-08-01 | RESEARCH + OURS |
Myths killed (do not repeat)
| “LazyGraphRAG gives 0.1% indexing cost” | It never shipped. Not in microsoft/graphrag v3.1.1, 0 arXiv results, blog-only. Structurally unreplicable. |
| “$33K to index 5GB, per Microsoft” | Microsoft never said it — a 1,600× extrapolation from $21/3.2MB at a non-default chunk size; ~9× too high at real defaults. |
| “LightRAG uses <100 tokens/query” | Measured 100,832 (~115× vanilla); its latency claim inverts on independent measurement (20–80× slower than GraphRAG-local). |
| “RAPTOR +20%” | The tree itself is worth +1.7pts; the +20% came from swapping GPT-4 in as reader. (Still the quality-per-dollar winner.) |
Declared UNVERIFIED / gaps
| 1 | The experiment nobody has run (and the field's most important absence): hybrid BM25 + 2026-tier embedder + cross-encoder + k≈20 vs graph-RAG, SAME reader, SAME token budget, ≥100k docs. Every published study omits at least one control. Run it before spending a dollar on graph machinery. |
| 2 | Several margin numbers in claim 1 are from single studies not yet replicated across harnesses; treated as directional. |
| 3 | No p50/p95 latency exists anywhere in the graph-RAG literature — all cost claims are token-denominated only. |
| 4 | ✓ CLOSED 2026-08-01: nx_page_ingest fixed (stale-mirror rebuild + manifest-poison guard, debt 1785623390 EATEN); the arXiv citations above (2502.11371, 2606.06003) are ARCHIVED in the sovereign library and search-verified. |
How this lands on Nishi
We already hold rungs 1–3's substrate: BM25/WAND (nishi_search), nx_recall_dense + nx_recall_fuse (gated, nDCG 530→964 / 494→659), nx_shelltool+nx_fs (the agent grep tools that beat Mem0 in Letta's test), nx_eco_graph CSR, and /code/ask with a mechanical hallucination guard. The honest gap vs cognee is typed entity/relation extraction — the exact stage their design makes local-model-hostile and ours would not have to be. Anything built here must be judged on the 5 winning classes at MATCHED token budget against a BM25+embedder+reranker baseline — otherwise we reproduce the field's failure verbatim.