nishi code wiki / research / systems perf math and tier economics
Systems perf math and model-tier economics
Researched 2026-07-29 (two 2-agent sweeps, agent-fetched primary sources); transcribed 2026-08-01. Corpus slug: reference-sota-perf-math-and-tier-numbers-2026-07-29. ⚠ These are research-grade citations, NOT locally reproduced — verify a number before it becomes load-bearing for a ship/revert decision; that is the ratchet rule.
What this brief is. The measured evidence behind the perf-and-tiered-routing doctrine, and the ranked build order those numbers imply for a no-float integer-serve engine. Its two headline findings each REFUTED work we had already planned — which is exactly what research is for.
Forks off — the lineage
| parent | inherited |
|---|---|
feedback-perf-and-tiered-routing-doctrine-2026-07-29 (corpus) | That file is the POLICY; this brief is the EVIDENCE it cites. |
project-nishi-search-wand-impact-postings-2026-07-25 (corpus) | The WAND rung this brief REFUTES (correction 1). |
The two corrections to our own banked roadmap
| 1 | DO NOT BUILD BlockMax-WAND. Measured 2024–26 on MS MARCO/SPLADE k=10 single-thread: MaxScore 120.6ms · BMW 614.2ms · BMP 10.5ms — BMW is 5× SLOWER than plain MaxScore on learned/heavy queries; Block-Max Pruning (SIGIR 2024) wins 58× over BMW. Successors: Dynamic Superblock Pruning 0.629ms; LSP/0 0.425ms with a 4-bit index at 8.44GB vs BMP's 29.0GB. Our .imp sidecar remains the right slot; the TRAVERSAL design changes. (arXiv 2405.01117 · 2504.17045 · 2602.02883) |
| 2 | The RTX 5080 is 0.64× a 4090 on PREFILL/compute, not ~0.98× (llama.cpp: pp512 9,488 vs 14,771; decode 184.7 vs 189.0 — decode misleads). Embedding throughput scales with COMPUTE ⇒ every plan that assumed ~0.9× was ~40% optimistic. Derived: bge-large-class ~21,600 tok/s ⇒ 100K docs ~40min, 1M ~6.6h. |
Load-bearing numbers by area (sources in the corpus file)
| area | the numbers that decide the design |
|---|---|
| Postings / top-k | 8-bit impacts + 16-bit saturating accumulators reproduce float effectiveness (IOQP/JASSv2, SPLADEv2 RR@10 0.368); anytime termination cut VBMW p99 116.3ms→10.0ms at RBO 0.931; BM25-guided traversal took DeepImpact 219.9→4.8ms (46×). Reference envelope: tantivy ~6M docs 2-term AND 32–76µs ⇒ our warm 185–400ms is ~3 orders off the ceiling and the gap is ARCHITECTURE, not hardware. Stay doc-sorted (SaaT merges become rebuilds). |
| Compression / succinct | Partitioned Elias-Fano: 4.10 bits/int, AND 14–26% faster than OptPFD; uniform-128 costs +11% and avoids a 95-min construction. SIMD bitpacking is memory-bound-free (BitPacker4x 5.5 G ints/s; AVX2 8x not worth it). rank 6ns@3.1%, select1 10ns@15.6%; popcnt+BMI2 = 2–3×. Marginalia's purpose-built intersect beats Roaring 5–20× on exactly our co-citation shape — re-derive at our boundaries, don't import the general case. |
| Streaming compaction | The named fix for our compactor dying at ~1.9GB: remap table FIRST, loser tree over K bounded cursors ⇒ memory O(K·B) independent of data (K=64/4MiB ≈ 520MB fixed). DuckDB k-way 3.39×; 8-way loser tree 15×. AND keep the compactor OUT of the page cache (O_DIRECT/fadvise): fio holds ~900K reads/s while mmap drops to NEAR ZERO when the cache fills — the measured mechanism behind our own round-17 edge timeout. Size-tiered K≈8–10, growth ≈2 ⇒ write-amp 2–4 not 10–30. |
| Vector / ANN, integer serve | RaBitQ (SIGMOD24/25): 3-bit >95% recall, no reranking, no raw vectors; PQ at equal code length >50% relative error. Binary + int8 rescore = ~96% of float quality at 32× memory, 24.76× speedup — the rescore stage is load-bearing (76% recall plateau without). Skip DiskANN below ~100M vectors; linear Hamming does ~60M codes/s/core. |
| Graph math | BVGraph: 105.9M nodes / 3.74B arcs ≈ 560MB — the whole CC-domain graph RAM-resident on 36GB. Precompute top-k neighbors at INGEST (L2AP up to 1600×); never compute similarity at serve. HyperBall: 100M nodes ≈ 1.6GB counters; GVE-LPA 1.4B edges/s. |
| Ingest GPU (one 5080, 16GB) | Continuous batching ~19× single-stream; 50 output tokens cost 4.3× the time of 600 input tokens ⇒ emit short integer-enum JSON, the bigger win than any engine swap. Use a 137M–335M embedder at our scale. fp16 8B does not fit 16GB; Q4 near-lossless for code. Skip speculative decoding at batch saturation. |
| Edge / delivery | ECDSA P-256 signs 20,508/s/core vs RSA-2048's 1,009/s — a 20× handshake cap decided purely by certificate choice (CHECK OURS). kTLS/sendfile only >64KiB (worse at 1–8KiB). io_uring only above ~150 active connections/core. |
| Tier economics | DeepSeek V3.2 = cheapest proven coder (Aider 74.2% at $1.30/run vs GPT-5-high 88.0% at $29.08 — 22× cost for +14pts). Route-down break-even: at a 100–150× price ratio the cheap tier only loses when p≈0. RouteLLM: 85% cost cut at 95% quality with 14% escalated. Test-time compute + an oracle we already own: 15.9%@1 → 56%@250 samples; coverage is log-linear ⇒ k=2–3 retries captures most, k>3 is waste — our gate suites ARE the verifier. Frontier writes the spec+oracle ONCE, cheap tiers execute N×. Batch endpoints = flat 50% off. Distillation from our own traces: 50–90% inference-cost cut. Pitfalls: nested JSON collapses small models (68–69% non-compliance) ⇒ FLAT schemas; no oracle ⇒ retries buy nothing; never let a cheap tier touch promote/deploy. |
The ranked build order (win per unit of effort, abridged)
1 loser-tree bounded-memory compactor · 2 compactor bypasses page cache · 3 BMP forward block plane · 4 anytime block budget · 5 sorted-list + 64-bit bloom for the CC domain plane · 6 docid reordering at merge · 7 popcnt+BMI2 flags · 8 PEF docids · 9 size-tiered compaction · 10 batched tagging + integer-enum output · 11 RaBitQ planes · 12 BV adjacency + HyperBall · 13 ECDSA cert + per-core epoll · 14 kTLS for the WARC bank only · 15 MIH (deferred). DO NOT BUILD: BlockMax-WAND · DiskANN below ~100M · BK-trees · leveled LSM · general Roaring for the domain plane · AVX2 BitPacker8x · speculative decoding for batched tagging.
Declared UNVERIFIED
| 1 | Every number is an agent-fetched citation, not locally reproduced — the brief's own opening warning. Re-verify before any ship/revert decision. |
| 2 | Tier prices/model availability churn monthly (the corpus notes Gemini 2.5 Flash retiring 2026-10-16, GLM +30%); re-census before committing routing budgets. |
| 3 | The individual paper links for the postings/compression/ANN claims live in the corpus file and NAS copy; they are not yet individually archived in the sovereign library — queue per rule A on next touch. |