code wiki / _hdl_build / nx_dr_ppmi_build.nx
nx_dr_ppmi_build.nx
buildroot/runtime/_hdl_build/nx_dr_ppmi_build.nx
about
nx_semppmi_build.nx -- EXPLICIT-VOCAB SPARSE PPMI embeddings = the CANONICAL count-based method at full
strength (Levy&Goldberg: full sparse PPMI rows rival trained embeddings on word similarity). This is the
last count-method rung before "trained-only" is airtight: v1 (nx_semcorpus_build) failed its probes with
HASHED rows (collisions) + a 256-dim PROJECTION (lossy) -- this build removes BOTH: every word gets its own
row; contexts are the full vocabulary; cells are PPMI (positive pointwise mutual information, log2 x16
fixed-point); rows stay SPARSE (sorted triples). Corpus = same banked benchmark prose (no gold leakage:
only JSON strings >= 48 chars).
PIPELINE (one corpus walk): tokens -> 61-bit djb2x hash stream (+ -1 markers at string ends) ->
vocab = sort(copy)+unique (explicit; 61-bit space => collisions ~0 at 60k words) ->
window +-4 pair emission as packed keys (aid<<20|bid, both directions) -> sort -> RLE -> triples(cnt>=2) ->
marginals -> PPMI16 -> row index + norms -> PROBES -> persist knowledge/index/semppmi_dr_v1.bin.
PROBES decide (same discipline as v1): if won~defeated fails to order again, count methods are DEAD at
canonical strength and the R1 trained thread is the proven only door. TEETH structural + neg-controls.
expect_exit: 0 license_tier: ORIGINAL
dependencies 1 imports · 0 importers
imports: nx_qabench_engine.nx
imported by: nobody (leaf or entry point)
call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown
structs
| none |
consts
| 16 | const SP_MAGIC_262144: i64 = 262144 |
| 17 | const SP_MAGIC_16777216: i64 = 16777216 |
| 18 | const SP_MAGIC_250000: i64 = 250000 |
| 19 | const SP_MAGIC_4000: i64 = 4000 |
| 20 | const SP_MAGIC_32768: i64 = 32768 |
| 21 | const SP_MAGIC_5000000: i64 = 5000000 |
| 22 | const SP_MAGIC_30000000: i64 = 30000000 |
| 23 | const SP_MAGIC_1048576: i64 = 1048576 |
| 24 | const SP_MAGIC_8000000: i64 = 8000000 |
| 25 | const SP_MAGIC_4096: i64 = 4096 |
| 26 | const SP_MAGIC_3000: i64 = 3000 |
| 27 | const SP_MAGIC_10000: i64 = 10000 |
| 28 | const SP_MAGIC_100000: i64 = 100000 |
| 29 | const SP_MAGIC_50000: i64 = 50000 |
| 31 | const SP_WIN: i64 = 4 |
| 32 | const SP_MINSTR: i64 = 48 |
| 33 | const SP_MAXV: i64 = 262144 |
| 34 | const SP_MINCNT: i64 = 2 |
functions
| 37 | func sp_hash(buf: *u8, off: i64, len: i64) -> i64 { return db_semhash(buf, off, len) } |
| 40 | func sp_wblock(fd: i64, buf: *u8, n: i64) -> i64 called by 1: main |
| 52 | func sp_isqrt(v: i64) -> i64 called by 1: sp_cos |
| 61 | func sp_ilog2x16(q: i64) -> i64 called by 1: main |
| 75 | func sp_qsort(a: *i64, lo: i64, hi: i64) -> i64 |
| 97 | func sp_bsearch(a: *i64, n: i64, v: i64) -> i64 { return db_bsearch_i64(a, n, v) } |
| 101 | func sp_walk(g: *i64, path: *u8, stream: *i64, c: *i64) -> i64 |
| 142 | func sp_cos(tctx: *i64, tval: *i64, ridx: *i64, nrm2: *i64, a: i64, b: i64) -> i64 |
| 165 | func sp_wid(vh: *i64, nv: i64, w: *u8) -> i64 |
| 172 | func main() -> i64 |