code wiki / _hdl_build / nx_semcorpus_build.nx

nx_semcorpus_build.nx

buildroot/runtime/_hdl_build/nx_semcorpus_build.nx

13920 B286 linesdepth 4pulls 5 transitivereach 0 importersview sourcekind tool
docsdependenciesstructsconstsfunctions

about

nx_semcorpus_build.nx -- R3b-scale CORPUS-LEVEL distributional embeddings (the rung after 3 measured-negative per-question climbs; see nx_qabench MODE2 header). Builds ONE persisted hashed count-embedding matrix from ALL banked benchmark context prose (~4.5MB Wikipedia-register: qab_squad_big + qab_hotpot_big + qab_musique_big), then nx_qabench's MODE2 loads it and selects sentences by corpus-informed cosine. METHOD (sovereign, integer, no training loop -- Random-Indexing/feature-hashing lineage): text = every JSON string value >= 48 chars (paragraphs+questions; SHORT strings -- keys, titles, GOLD ANSWERS -- are structurally excluded = no label leakage; corpus is the inference-visible context pool, unsupervised). token row = djb2(token) % 16384; for each content token, each stop-filtered context token within +-4 hashes to one of 256 dims: M[row*256 + dim] += 1. Persisted -> knowledge/index/semcorpus_v1.bin (NXSEMC1 header). HONEST CAVEATS baked in: hash collisions (~2-4 words/row) + raw counts (no PPMI) = v1; probes below REPORT whether corpus stats order won~defeated > won~purple (the exact wall) -- reported, not asserted. TEETH: T1 strings>=3000 T2 token-events>=200000 T3 nonzero cells>=50000 T4 write+reload spot-check T5 NEG: identity cos=1000 and a gibberish token's empty row scores 0 vs everything. expect_exit: 0 license_tier: ORIGINAL

dependencies 1 imports · 0 importers

nx_qabench_engine.nx nx_semcorpus_build.nx

imports: nx_qabench_engine.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main db_w sc_ingest db_read_raw sys_read db_w ↻ db_body_start db_b db_dec_str db_skip_ws db_b ↻ db_b ↻ qs_norm qs_tok qs_is_article db_is_stop db_tok_is sc_hash db_n nxi_out nxi_fd sys_mmap ccz_cat_num sys_write sys_munmap db_n ↻ sc_isqrt sc_probe sc_hash ↻ sc_len sc_cos_rows sc_isqrt ↻ sc_hash ↻ sc_cos_rows ↻ sys_openat_wr sys_read ↻

structs

none

consts

16const SC_MAGIC_5381: i64 = 5381
17const SC_MAGIC_1073741789: i64 = 1073741789
18const SC_MAGIC_16777216: i64 = 16777216
19const SC_MAGIC_250000: i64 = 250000
20const SC_MAGIC_4000: i64 = 4000
21const SC_MAGIC_2654435761: i64 = 2654435761
22const SC_MAGIC_262144: i64 = 262144
23const SC_MAGIC_32768: i64 = 32768
24const SC_MAGIC_4096: i64 = 4096
25const SC_MAGIC_65536: i64 = 65536
26const SC_MAGIC_3000: i64 = 3000
27const SC_MAGIC_200000: i64 = 200000
28const SC_MAGIC_50000: i64 = 50000
30const SC_ROWS: i64 = 16384
31const SC_DIMS: i64 = 256
32const SC_WIN: i64 = 4
33const SC_MINSTR: i64 = 48

functions

35func sc_hash(buf: *u8, off: i64, len: i64) -> i64
42func sc_isqrt(v: i64) -> i64
called by 2: sc_cos_rowsmain
51func sc_cos_rows(M: *i64, ra: i64, rb: i64) -> i64
called by 2: sc_probemain calls 1: sc_isqrt
74func sc_probe(M: *i64, a: *u8, b: *u8) -> i64
called by 1: main calls 3: sc_hashsc_lensc_cos_rows
79func sc_len(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
called by 1: sc_probe
83func sc_ingest(g: *i64, path: *u8, M: *i64, c: *i64) -> i64
137func main() -> i64