code wiki / _hdl_build / nx_ppmi_svd.nx
nx_ppmi_svd.nx
buildroot/runtime/_hdl_build/nx_ppmi_svd.nx
about
nx_ppmi_svd.nx -- the sovereign semantic upgrade (Researcher-specced, built bits-up): PPMI weighting
over the team's co-occurrence counts. Raw co-occurrence over-credits FREQUENT terms (a term that is
in every doc co-occurs with everything, so it inflates unrelated docs). Pointwise Mutual Information
fixes this: PMI(a,b) = log( P(a,b) / (P(a)P(b)) ) = log( cooc(a,b)*D / (df(a)*df(b)) ); PPMI keeps the
POSITIVE part, so a term that co-occurs only AT CHANCE scores 0 and stops polluting retrieval. This is
the count-based dense-embedding rung (Levy & Goldberg 2014: PPMI is competitive with word2vec; the
further SVD dimensionality reduction is the next sub-rung, flagged not faked). Integer-only: PMI uses
the fixed-point ln from nx_bm25. license_tier: ORIGINAL
dependencies 3 imports · 1 importers
imports: nx_research_extract.nxnx_bm25.nxnx_syscalls.nx
imported by: nx_ppmi_svd_test.nx
structs
| none |
consts
| none |
functions
| 14 | func ps_doc(ptrs: *i64, k: i64) -> *u8 { return ptrs[k] as *u8 } |
| 17 | func ps_df(ptrs: *i64, lens: *i64, N: i64, term: *u8) -> i64 |
| 22 | func ps_cooc(ptrs: *i64, lens: *i64, N: i64, a: *u8, b: *u8) -> i64 |
| 29 | func ps_ppmi_micro(cooc: i64, df_a: i64, df_b: i64, D: i64) -> i64 |
| 39 | func ps_build_qvec(ptrs: *i64, lens: *i64, N: i64, qterm: *u8, vocab: *i64, V: i64, D: i64, qvec: *i64) -> i64 |
| 53 | func ps_score(ptrs: *i64, lens: *i64, k: i64, qvec: *i64, vocab: *i64, V: i64) -> i64 |
| 59 | func ps_best(ptrs: *i64, lens: *i64, N: i64, qvec: *i64, vocab: *i64, V: i64) -> i64 |
| 66 | func ps_build_raw_qvec(ptrs: *i64, lens: *i64, N: i64, qterm: *u8, vocab: *i64, V: i64, qvec: *i64) -> i64 |