code wiki / _hdl_build / nx_library_search.nx
nx_library_search.nx
buildroot/runtime/_hdl_build/nx_library_search.nx
about
nx_library_search.nx -- S-class search/retrieval over a LIBRARY of information (not one source).
The Researcher recognizes an index page as a LIBRARY, harvests every doc it lists, then answers a
SEARCH PROMPT by RANKING the whole corpus -- TF-IDF style, which exceeds naive substring search:
* substring search can only say present/absent in one blob and cannot rank or pick the best doc
* this ranks N docs by sum over query terms of tf(term,doc) * idf(term), where RARER terms (low
document-frequency) weigh MORE -- so a doc that matches the distinctive query terms wins over a
doc that merely mentions a common one. That is the retrieval quality real search engines use.
Corpus is held as parallel arrays: ptrs[k] = (doc text *u8 stored as i64), lens[k] = its length.
license_tier: ORIGINAL Pairs with nx_research_extract (re_count/re_has/re_find) + nx_library_cache.
dependencies 3 imports · 5 importers
imports: nx_research_extract.nxnx_bm25.nxnx_syscalls.nx
imported by: nx_engineer_wire_sclass_gate.nxnx_engineer_wire_test.nxnx_library_search_gate.nxnx_library_search_test.nxnx_referee_test.nx
structs
| none |
consts
| none |
functions
| 18 | func ls_is_library(text: *u8, n: i64) -> i64 { if re_count(text, n, "[doc] " as *u8) >= 3 { return 1 } return 0 } |
| 19 | func ls_entry_count(text: *u8, n: i64) -> i64 { return re_count(text, n, "[doc] " as *u8) } |
| 23 | func ls_entry_path(text: *u8, n: i64, k: i64, out: *u8, cap: i64) -> i64 |
| 46 | func ls_doc(ptrs: *i64, k: i64) -> *u8 { return ptrs[k] as *u8 } |
| 49 | func ls_tf(ptrs: *i64, lens: *i64, k: i64, term: *u8) -> i64 { return re_count(ls_doc(ptrs, k), lens[k], term) } |
| 52 | func ls_df(ptrs: *i64, lens: *i64, ndocs: i64, term: *u8) -> i64 |
| 59 | func ls_idf(ndocs: i64, df: i64) -> i64 { return ndocs - df + 1 } called by 1: ls_score |
| 62 | func ls_score(ptrs: *i64, lens: *i64, ndocs: i64, k: i64, qterms: *i64, nq: i64) -> i64 |
| 74 | func ls_best(ptrs: *i64, lens: *i64, ndocs: i64, qterms: *i64, nq: i64) -> i64 |
| 86 | func ls_best_bm25(ptrs: *i64, lens: *i64, ndocs: i64, qterms: *i64, nq: i64) -> i64 |