code wiki / (root) / nx_simhash.nx

nx_simhash.nx

buildroot/runtime/nx_simhash.nx

5302 B136 linesdepth 4pulls 6 transitivereach 60 importersview sourcekind library
docsdependenciesstructsconstsfunctions

about

nx_simhash.nx -- SimHash near-duplicate / content-novelty kernel. module: nishi-core.search.simhash depends: fx.nx (Q16.16), nx_search_inverted.nx (reuse FNV-1a + tokenizer) capability: CORE_COMPUTE wired_status: FULLY_WIRED genealogy_id: charikar_2002_simhash + manku_2007_nearduplicate WHY (operator cardinal 2026-05-29 -- deliver info, additive, NO prejudging): the SEO-resistance + anti-junk mechanism must be EARNED and MEASURED, never assumed from a domain or title keyword. SimHash measures CONTENT overlap: a page that merely re-copies a higher-quality source (the dominant SEO-spam pattern -- mass-produced/templated/scraped content) collapses onto the canonical original, while genuinely UNIQUE content (Comic-Con pics, candids, original reporting) keeps a large Hamming distance and is preserved + valued. To escape demotion an adversary must actually add unique information -- which is exactly the point. Deterministic: token FNV-1a -> per-bit accumulate -> 64-bit fingerprint -> Hamming distance. Integer-only, no floats.

dependencies 4 imports · 20 importers

fx.nx syscalls.nx nx_search_inverted.nx nx_bits.nx nx_simhash.nx _simhash_popcount_gate.nx nx_asset_autotag.nx nx_crawl_bfs.nx nx_crawl_doc.nx nx_crawl_https.nx nx_crawl_main.nx nx_crawl_polite.nx nx_crawl_run.nx nx_crawl_web.nx nx_diora_dedup_demo.nx

diagram shows first 10 each side; +0 more imports, +10 more importers in the complete lists below.

imports: fx.nxsyscalls.nxnx_search_inverted.nxnx_bits.nx

imported by: _simhash_popcount_gate.nxnx_asset_autotag.nxnx_crawl_bfs.nxnx_crawl_doc.nxnx_crawl_https.nxnx_crawl_main.nxnx_crawl_polite.nxnx_crawl_run.nxnx_crawl_web.nxnx_diora_dedup_demo.nxnx_imgkp.nxnx_phash.nxnx_phash_index.nxnx_recording.nxnx_recording_gate.nxnx_revimg_discover.nxnx_search_crawl_server.nxnx_simhash_test.nxnx_web_crawl_step.nxnx_web_ingest.nx

structs

none

consts

none

functions

31func nx_simhash_fingerprint(text: *u8, n: i64) -> i64
called by 9: mainnx_crawl_extractmainmainmainmain+3 calls 2: nx_inv_is_token_charnx_inv_hash_bytes_lower
77func nx_simhash_hamming(a: i64, b: i64) -> i64
called by 44: mainmainat_hammingmainmainmain+38 calls 1: nx_bits_popcount64_soft
86func nx_simhash_is_near_dup(a: i64, b: i64, threshold: i64) -> i64
called by 1: main calls 1: nx_simhash_hamming
92func nx_simhash_similarity_q16(a: i64, b: i64) -> i64
called by 1: main calls 2: nx_simhash_hammingfx_from_frac
101func nx_simhash_min_hamming(fp: i64, seen: *i64, n_seen: i64) -> i64
called by 2: nx_crawl_should_indexmain calls 1: nx_simhash_hamming
118func nx_simhash_cluster(fps: *i64, n: i64, threshold: i64, cluster_of: *i64) -> i64
called by 2: mainmain calls 1: nx_simhash_hamming