code wiki / _hdl_build / nx_site_ingest.nx

nx_site_ingest.nx

buildroot/runtime/_hdl_build/nx_site_ingest.nx

6720 B137 linesdepth 3pulls 3 transitivereach 1 importersview sourcekind librarytopic site
docsdependenciesstructsconstsfunctions

about

nx_site_ingest.nx -- FLAG ingestion-friendly sites + AUTO-INGEST the full site to the local Nishi library (operator: "flag good sites of research that allow ingestion and ... auto ingest the full site for local use via the nishi library", so "bot-blocking cant starve us"). The discipline: 1. PERMISSION FIRST -- never bulk-ingest a site that doesn't allow it. si_robots_allows parses robots.txt Disallow rules; si_is_open_access reads license/repository signals (CC / arXiv / PMC / DOI / MIT / public-domain). Only INGEST_OK sites get flagged + mirrored. 2. WHOLE-SITE MIRROR -- enumerate the site's URLs from its sitemap, fetch each POLITELY (via nx_polite_crawl's per-host pacing), and write each page into the library on disk. 3. LOCAL-FIRST / STARVE-PROOF -- once mirrored, the team reads the LOCAL copy; if the live site goes dark or bot-blocks us, the corpus is already ours. Composes nx_polite_crawl (the fetch governor that keeps it polite) + nx_research_synth/claim_extract (which then corroborate/extract the mirrored text). LAWS: struct-free, integer-only. license_tier: ORIGINAL

dependencies 2 imports · 1 importers

nx_polite_crawl.nx nx_syscalls.nx nx_site_ingest.nx nx_site_ingest_test.nx

imports: nx_polite_crawl.nxnx_syscalls.nx

imported by: nx_site_ingest_test.nx

structs

none

consts

16const SI_INGEST_OK: i64 = 2 // allowed + clearly licensed/repository -> flag + auto-ingest
17const SI_INGEST_PARTIAL: i64 = 1 // allowed by robots but unclear license -> ingest cautiously, flag-for-review
18const SI_INGEST_FORBIDDEN: i64 = 0 // robots disallows -> DO NOT bulk-ingest (respect the site)

functions

20func si_strlen(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} return n }
called by 2: si_count_subsi_robots_disallows
22func si_match_at(buf: *u8, pos: i64, n: i64, needle: *u8) -> i64
called by 2: si_count_subsi_robots_disallows
32func si_count_sub(buf: *u8, n: i64, needle: *u8) -> i64
called by 2: si_open_signalssi_sitemap_url_count calls 2: si_strlensi_match_at
41func si_is_prefix(buf: *u8, vs: i64, vl: i64, path: *u8, pathlen: i64) -> i64
called by 1: si_robots_disallows
52func si_robots_disallows(robots: *u8, n: i64, path: *u8, pathlen: i64) -> i64
called by 1: si_robots_allows calls 3: si_strlensi_match_atsi_is_prefix
70func si_robots_allows(robots: *u8, n: i64, path: *u8, pathlen: i64) -> i64
called by 1: main calls 1: si_robots_disallows
76func si_open_signals(buf: *u8, n: i64) -> i64
called by 1: main calls 1: si_count_sub
91func si_ingest_verdict(robots_allows: i64, open_signals: i64, is_repository: i64) -> i64
called by 1: main
99func si_flag_good_site(verdict: i64) -> i64 { if verdict == SI_INGEST_OK { return 1 } return 0 }
called by 1: main
102func si_sitemap_url_count(sitemap: *u8, n: i64) -> i64 { return si_count_sub(sitemap, n, "<loc>" as *u8) }
calls 1: si_count_sub
105func si_ingest_progress_permil(ingested: i64, total: i64) -> i64
109func si_fully_mirrored(ingested: i64, total: i64) -> i64
115func si_starve_proof(fully_mirrored: i64) -> i64 { return fully_mirrored }
119func si_library_write(path: *u8, data: *u8, len: i64) -> i64
calls 3: sys_openat_wrsys_writesys_close
133func si_ingest_admissible(verdict: i64, paced_through_governor: i64) -> i64
called by 1: main