nx_crawl_frontier.nx
buildroot/runtime/nx_crawl_frontier.nx
about
nx_crawl_frontier.nx -- outbound-link (frontier) extraction for the crawler.
module: nishi-core.search.crawl_frontier
depends: syscalls.nx
capability: CORE_COMPUTE
wired_status: FULLY_WIRED
The discovery half of the crawl loop: from a fetched HTML page, extract the
outbound href links to enqueue for crawling. Byte-scans for href="..." or
href='...', keeps absolute http(s) links, and dedups exact repeats. The
frontier is what lets the crawler walk an entity's real footprint (its own
links, official-site links, primary-source citations) rather than only
whatever ranks -- the knowledge-graph / link-graph approach to QUALITY
coverage, not SEO-bait. (Relative-URL resolution against a base href is a v2
refinement; absolute links are the bulk of a frontier and the safe v1.)
dependencies 1 imports · 4 importers
imports: syscalls.nx
imported by: nx_crawl_bfs.nxnx_crawl_frontier_test.nxnx_crawl_main.nxnx_crawl_polite.nx
structs
| none |
consts
| none |
functions
| 19 | func nx_cf_lc(c: i64) -> i64 called by 1: nx_cf_is_href_eq_at |
| 25 | func nx_cf_is_href_eq_at(html: *u8, html_len: i64, off: i64) -> i64 |
| 36 | func nx_cf_has_prefix(html: *u8, html_len: i64, off: i64, lit: *u8, lit_len: i64) -> i64 called by 1: nx_crawl_extract_links |
| 47 | func nx_cf_eq(html: *u8, cs: i64, clen: i64, buf: *u8, bo: i64, blen: i64) -> i64 called by 1: nx_crawl_extract_links |
| 60 | func nx_crawl_extract_links(html: *u8, html_len: i64, |