code wiki / (root) / nx_crawl_bfs.nx

nx_crawl_bfs.nx

buildroot/runtime/nx_crawl_bfs.nx

7070 B163 linesdepth 6pulls 18 transitivereach 0 importersview sourcekind tooltopic crawl
docsdependenciesstructsconstsfunctions

about

nx_crawl_bfs.nx -- autonomous link-following crawler (frontier BFS). module: nishi-core.search.crawl_bfs depends: nx_http_client, nx_html_to_text, nx_simhash, nx_bm25, nx_crawl_frontier, fx, nx_str capability: APP_RUNNABLE The real crawler: from ONE seed it FETCHES, extracts outbound links (nx_crawl_frontier), enqueues the unseen ones (visited-set guards re-crawl), and repeats -- discovering the site graph, not a hardcoded list. Each page is content-deduped (nx_simhash) and indexed (nx_search_inverted). Proven on the loopback mock-web: seed "/" -> discovers + crawls /wiki, /thothub, /fan.

dependencies 7 imports · 0 importers

fx.nx nx_str.nx nx_http_client.nx nx_html_to_text.nx nx_simhash.nx nx_bm25.nx nx_crawl_frontier.nx nx_crawl_bfs.nx

imports: fx.nxnx_str.nxnx_http_client.nxnx_html_to_text.nxnx_simhash.nxnx_bm25.nxnx_crawl_frontier.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main nx_http_client_sockaddr_ip nx_str_len nx_inv_new nx_puts nx_str_len ↻ nx_str_eq nx_http_client_get sys_socket nx_connect_bounded nx_fcntl sys_connect sys_mmap sys_poll sys_close sys_mmap ↻ nx_http_client_build_reque hc_put_caps sys_mmap ↻ nx_codec_caps_headers sys_mmap ↻ nx_codec_caps_accept_docum sys_mmap ↻ nx_codec_caps_accept_image nx_cc_put nx_codec_caps_accept_encod cc_encoding_have nx_cc_put ↻ cc_encoding_name nx_cc_put ↻ sys_write _drain sys_read nx_body_off nx_html_to_text nx_html_to_text_x h2t_state_new sys_mmap ↻ sys_mmap ↻ h2t_scan_tag

structs

none

consts

21const K_MAGIC_8088: i64 = 8088
22const K_MAGIC_16384: i64 = 16384
23const K_MAGIC_8192: i64 = 8192

functions

25func nx_putc(c: i64) -> i64 { let b: *u8 = sys_mmap(1); b[0] = c; sys_write(1, b, 1); return 0 }
called by 2: nx_put_i64nx_crawl_search
26func nx_puts(s: *u8) -> i64 { sys_write(1, s, nx_str_len(s)); return 0 }
called by 2: nx_crawl_searchmain calls 1: nx_str_len
27func nx_put_i64(n: i64) -> i64
called by 1: main calls 1: nx_putc
34func nx_body_off(resp: *u8, n: i64) -> i64
called by 1: main
41func nx_url_path(url: *u8, out: *u8, outcap: i64) -> i64
called by 1: main calls 1: nx_str_len
72func main() -> i64
calls 17: nx_http_client_sockaddr_ipv4nx_str_lennx_inv_newnx_putsnx_str_eqnx_http_client_get+11