nx_crawl_polite.nx
buildroot/runtime/nx_crawl_polite.nx
about
nx_crawl_polite.nx -- hardened, polite crawler (don't get blacklisted).
module: nishi-core.search.crawl_polite
depends: nx_syscalls, nx_str, nx_http_client, nx_html_to_text, nx_simhash,
nx_bm25, nx_crawl_frontier, nx_robots
capability: APP_RUNNABLE
nx_crawl_bfs + the anti-blacklist hardening (RFC 9309 + polite-crawling):
* fetch + honor robots.txt (skip Disallow paths) -- the host governs its
own content (the not-god cardinal), and honoring it avoids bans
* descriptive User-Agent header (identify ourselves, contactable)
* Crawl-delay pacing (sleep between requests; robots value or a default)
* 429/503 back-off (status-aware; honor rate-limit signals)
Proven on loopback: discovers the graph, SKIPS the robots-disallowed
/private, paces by Crawl-delay, sends our UA.
dependencies 9 imports · 0 importers
imports: nx_syscalls.nxnx_str.nxnx_http_client.nxnx_http_get_ua.nxnx_html_to_text.nxnx_simhash.nxnx_bm25.nxnx_crawl_frontier.nxnx_robots.nx
imported by: nobody (leaf or entry point)
call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown
structs
| none |
consts
| 26 | const K_MAGIC_8088: i64 = 8088 |
| 27 | const K_MAGIC_8192: i64 = 8192 |
| 28 | const K_MAGIC_16384: i64 = 16384 |
functions
| 30 | func nx_putc(c: i64) -> i64 { let b: *u8 = sys_mmap(1); b[0] = c; sys_write(1, b, 1); return 0 } |
| 31 | func nx_puts(s: *u8) -> i64 { sys_write(1, s, nx_str_len(s)); return 0 } |
| 32 | func nx_put_i64(n: i64) -> i64 |
| 39 | func nx_body_off(resp: *u8, n: i64) -> i64 |
| 44 | func bput(buf: *u8, p: *i64, cap: i64, s: *u8) -> i64 { let n: i64 = nx_str_len(s); var i: i64 = 0; while i < n { if p[0] < cap { buf[p[0]] = s[i]; p[0] = p[0]+1 } i = i+1 } return 0 } calls 1: nx_str_len |
| 49 | func nx_polite_fetch(addr: *u8, path: *u8, plen: i64, host: *u8, hlen: i64, calls 1: nx_http_get_ua |
| 54 | func main() -> i64 |