nx_robots.nx

buildroot/runtime/nx_robots.nx

7384 B188 linesdepth 4pulls 4 transitivereach 6 importersview sourcekind librarytopic robots

docs dependencies structs consts functions

about

nx_robots.nx -- robots.txt parser + matcher (RFC 9309). module: nishi-core.search.robots depends: nx_str.nx capability: CORE_COMPUTE wired_status: FULLY_WIRED genealogy_id: rfc9309_robots_exclusion_protocol WHY (operator: don't get our IP blacklisted; and the not-god cardinal): a host's robots.txt is how the HOST governs crawling of ITS content. Honoring it is both polite (avoids bans) and exactly on-cardinal -- the host's rules over the host's content, which we respect rather than override. RFC 9309 group selection (most-specific user-agent, else '*') + longest-match Allow/Disallow precedence (Allow wins ties). Plus Crawl-delay extraction for politeness pacing. Prefix + '$' end-anchor matching (the common forms); full '*' wildcard is a refinement. Deterministic, integer-only.

dependencies 1 imports · 6 importers

imports: nx_str.nx

imported by: nx_crawl_https.nxnx_crawl_main.nxnx_crawl_polite.nxnx_media_import.nxnx_media_import_gate.nxnx_robots_test.nx

structs

none

consts

none

functions

21	func _rb_lc(c: i64) -> i64 { if c >= 0x41 { if c <= 0x5A { return c + 0x20 } } return c } called by 2: _rb_directive _rb_ua_token_matches
24	func _rb_line_end(robots: *u8, rlen: i64, ls: i64) -> i64 called by 3: _rb_has_specific nx_robots_allowed nx_robots_crawl_delay
33	func _rb_directive(robots: u8, ls: i64, le: i64, dir: u8) -> i64 called by 3: _rb_has_specific nx_robots_allowed nx_robots_crawl_delay calls 2: nx_str_len _rb_lc
45	func _rb_val_end(robots: *u8, vs: i64, le: i64) -> i64 called by 3: _rb_has_specific nx_robots_allowed nx_robots_crawl_delay
60	func _rb_ua_token_matches(robots: u8, vs: i64, ve: i64, ua: u8, ualen: i64) -> i64 called by 3: _rb_has_specific nx_robots_allowed nx_robots_crawl_delay calls 1: _rb_lc
75	func _rb_is_star(robots: *u8, vs: i64, ve: i64) -> i64 called by 3: _rb_has_specific nx_robots_allowed nx_robots_crawl_delay
82	func _rb_path_match(path: u8, plen: i64, robots: u8, vs: i64, ve: i64) -> i64 called by 1: nx_robots_allowed
96	func _rb_has_specific(robots: u8, rlen: i64, ua: u8, ualen: i64) -> i64 called by 2: nx_robots_allowed nx_robots_crawl_delay calls 5: _rb_line_end _rb_directive _rb_val_end _rb_is_star _rb_ua_token_matches
112	func nx_robots_allowed(robots: u8, rlen: i64, ua: u8, ualen: i64, path: *u8, plen: i64) -> i64 called by 3: main main main calls 7: _rb_has_specific _rb_line_end _rb_directive _rb_val_end _rb_is_star _rb_ua_token_matches+1
157	func nx_robots_crawl_delay(robots: u8, rlen: i64, ua: u8, ualen: i64) -> i64 called by 1: main calls 6: _rb_has_specific _rb_line_end _rb_directive _rb_val_end _rb_is_star _rb_ua_token_matches