topic: crawl
19 modules sharing the crawl name family (derived from the tree's prefix discipline).
The 'crawl' topic family in the Nishi sovereign ecosystem enables systematic and secure web traversal, indexing, and processing. The nx_crawl_bfs module drives autonomous link-following using breadth-first search, while nx_crawl_burst manages distributed crawling with budget control and shard integration. nx_crawl_https ensures robust, validated HTTPS navigation, forming the backbone of reliable data ingestion. Together, these modules support a cohesive pipeline from frontier extraction to document processing, ensuring comprehensive and trustworthy web coverage.
auto-narrated by the local model from this topic's module headers; links verified against the wiki index.
narrated overview -- maintained by the narration lane, module links verified against this wiki.
| module | description | lines | funcs |
|---|---|---|---|
| nx_crawl_bfs.nx | autonomous link-following crawler (frontier BFS). | 163 | 7 |
| nx_crawl_burst.nx | Coordinates distributed web crawling with governed budgets, lease atomicity, and shard merging into a unified index. | 96 | 4 |
| nx_crawl_callee_probe.nx | PER-CALLEE VmSize PROBE for the crawler's per-page pipeline (debt 1787082132). | 203 | 6 |
| nx_crawl_coverage_census.nx | HONEST access-coverage census for the sovereign crawler (operator 2026-07-04: | 97 | 5 |
| nx_crawl_doc.nx | per-page processing unit for the sovereign crawler. | 42 | 2 |
| nx_crawl_doc_test.nx | KAT for the crawl back-half on real CONTENT. | 76 | 1 |
| nx_crawl_frontier.nx | outbound-link (frontier) extraction for the crawler. | 115 | 5 |
| nx_crawl_frontier_test.nx | KAT for outbound-link extraction. | 43 | 1 |
| nx_crawl_https.nx | the hardened crawler on the REAL VALIDATED HTTPS web. | 138 | 6 |
| nx_crawl_main.nx | the deployable Nishi crawl+serve daemon. | 256 | 8 |
| nx_crawl_pace.nx | SOTA polite crawler pacing (operator 2026-07-05: "pacing is a great callout on all the | 440 | 24 |
| nx_crawl_pace_gate.nx | deterministic gate for the pacing POLICY (no network): honor-Retry-After, | 316 | 4 |
| nx_crawl_polite.nx | hardened, polite crawler (don't get blacklisted). | 162 | 7 |
| nx_crawl_preserve.nx | the CRAWL DRIVER: sweep a list of URLs, render-fetch each as a browser (browser headers | 72 | 4 |
| nx_crawl_run.nx | end-to-end crawl: fetch (real HTTP) -> extract -> dedup | 109 | 6 |
| nx_crawl_sufficiency.nx | ADAPTIVE STOP: "do we know enough to stop fetching?" -- the /compare/webscraping | 391 | 15 |
| nx_crawl_sufficiency_cli.nx | CLI over nx_crawl_sufficiency (the /compare/webscraping R5 lib): judge a set of | 74 | 5 |
| nx_crawl_sufficiency_gate.nx | GATE for nx_crawl_sufficiency (the /compare/webscraping R5 contract | 162 | 4 |
| nx_crawl_web.nx | REAL-WEB BFS crawler: "crawl whatever is out there", sovereign + polite. Combines what the | 198 | 13 |