code wiki / crawl
topic: crawl
15 modules sharing the crawl name family (derived from the tree's prefix discipline).
| module | description | lines | funcs |
|---|---|---|---|
| nx_crawl_bfs.nx | autonomous link-following crawler (frontier BFS). | 163 | 7 |
| nx_crawl_burst.nx | Coordinates distributed web crawling with governed budgets, lease atomicity, and shard merging into a unified index. | 96 | 4 |
| nx_crawl_coverage_census.nx | HONEST access-coverage census for the sovereign crawler (operator 2026-07-04: | 97 | 5 |
| nx_crawl_doc.nx | per-page processing unit for the sovereign crawler. | 42 | 2 |
| nx_crawl_doc_test.nx | KAT for the crawl back-half on real CONTENT. | 76 | 1 |
| nx_crawl_frontier.nx | outbound-link (frontier) extraction for the crawler. | 115 | 5 |
| nx_crawl_frontier_test.nx | KAT for outbound-link extraction. | 43 | 1 |
| nx_crawl_https.nx | the hardened crawler on the REAL VALIDATED HTTPS web. | 138 | 6 |
| nx_crawl_main.nx | the deployable Nishi crawl+serve daemon. | 256 | 8 |
| nx_crawl_pace.nx | SOTA polite crawler pacing (operator 2026-07-05: "pacing is a great callout on all the | 176 | 13 |
| nx_crawl_pace_gate.nx | deterministic gate for the pacing POLICY (no network): honor-Retry-After, | 84 | 4 |
| nx_crawl_polite.nx | hardened, polite crawler (don't get blacklisted). | 162 | 7 |
| nx_crawl_preserve.nx | the CRAWL DRIVER: sweep a list of URLs, render-fetch each as a browser (browser headers | 72 | 4 |
| nx_crawl_run.nx | end-to-end crawl: fetch (real HTTP) -> extract -> dedup | 109 | 6 |
| nx_crawl_web.nx | REAL-WEB BFS crawler: "crawl whatever is out there", sovereign + polite. Combines what the | 198 | 13 |