code wiki / _hdl_build / nx_sitemap_extract.nx
nx_sitemap_extract.nx
buildroot/runtime/_hdl_build/nx_sitemap_extract.nx
about
nx_sitemap_extract.nx -- sitemap discovery (grow-capabilities 2026-07-06): the standard SOTA-crawler bulk
URL-discovery source. robots.txt "Sitemap:" lines point to sitemap XML; a sitemap is either a <urlset>
(page <loc>s) or a <sitemapindex> (sub-sitemap <loc>s, often .xml.gz). MEASURED: nyt/bbc/nhentai/wikipedia
all declare sitemaps. This extracts (a) Sitemap: URLs from robots.txt and (b) <loc> URLs from a sitemap --
both feed the crawler frontier so coverage isn't limited to <a>-linked pages. A parser, not a fetcher.
license_tier: ORIGINAL
dependencies 1 imports · 1 importers
imports: nx_syscalls.nx
imported by: nx_sitemap_extract_gate.nx
structs
| none |
consts
| 8 | const K_MAGIC_1000000: i64 = 1000000 |
functions
| 10 | func sm_lc(c: i64) -> i64 { if c >= 65 { if c <= 90 { return c + 32 } } return c } |
| 12 | func sm_ci_at(hay: *u8, off: i64, hlen: i64, b: *u8, blen: i64) -> i64 |
| 19 | func sm_copy_url(src: *u8, s: i64, e: i64, out: *u8, cap: i64) -> i64 |
| 39 | func sm_robots_sitemaps(rob: *u8, n: i64, out: *u8, slot: i64, max: i64) -> i64 |
| 64 | func sm_extract_locs(xml: *u8, n: i64, out: *u8, slot: i64, max: i64) -> i64 |
| 85 | func sm_is_subsitemap(u: *u8, ul: i64) -> i64 |