nx_urlcanon.nx

nx_urlcanon.nx -- URL canonicalization for crawl de-duplication. module: nishi-core.search.urlcanon depends: nx_str.nx capability: CORE_COMPUTE wired_status: FULLY_WIRED WHY: the same resource is reachable under many URL spellings (host-case, #fragment, default :80/:443). Canonicalizing before the visited-set check means we DON'T re-fetch the same page under a cosmetic variant -- saving crawl budget and, crucially, not hammering hosts (-> not getting blacklisted). RFC 3986 syntax-based normalization (the safe subset): * lowercase scheme + host (case-insensitive per RFC 3986 sec 6.2.2.1) * drop the fragment (never sent to the origin server) * drop the default port (:80 for http, :443 for https) Path + query are preserved verbatim (case- and order-significant). Tracking- param stripping is a queued refinement.