robots_txt.nx

robots_txt.nx -- build robots.txt files. Robots Exclusion Protocol (RFC 9309 -- finally standardised 2022 after 30 years of informal use). Served at /robots.txt at the site root; crawlers fetch it before any other URL on the site. Format: # comments User-agent: * Disallow: /private Disallow: /admin Allow: /private/public-page User-agent: Googlebot Disallow: /experimental Sitemap: https://example.com/sitemap.xml Groups: lines grouped under User-agent headers apply only to that agent. "*" matches everything without a dedicated group. Sitemap lines are global and can appear anywhere. Use cases: nishi-pages emits this at /robots.txt pointing at sitemap.xml and blocking /admin. Every site that wants to appear (or NOT appear) in search results needs it. Invariants: RT1 Output is plain ASCII; no escaping (robots.txt doesn't have a concept of encoding). RT2 Path values are literals -- caller is responsible for URL-encoding special characters if needed (rare). RT3 Stream builder: robots_agent -> robots_disallow* / robots_allow*, then robots_sitemap for global links.