nx_bulk_ingest.nx

nx_bulk_ingest.nx -- the POLITE-FETCH -> BULK-CATALOG HAND-OFF (operator 2026-06-05): "a hand off function when its being polite that if we can download the whole catalog of research papers like arxiv that triggers like we were doing with the nishi search engine and we grow and grow our library so we can self serve more." When the team's polite single-fetch (nx_pipeline_fetch) lands on a page that is actually a HARVESTABLE REPOSITORY (a catalog/listing/OAI-PMH/sitemap), it HANDS OFF to bulk-catalog ingest: walk the catalog, mirror each doc into the Library (content-addressed dedup), so over time the team SELF-SERVES from its own Library and depends on the live web less. Two hard rules from the operator's prior laws: - STAY POLITE: the bulk walk runs at the SAME one-IP polite floor (pf_polite_floor) -- never a blast. - SERVE THE RAISED HANDS: prioritize repositories that fill the gaps the team raised its hand about. Convergent (K dry rounds = catalog exhausted), resumable, never infinite. license_tier: ORIGINAL