← Back to results

Bulk Repository Discovery and Polite Harvesting arXiv pr

ovides bulk access to its full-text corpus of about 9.2 terabytes via Amazon S3 requester-pays buckets where the downloader pays AWS egress and no extra load hits arXiv servers. arXiv bulk tar files are about 500 megabytes each and ship with XML manifests and checksums, giving content-hash dedup and resumable harvest. Other open bulk channels include OpenAlex full-data downloads, Common Crawl WARC WAT and WET files on S3, and the OAI-PMH metadata harvesting protocol. A harvester should detect that a URL is a harvestable repository or catalog, for example an OAI-PMH endpoint a sitemap a WARC index or an S3 bucket, then switch from single-fetch to bulk-catalog ingest. Politeness mechanisms are robots.txt crawl-delay, rate limiting, conditional GET with ETags, resumable harvest, and dedup by content hash. A listing or catalog page is recognized by a high link count where most links share one URL pattern, versus an ordinary article page with few links. Bulk ingest should stay polite by running the catalog walk at the same one-IP polite pace as ordinary fetch, continuous through a pipeline rather than a parallel blast. Harvest convergence stops after K consecutive rounds add no new documents, with a hard backstop so the walk is never infinite.