nx_rate_limiter.nx

nx_rate_limiter.nx -- token-bucket rate limiter for NX-INGEST. module: nishi-core.ingest.rate_limiter depends: nishi-core.io.syscalls, nishi-core.io.iso8601 disk_kb: 4 capability: CORE_IO license_tier: PUBLIC_NISHI_SUBSTRATE genealogy_id: token_bucket_algorithm_floyd_jacobson_1992 + rfc_6585_http_429_too_many_requests + aws_token_bucket_rate_limiter_pattern + nishi_ingestion_s_class_cardinal_2026 Polite-pool rate limiter for every ingestion source. Token bucket per RFC 6585 + Floyd-Jacobson 1992. Substrate Cardinal 14 graceful degradation: when bucket is empty, fetch waits or yields THROTTLED verdict (caller decides between block-and-wait vs return-control). ===== Why token bucket ========================================= Token bucket gives both: - average rate enforcement (replenish at rate R) - burst tolerance (capacity B allows short bursts up to B requests when upstream is healthy) Leaky bucket would enforce strict rate but disallow bursts — worse for real-world polite-pool patterns where upstreams tolerate burst-then-pause. ===== Polite-pool defaults ===================================== Per the nishi-library sources.toml convention + RFC 6585: - arXiv: 1 req / 3 sec (cap 3 tokens; refill 1 every 3s) - bioRxiv / medRxiv: 1 req / sec (cap 5; refill 1 every 1s) - USDA APIs: 1 req / sec (cap 5; refill 1 every 1s) - Kew SID: 0.5 req / sec (cap 2; refill 1 every 2s) - GBIF: 1 req / sec - OpenFarm: 1 req / sec All values in the source-adapter's SourceDescriptor.