code wiki / _hdl_build / nx_pipeline_fetch.nx
nx_pipeline_fetch.nx
buildroot/runtime/_hdl_build/nx_pipeline_fetch.nx
about
nx_pipeline_fetch.nx -- the team's LIVE-FETCH SCHEDULING POLICY, encoding the operator's architecture law
(2026-06-05): "we cant pull like you as we dont have lots of ip addresses to cycle through so we win in
parallelization through the PROCESS not like you and how you do it all at once on each step before moving on."
The constraint: from ONE IP, fetches CANNOT parallelize -- hammering N requests at once from one address gets
rate-limited / bot-blocked. So the fetch stage is serial, paced at POLITE_MS (respect the host). Claude wins
breadth by fanning out across MANY IPs all-at-once per step (a BARRIER between steps). The team can't.
The team's WIN = PIPELINE (process) parallelism: while source i is being EXTRACTED and source i-1 CATALOGED,
source i+1 is already FETCHING. The CPU stages overlap UNDER the unavoidable polite fetch pacing, so the only
real floor is the pacing itself + one item's tail -- NOT n x (fetch+extract+catalog) as a naive barrier would be.
This module computes the wall-clock of three schedules so the policy is a MEASURED fact, not a claim:
pf_claude_barrier -- many IPs, fan-out-all-per-step (what Claude does; needs many IPs the team lacks)
pf_team_barrier -- one IP, naive: all fetches, THEN all extracts, THEN all catalogs (the WRONG team policy)
pf_team_pipeline -- one IP, process-parallel overlap (the RIGHT team policy: hide CPU under fetch pacing)
license_tier: ORIGINAL
dependencies 1 imports · 2 importers
imports: nx_syscalls.nx
imported by: nx_bulk_ingest.nxnx_pipeline_fetch_test.nx
structs
| none |
consts
| none |
functions
| 20 | func pf_max3(a: i64, b: i64, c: i64) -> i64 called by 1: pf_team_pipeline |
| 27 | func pf_claude_barrier(fetch_latency: i64, extract_ms: i64, catalog_ms: i64) -> i64 called by 1: main |
| 33 | func pf_team_barrier(n: i64, fetch_pace: i64, extract_ms: i64, catalog_ms: i64) -> i64 |
| 41 | func pf_team_pipeline(n: i64, fetch_pace: i64, extract_ms: i64, catalog_ms: i64) -> i64 |
| 48 | func pf_pipeline_saving(n: i64, fetch_pace: i64, extract_ms: i64, catalog_ms: i64) -> i64 |
| 54 | func pf_polite_floor(n: i64, fetch_pace: i64) -> i64 { return n * fetch_pace } |
| 58 | func pf_is_polite_optimal(n: i64, fetch_pace: i64, extract_ms: i64, catalog_ms: i64) -> i64 |