code wiki / _hdl_build / nx_pipeline_fetch.nx

nx_pipeline_fetch.nx

buildroot/runtime/_hdl_build/nx_pipeline_fetch.nx

4366 B64 linesdepth 2pulls 2 transitivereach 3 importersview sourcekind librarytopic pipeline
docsdependenciesstructsconstsfunctions

about

nx_pipeline_fetch.nx -- the team's LIVE-FETCH SCHEDULING POLICY, encoding the operator's architecture law (2026-06-05): "we cant pull like you as we dont have lots of ip addresses to cycle through so we win in parallelization through the PROCESS not like you and how you do it all at once on each step before moving on." The constraint: from ONE IP, fetches CANNOT parallelize -- hammering N requests at once from one address gets rate-limited / bot-blocked. So the fetch stage is serial, paced at POLITE_MS (respect the host). Claude wins breadth by fanning out across MANY IPs all-at-once per step (a BARRIER between steps). The team can't. The team's WIN = PIPELINE (process) parallelism: while source i is being EXTRACTED and source i-1 CATALOGED, source i+1 is already FETCHING. The CPU stages overlap UNDER the unavoidable polite fetch pacing, so the only real floor is the pacing itself + one item's tail -- NOT n x (fetch+extract+catalog) as a naive barrier would be. This module computes the wall-clock of three schedules so the policy is a MEASURED fact, not a claim: pf_claude_barrier -- many IPs, fan-out-all-per-step (what Claude does; needs many IPs the team lacks) pf_team_barrier -- one IP, naive: all fetches, THEN all extracts, THEN all catalogs (the WRONG team policy) pf_team_pipeline -- one IP, process-parallel overlap (the RIGHT team policy: hide CPU under fetch pacing) license_tier: ORIGINAL

dependencies 1 imports · 2 importers

nx_syscalls.nx nx_pipeline_fetch.nx nx_bulk_ingest.nx nx_pipeline_fetch_test.nx

imports: nx_syscalls.nx

imported by: nx_bulk_ingest.nxnx_pipeline_fetch_test.nx

structs

none

consts

none

functions

20func pf_max3(a: i64, b: i64, c: i64) -> i64
called by 1: pf_team_pipeline
27func pf_claude_barrier(fetch_latency: i64, extract_ms: i64, catalog_ms: i64) -> i64
called by 1: main
33func pf_team_barrier(n: i64, fetch_pace: i64, extract_ms: i64, catalog_ms: i64) -> i64
41func pf_team_pipeline(n: i64, fetch_pace: i64, extract_ms: i64, catalog_ms: i64) -> i64
48func pf_pipeline_saving(n: i64, fetch_pace: i64, extract_ms: i64, catalog_ms: i64) -> i64
54func pf_polite_floor(n: i64, fetch_pace: i64) -> i64 { return n * fetch_pace }
58func pf_is_polite_optimal(n: i64, fetch_pace: i64, extract_ms: i64, catalog_ms: i64) -> i64