code wiki / (root) / nx_distexec.nx

nx_distexec.nx

buildroot/runtime/nx_distexec.nx

8513 B208 linesdepth 3pulls 3 transitivereach 3 importersview sourcekind tool
docsdependenciesstructsconstsfunctions

about

nx_distexec.nx -- DISTRIBUTED EXECUTION ALGEBRA (distributed-compute, F-dist). Spark's essence is not the cluster -- it is the PARTITION -> MAP -> MERGE algebra: split the data into partitions, compute a PARTIAL result per partition, then MERGE the partials into the whole answer. That algebra is provable and buildable single-process from the first byte up; only its PARALLEL execution (a partition per worker core/node) needs the threading/swarm arc. This organ is the algebra, proven PARTITION-INVARIANT: the merged result is byte-identical for ANY partition count P, which is exactly the correctness a scheduler relies on to run partitions anywhere. Composes nx_vecexec (each partition aggregates vectorized) over the zero-copy nx_colframe. Honest: this is the algebra + a sequential executor; real multi-core/multi-node parallelism is the named finer gap. license_tier: ORIGINAL No hardware writes (Rule 26).

dependencies 2 imports · 2 importers

nx_syscalls.nx nx_colframe.nx nx_distexec.nx nx_distexec_gate.nx nx_parexec.nx

imports: nx_syscalls.nxnx_colframe.nx

imported by: nx_distexec_gate.nxnx_parexec.nx

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main sys_mmap de_atoi de_demo sys_mmap ↻ cf_encoded_bytes cf_slen cf_align8 cf_encode cf_align8 ↻ de_run_agg cf_nrows sys_mmap ↻ de_bound de_agg_range cf_col cf_data_off de_merge de_raw de_num sys_mmap ↻ sys_write

structs

none

consts

12const DE_MAGIC_4242: i64 = 4242
13const DE_MAGIC_1103515245: i64 = 1103515245
14const DE_MAGIC_12345: i64 = 12345
15const DE_MAGIC_100000: i64 = 100000
16const DE_MAGIC_4000000: i64 = 4000000
18const DE_VEC: i64 = 1024
19const DE_MAXK: i64 = 256 // group-by key domain cap (buckets)
20const DE_OUT: i64 = 8192
21const DE_ZERO: i64 = 48
23const DE_SUM: i64 = 0
24const DE_CNT: i64 = 1
25const DE_MIN: i64 = 2
26const DE_MAX: i64 = 3

functions

29func de_agg_range(frame: *u8, ci: i64, lo: i64, hi: i64, out4: *i64) -> i64
called by 2: de_run_aggpe_agg_parallel calls 1: cf_col
57func de_merge(a: *i64, b: *i64) -> i64
73func de_bound(n: i64, P: i64, p: i64) -> i64
78func de_run_agg(frame: *u8, ci: i64, P: i64, out4: *i64) -> i64
99func de_gb_range(frame: *u8, keyci: i64, valci: i64, lo: i64, hi: i64, buckets: *i64, K: i64) -> i64
called by 1: de_run_groupby calls 1: cf_col
110func de_run_groupby(frame: *u8, keyci: i64, valci: i64, K: i64, P: i64, buckets: *i64) -> i64
133func de_raw(out: *u8, o: i64, s: *u8) -> i64 { var i: i64 = 0; while s[i] != (0 as u8) { out[o] = s[i]; o = o + 1; i = i + 1 } return o }
called by 1: de_demo
134func de_num(out: *u8, o: i64, v: i64) -> i64
called by 1: de_demo calls 1: sys_mmap
146func de_atoi(s: *u8) -> i64 { var v: i64 = 0; var i: i64 = 0; var go: i64 = 1; while go == 1 { let c: i64 = s[i] as i64; if c < 48 { go = 0 } else { if c > 57 { go = 0 } else { v = v * 10 + (c - 48); i = i + 1 } } } return v }
called by 1: main
149func de_demo(nrows: i64, P: i64, out: *u8) -> i64
197func main(argc: i64, argv: *i64) -> i64