code wiki / (root) / nx_swarm_job_evidence_t67.nx

nx_swarm_job_evidence_t67.nx

buildroot/runtime/nx_swarm_job_evidence_t67.nx

23643 B513 linesdepth 4pulls 5 transitivereach 1 importersview sourcekind tooltopic swarm
docsdependenciesstructsconstsfunctions

about

nx_swarm_job.nx -- SWARM FABRIC job engine (SF-R3): decompose a chunky GEN job into independent chunks and run them across opportunistic heterogeneous devices, straggler-proof + crash-safe. The north-star compute substrate ("unified supercomputer for gen, any device") -- GREENFIELD from OUR primitives, NOT a Ray/K3s clone. THE DESIGN (durable-execution + speculative-execution fused, shared-nothing): * DURABLE COMPLETION JOURNAL -- the ONLY thing that must survive a crash. A chunk is DONE iff a "DONE <cid>" record exists (framed-append floor = flock'd, torn-write-proof). You never re-run a completed chunk, ever. * TRANSIENT O_EXCL LEASES -- a chunk is LEASED iff a fresh "<lockdir>/<job>.<cid>.lease" lockfile exists (O_CREAT|O_EXCL = exactly one claimer wins, the dispatch_lease idiom). The lockfile is stamped with a DEADLINE; a contender STEALS an expired lease (straggler / crashed holder). A hub crash simply loses the lockfiles -> every in-flight chunk returns to PENDING = CORRECT. * SPECULATIVE HARVEST -- whichever node completes a chunk FIRST wins; a duplicate complete of an already-DONE chunk is IGNORED (idempotent). So a slow straggler racing its speculative re-lease never corrupts the result and never stalls the job (ray.wait's "any finishes" -- but journaled). COMPOSES (check-before-build 2026-07-15): the O_EXCL+stale-steal lease = the PROVEN nx_dispatch_lease mechanism (generalized fixed DL_STALE_SEC -> per-lease deadline); the journal floor = nx_framed_append; row/read/parse helpers = nx_swarm_lib. Placement (which node) = the SEPARATE nx_swarm_place organ, composed by the DRIVER (rule 9 single-responsibility: this organ queues; place picks; a script wires). submit <journal.log> <njobs> -- define the chunk universe 0..njobs-1 lease <journal.log> <lockdir> <job> <njobs> <ttl_s> -- claim the next PENDING/stale chunk (O_EXCL) complete <journal.log> <lockdir> <job> <cid> <node> -- durable DONE (idempotent; dup -> IGNORED) status <journal.log> <lockdir> <job> <njobs> -- done/leased/pending + COMPLETE verdict harvest <journal.log> <lockdir> <job> <njobs> <ttl> <node> [maxiter] -- in-proc drain (sim exec) [gate] -- self-gate, 8 tests incl liar-killers PATH LAW: journal must pass sb_path_ok (.log); lockdir must be a bare name or under /tmp/. license_tier: ORIGINAL expect_exit:0

dependencies 2 imports · 1 importers

nx_swarm_lib.nx nx_readcap_lib.nx nx_swarm_job_evidence_t67.nx nx_swarm_journal_evidence_gate_t67

imports: nx_swarm_lib.nxnx_readcap_lib.nx

imported by: nx_swarm_journal_evidence_gate_t67.nx

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main sj_eq sj_puts sb_path_ok fa_len sb_pint sj_submit fa_cat fa_catn sys_mmap nxa_die sys_write sys_exit nxa_lock_take nxa_lock_addr sys_write ↻ nxa_lock_give nxa_lock_addr ↻ nxa_report_overrun sys_write ↻ nxa_dump_printable sys_write ↻ nxa_dump_sizes sys_write ↻ fa_append sys_mmap ↻ sys_openat_append sys_flock fa_write_all sys_write ↻ sys_close sj_lockdir_ok sj_lease sj_read_journal rc_fill_file rc_fill sys_read sys_mmap ↻ sys_read ↻ rc_complete

structs

none

consts

33const SJ_EVIDENCE_IO:i64=0-20
34const SJ_EVIDENCE_TRUNCATED:i64=0-21
35const SJ_EVIDENCE_FRAME:i64=0-22
36const SJ_EVIDENCE_RECORD:i64=0-23
37const SJ_DECIMAL_MAX:i64=9223372036854775807
38const SJ_MAGIC_1970: i64 = 1970
39const SJ_MAGIC_100000: i64 = 100000
41const SJ_OEXCL: i64 = 193 // O_CREAT|O_EXCL|O_WRONLY
42const SJ_MODE: i64 = 420 // 0644
43const SJ_JBUF: i64 = 262144
44const SJ_REC: i64 = 256

functions

46func sj_puts(s: *u8) -> i64 { sys_write(1, s, fa_len(s)); return 0 }
called by 2: sj_gatemain
48func sj_eq(a: *u8, b: *u8) -> i64
called by 1: main
56func sj_lockdir_ok(p: *u8) -> i64
called by 1: main
77func sj_lease_path(lockdir: *u8, job: *u8, cid: i64, out: *u8) -> i64
called by 4: sj_leasesj_completesj_statussj_gate calls 2: fa_catfa_catn
89func sj_writeint_fd(fd: i64, v: i64) -> i64
called by 2: sj_lock_trysj_gate
102func sj_readint(path: *u8) -> i64
called by 2: sj_lock_trysj_lease_fresh
118func sj_lock_try(lockpath: *u8, deadline_sec: i64, now_sec: i64) -> i64
called by 1: sj_lease calls 3: sj_writeint_fdsj_readintsj_unlink
131func sj_lease_fresh(lockpath: *u8, now_sec: i64) -> i64
called by 1: sj_status calls 1: sj_readint
141func sj_done_record(buf:*u8,start:i64,end:i64)->i64
called by 1: sj_derive_done
159func sj_derive_done(jbuf:*u8,jn:i64,njobs:i64,done:*i64)->i64
called by 5: sj_leasesj_completesj_statussj_gatemain calls 1: sj_done_record
175func sj_evidence_refused(code:i64)->i64
called by 3: sj_leasesj_completesj_status calls 2: fa_catfa_catn
179func sj_read_journal(journal:*u8,jbuf:*u8)->i64
called by 6: sj_leasesj_completesj_statussj_journal_countsj_gatemain calls 2: rc_fill_filerc_complete
191func sj_submit(journal: *u8, njobs: i64) -> i64
called by 2: sj_gatemain calls 3: fa_catfa_catnfa_append
203func sj_lease(journal: *u8, lockdir: *u8, job: *u8, njobs: i64, ttl_sec: i64) -> i64
called by 4: sj_harvestsj_gatemainmain calls 5: sj_read_journalsj_evidence_refusedsj_derive_donesj_lease_pathsj_lock_try
228func sj_complete(journal: *u8, lockdir: *u8, job: *u8, njobs: i64, cid: i64, node: *u8) -> i64
called by 4: sj_harvestsj_gatemainmain calls 8: sj_read_journalsj_evidence_refusedsj_derive_donefa_catfa_catnfa_append+2
253func sj_status(journal: *u8, lockdir: *u8, job: *u8, njobs: i64) -> i64
called by 4: sj_harvestsj_gatemainmain calls 7: sj_read_journalsj_evidence_refusedsj_derive_donesj_lease_pathsj_lease_freshfa_cat+1
291func sj_harvest(journal: *u8, lockdir: *u8, job: *u8, njobs: i64, ttl_sec: i64, node: *u8, maxiter: i64) -> i64
called by 2: sj_gatemain calls 5: sj_leasesj_completefa_catfa_catnsj_status
317func sj_wfile_trunc(path: *u8) -> i64
called by 1: sj_gate
324func sj_journal_count(journal: *u8, needle: *u8) -> i64
called by 1: sj_gate calls 1: sj_read_journal
340func sj_gate() -> i64
called by 1: main calls 15: fa_catfa_catnsj_wfile_truncsj_submitsj_statussj_puts+9
439func main(argc: i64, argv: *i64) -> i64
calls 13: sj_eqsj_putssb_path_oksb_pintsj_submitsj_lockdir_ok+7