nx_colframe.nx
buildroot/runtime/nx_colframe.nx
about
nx_colframe.nx -- SOVEREIGN ZERO-COPY COLUMNAR INTERCHANGE (arrow-interchange, F-arrow). The Arrow idea,
built from the first byte up as our OWN format, not a clone: a single self-describing contiguous buffer
that carries N columns COLUMNAR (each column packed contiguously, 8-byte aligned), so any organ that
receives the buffer reads a whole column with ONE pointer cast -- zero per-value deserialization, zero
copy on the read side. Cross-organ dataset passing becomes "hand over a pointer + a length".
THE EXACT BYTE LAYOUT (all fields native i64 little-endian; the value IS the layout):
[0] magic = CF_MAGIC (identifies + versions the frame)
[1] version = CF_VERSION
[2] ncol
[3] nrows
then a SCHEMA table of ncol entries, 3 i64 each (i64-index 4 + i*3):
[+0] type (CF_T_I64 = 0; the v1 type)
[+1] name_off (byte offset to the column's NUL-terminated name)
[+2] data_off (byte offset to the column's data; 8-BYTE ALIGNED so the *i64 cast is valid+fast)
then the packed name strings, then each column's nrows*8 bytes of i64 data.
ZERO-COPY is provable, not asserted: cf_col(buf,i) returns buf+data_off -- a pointer INTO the shared
buffer. Called twice it returns the SAME address (a view, never a per-call materialisation), and an
analytics organ runs DIRECTLY on that view. EXCEED vs Arrow: 100% integer/fixed layout => bit-identical
frames across runs (Arrow's buffers carry padding/impl variance). license_tier: ORIGINAL No hw writes.
dependencies 1 imports · 5 importers
imports: nx_syscalls.nx
imported by: nx_colframe_gate.nxnx_distexec.nxnx_mrshuffle.nxnx_parexec.nxnx_vecexec.nx
structs
| none |
consts
| 24 | const CF_MAGIC: i64 = 5136462849 // "NXCF" epoch tag; distinctive, non-zero |
| 25 | const CF_VERSION: i64 = 1 |
| 26 | const CF_T_I64: i64 = 0 |
| 27 | const CF_HDR_I64: i64 = 4 // magic,version,ncol,nrows |
| 28 | const CF_SCH_I64: i64 = 3 // per-column schema words |
functions
| 30 | func cf_align8(x: i64) -> i64 { let r: i64 = x % 8; if r == 0 { return x } return x + (8 - r) } |
| 31 | func cf_slen(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n } called by 1: cf_encoded_bytes |
| 34 | func cf_encoded_bytes(names: *i64, ncol: i64, nrows: i64) -> i64 |
| 46 | func cf_encode(cols: *i64, names: *i64, ncol: i64, nrows: i64, out: *u8) -> i64 |
| 83 | func cf_valid(buf: *u8) -> i64 { let w: *i64 = buf as *i64; if w[0] == CF_MAGIC { if w[1] == CF_VERSION { return 1 } } return 0 } called by 1: main |
| 84 | func cf_ncol(buf: *u8) -> i64 { let w: *i64 = buf as *i64; return w[2] } |
| 85 | func cf_nrows(buf: *u8) -> i64 { let w: *i64 = buf as *i64; return w[3] } |
| 86 | func cf_type(buf: *u8, i: i64) -> i64 { let w: *i64 = buf as *i64; return w[CF_HDR_I64 + i * CF_SCH_I64] } called by 1: main |
| 87 | func cf_name(buf: *u8, i: i64) -> *u8 { let w: *i64 = buf as *i64; let off: i64 = w[CF_HDR_I64 + i * CF_SCH_I64 + 1]; return (((buf as i64) + off) as *u8) } called by 1: cf_col_index |
| 88 | func cf_data_off(buf: *u8, i: i64) -> i64 { let w: *i64 = buf as *i64; return w[CF_HDR_I64 + i * CF_SCH_I64 + 2] } |
| 90 | func cf_col(buf: *u8, i: i64) -> *i64 { return (((buf as i64) + cf_data_off(buf, i)) as *i64) } called by 10: mainde_agg_rangede_gb_rangemr_groupby_shufflepe_hash_rangevx_sum+4 calls 1: cf_data_off |
| 92 | func cf_col_index(buf: *u8, name: *u8) -> i64 |