nx_log_chunked.nx
buildroot/runtime/nx_log_chunked.nx
about
nx_log_chunked.nx -- append-only Merkle-linked chunk log.
Sovereign storage substrate for live-fire trace data. Foundation
for the upcoming nx_trace_query and substrate-native dashboards.
Architecture (per QMDB 2025, Trillian, IPFS Merkle-DAG, Git):
* Spans land in an in-memory chunk buffer.
* When the chunk fills (or is explicitly flushed), it is
SHA-256 hashed via the existing nx_sha256 primitive.
* The chunk's hash + the PARENT chunk's hash + chunk metadata
form a Merkle-linked log: any tamper of an earlier chunk
invalidates every subsequent chunk hash.
* A separate manifest table records (chunk_hash, n_spans,
first_trace_id, last_trace_id, written_at) per chunk for
replay + integrity verification.
Why not SQLite / LMDB / RocksDB:
* each ships ~250KB+ of C; not sovereign.
* the LSM-tree write path can be expressed in NishiLang directly
(memtable = current chunk, segment files = flushed chunks).
* Bloom filters + segment indexes queued for v2.
Why not Parquet/Arrow:
* spans are small irregular shapes; columnar wins are tiny here.
* column store can land later as a separate projection (CQRS:
log is source of truth; columns are derived).
Wire format inside chunks:
* each span is CBOR-encoded via nx_cbor.nx (machine-readable
binary; spec-compliant; multiple-language decoders exist).
* the JSONL human-readable wire stays on nx_trace_emit.nx.
genealogy_id: oneill_1996_lsm_tree + merkle_1979 + trillian_transparency +
qmdb_2025_arxiv_2501_05262
lineage_id: sovereign_log_v1
dependencies 4 imports · 1 importers
imports: nx_syscalls.nxnx_tier.nxnx_sha256.nxnx_cbor.nx
imported by: nx_log_chunked_test.nx
structs
| 92 | struct ChunkedLog |
consts
| 49 | const NX_LC_HASH_BYTES: nx_int = 32 // SHA-256 |
| 50 | const NX_LC_DEFAULT_CHUNK: nx_int = 4096 // bytes per chunk before flush |
| 56 | const NX_LC_VERIFY_VALID: nx_int = 0 |
| 57 | const NX_LC_VERIFY_HASH_MISMATCH: nx_int = 1 // chunk bytes don't hash to recorded hash |
| 58 | const NX_LC_VERIFY_PARENT_BROKEN: nx_int = 2 // recorded parent hash != prior chunk's hash |
| 59 | const NX_LC_VERIFY_ORDER_REVERSED: nx_int = 3 // chunk index decreased |
| 60 | const NX_LC_VERIFY_N_VERDICTS: nx_int = 4 |
| 81 | const NX_LC_REC_F_INDEX: nx_int = 0 |
| 82 | const NX_LC_REC_F_N_SPANS: nx_int = 1 |
| 83 | const NX_LC_REC_F_WRITTEN_AT: nx_int = 2 |
| 84 | const NX_LC_REC_F_HASH_OFF: nx_int = 3 |
| 85 | const NX_LC_REC_F_PARENT_HASH_OFF: nx_int = 4 |
| 86 | const NX_LC_REC_F_FIRST_TRACE: nx_int = 5 |
| 87 | const NX_LC_REC_F_LAST_TRACE: nx_int = 6 |
| 88 | const NX_LC_REC_FIELDS: nx_int = 7 |
| 106 | const NX_LC_LOG_BYTES: nx_int = 88 // 11 fields * 8 |
functions
| 62 | func nx_lc_verify_verdict_is_valid(v: nx_int) -> nx_int called by 1: main |
| 110 | func nx_lc_alloc(chunk_cap: nx_int, manifest_cap: nx_int) -> *ChunkedLog |
| 140 | func nx_lc_flush(log: *ChunkedLog, now_ms: nx_int) -> nx_int |
| 176 | func nx_lc_append_span(log: *ChunkedLog, |
| 204 | func _nx_lc_hash_eq(a: *u8, b: *u8) -> nx_int called by 1: nx_lc_verify_chunk_bytes |
| 218 | func nx_lc_rec_get(log: *ChunkedLog, chunk_index: nx_int, field: nx_int) -> nx_int called by 1: main |
| 233 | func nx_lc_verify_chain(log: *ChunkedLog) -> nx_int called by 1: main |
| 262 | func nx_lc_verify_chunk_bytes(log: *ChunkedLog, chunk_index: nx_int, |