nx_ingest_batch.nx
buildroot/runtime/nx_ingest_batch.nx
about
nx_ingest_batch.nx -- the actual ingestion SYSTEM.
module: nishi-core.ingest.batch
depends: nishi-core.io.syscalls, nishi-core.io.dir,
nishi-core.data.value, nishi-core.data.value_parse_json
disk_kb: 6
capability: CORE_IO
wired_status: FULLY_WIRED
license_tier: PUBLIC_NISHI_SUBSTRATE
genealogy_id: nishi_build_the_system_cardinal_2026 +
unix_pipeline_composition_tradition +
wiremock_2011_http_stub_pattern
Brick #3 of the bits-up ingestion stack. No more hand-coded
per-fixture paths or curl loops. Composition:
nx_dir_list(dir_path, ...) -- enumerate files
-> for each NxDirRow with name ending in suffix:
sys_read_file(<dir>/<name>) -- load bytes
-> nx_value_parse_json -- bytes -> NxValue tree
-> caller-supplied extract callback emits one
JSONL row to the output file descriptor
-> aggregate verdict + count
"Caller-supplied" means: the batch driver itself is generic over
upstream-source shape. Today's smoke ships an extractor for
GBIF /v1/species/match responses, but any caller can drop in a
different field-extractor for a different upstream shape and the
batch substrate composes identically.
dependencies 5 imports · 2 importers
imports: syscalls.nxnx_dir.nxnx_value.nxnx_value_parse_json.nxnx_value_parse_csv.nx
imported by: nx_ingest_batch_test.nxnx_ingest_bulk_test.nx
structs
| 59 | struct NxIngestReport |
consts
| 37 | const NX_MAGIC_1024: i64 = 1024 |
| 38 | const NX_MAGIC_1048576: i64 = 1048576 |
| 42 | const NX_INGEST_OK: i64 = 1 |
| 43 | const NX_INGEST_DIR_OPEN_FAILED: i64 = 2 |
| 44 | const NX_INGEST_OUT_OPEN_FAILED: i64 = 3 |
| 45 | const NX_INGEST_BAD_ARGS: i64 = 4 |
| 46 | const NX_INGEST_NO_FILES_MATCHED: i64 = 5 |
| 72 | const NX_INGEST_REPORT_BYTES: i64 = 80 // 10 fields * 8 bytes |
| 180 | const NX_INGEST_MAX_PATH: i64 = 1024 |
| 224 | const NX_INGEST_MAX_ROWS_CAP: i64 = 8192 |
| 225 | const NX_INGEST_NAME_ARENA_BYTES: i64 = 1048576 // 1 MiB |
functions
| 48 | func nx_ingest_verdict_name(v: i64) -> *u8 |
| 76 | func nx_ingest_w(fd: i64, s: *u8, n: i64) -> i64 |
| 80 | func nx_ingest_w_int(fd: i64, value: i64) -> i64 called by 1: nx_ingest_emit_jsonl_flat |
| 123 | func nx_ingest_emit_jsonl_flat(fd: i64, obj: *NxValue) -> i64 |
| 182 | func nx_ingest_join_path( called by 1: nx_ingest_batch_run |
| 227 | func nx_ingest_batch_run( |