code wiki / (root) / nx_lean_dir_ingest.nx

nx_lean_dir_ingest.nx

buildroot/runtime/nx_lean_dir_ingest.nx

3444 B87 linesdepth 8pulls 20 transitivereach 0 importersview sourcekind tooltopic lean
docsdependenciesstructsconstsfunctions

about

nx_lean_dir_ingest.nx -- ingest every Lean decl in the corpus at nxc2/_offc/lean_corpus.txt (assembled by nx_ingest_dir.sh from a caller-pointed directory) and emit shards under /tmp/nx_ingest_lean-*. Thin driver: read corpus -> nx_lean_ingest_corpus -> offer through nx_ingest_runner via nx_lean_stream_offer_all -> close. The per-source bridge does the JSONL row layout; the runner handles dedup / disk_budget / shard rotation / checkpoint / watchdog. "Point at X" capability: only the bash side (nx_ingest_dir.sh) needs to know which directory to read from. Everything else is identical across the 5 source types. genealogy_id: substrate_self_hosting_synth_2026_05_14 + point_at_x_ingester lineage_id: per_source_dir_driver

dependencies 7 imports · 0 importers

nx_syscalls.nx nx_runtime.nx nx_tier.nx nx_lean_ingest.nx nx_ingest_runner.nx nx_lean_stream_ingest.nx nx_bloom_capacity.nx nx_lean_dir_ingest.nx

imports: nx_syscalls.nxnx_runtime.nxnx_tier.nxnx_lean_ingest.nxnx_ingest_runner.nxnx_lean_stream_ingest.nxnx_bloom_capacity.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main sys_mmap sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close sys_write print_i64 sys_mmap ↻ itoa sys_mmap ↻ sys_write ↻ nx_lean_ingest_corpus nx_lean_db_alloc sys_mmap ↻ sys_mmap ↻ nx_lean_next_token nx_lean_skip_ws nx_lex_is_ws nx_ascii_is_id_start nx_ascii_is_alpha nx_ascii_is_lower nx_ascii_is_upper nx_lex_is_id_cont_qualifie nx_ascii_is_id_start ↻ nx_lean_decl_at nx_ingest_run_new sys_mmap ↻ nx_disk_budget_new sys_mmap ↻ nx_shard_writer_open sys_mmap ↻ nx_str_len nx_str_cpy nx_shardw_open_shard sys_mmap ↻ nx_shardw_path nx_str_cpy ↻

structs

none

consts

31const NX_LEAN_DIR_DISK_BUDGET: nx_size = 1073741824 // 1 GiB
32const NX_LEAN_DIR_SHARD_BYTES: nx_size = 104857600 // 100 MiB
33const NX_LEAN_DIR_BLOOM_CAPACITY: nx_int = 1000000 // size for 1M items
34const NX_LEAN_DIR_WATCHDOG_MS: nx_int = 60000

functions

36func main() -> nx_exit
calls 11: sys_mmapsys_read_filesys_writeprint_i64nx_lean_ingest_corpusnx_ingest_run_new+5