nx_parquet_lib.nx

nx_parquet_lib.nx -- THE SOVEREIGN PARQUET READER (search R0d be_bright: the BRIGHT benchmark ships as parquet). WHY. BRIGHT (huggingface xlangai/BRIGHT, cc-by-4.0, search.refs brightds26) ships twelve documents files and twelve examples files as Apache Parquet, and on 2026-09-14 the estate held no parquet, thrift-compact or snappy reader (nx_catalog ABSENT on every name tried; a corpus-complete grep over 25,141 sources found the word only in prose). A dataset the estate cannot open is a bar the estate cannot measure, so this lib is the first byte of R0d. It reads what those files carry -- BYTE_ARRAY (string) columns, flat or nested as lists, PLAIN or dictionary encoded, uncompressed or snappy, data pages V1 or V2 -- and refuses everything else BY NAME (PQ_E_UNSUPPORTED with the offending enum value left in pq[PQ_WHAT]), never a silent zero or a quietly truncated column. THE FORMAT, from the mirrored primary sources read 2026-09-14 (search.refs pqthrift26 pqenc26 thriftcompact26 snappyfmt26): file = PAR1 .. pages .. FileMetaData (thrift compact) .. i32 LE footer length .. PAR1 thrift compact: field header byte = (id delta << 4) | type, delta 0 => a zigzag i16 follows; ints are zigzag varints; binary = varint length + bytes; list header = (size << 4) | element type, size 15 => a varint size follows; a bool list element is one byte (1 or 2); a bool struct field carries its value in the type nibble; STOP = 0. schema = depth-first list, root first. A leaf's max definition level = its non-REQUIRED ancestors plus itself, its max repetition level = its REPEATED ancestors plus itself (the Dremel shredding rules): a value is present when its definition level equals the max, and a new row starts at repetition level 0. page = PageHeader (thrift) then compressed_page_size bytes. DATA_PAGE (V1) payload = [repetition levels] [definition levels] [values], each level block an i32 LE byte length then hybrid runs; DATA_PAGE_V2 = levels first, never compressed, lengths in the header, then the values (compressed iff is_compressed); DICTIONARY_PAGE = PLAIN values. Dictionary-encoded data = one byte of bit width then hybrid runs of indices. hybrid = varint header; even => an RLE run of header >> 1 copies of one value (ceil(bw / 8) bytes LE); odd => header >> 1 groups of eight bit-packed values, LSB first. snappy = varint uncompressed length, then tagged elements: 00 literal (length in the tag up to 60, else 1..4 LE bytes follow), 01 copy with a 1-byte offset (length 4..11, 11-bit offset), 10 copy with a 2-byte offset, 11 copy with a 4-byte offset (both length 1..64). EVERY READ IS BOUNDED by the buffer it reads and every write by an arena sized from the column chunk's own total_uncompressed_size (the format's bound on decoded page bytes): a decompressor is a parser pointed at foreign bytes (nx_inflate's law), so a truncated stream, a back-reference before the buffer, a level count that disagrees with the page header and a footer longer than the file each REFUSE with a distinct negative code. COMPOSES sys_read_file (buffer sized from the file, no cap), sys_mmap, sys_munmap, sys_write. Allocation happens once per open and once per column chunk read, never per page or per value. No hw writes (Rule 26). license_tier: ORIGINAL