nx_q8_0_from_q5_0.nx
buildroot/runtime/nx_q8_0_from_q5_0.nx
about
nx_q8_0_from_q5_0.nx -- re-quantize a Q5_0 tensor to Q8_0 IN MEMORY.
THE KEY MOVE (2026-07-08): Q5_0's nibble+qh unpack is what made a per-token
Q5_0 SIMD dequant-dot COMPUTE-bound (2x slower). But Q5_0 -> Q8_0 is EXACT
(the 5-bit values -16..15 fit in int8, same per-block f16 scale d), so we
repack ONCE AT LOAD -- amortizing the unpack -- and the forward then runs
the TRIVIAL Q8_0 SIMD dequant-dot (__f32_i8dot32, PROVEN 10.2x) per token.
The unpack uses the blessed __q5_unpack32 intrinsic itself.
Q5_0 block (22B): d(f16) + qh(u32) + qs(16B). Q8_0 block (34B): d(f16) +
32 int8. Repack: copy d; __q5_unpack32(qh+qs) -> the 32 int8 = q5-16.
lineage_id: q8_0_from_q5_0_v1
dependencies 4 imports · 2 importers
imports: nx_syscalls.nxnx_le.nxnx_q5_0_to_f32.nxnx_q8_0_to_f32.nx
imported by: nx_f32_llama_layer_lazy_load.nxnx_natq5_probe.nx
structs
| none |
consts
| none |
functions
| 19 | func _q5repack_consts() -> *u8 |
| 41 | func nx_q8_0_from_q5_0(q5buf: *u8, q5off: i64, n_values: i64, out_q8: *u8) -> i64 |