nx_llm_sched.nx
buildroot/runtime/nx_llm_sched.nx
about
nx_llm_sched.nx -- sovereign CONTINUOUS-BATCHING scheduler (Orca / vLLM
serving core). Requests ADMIT at any time (chunked prefill onto their own
paged sequence, forked-nothing: independent prompts) and JOIN the decode
batch mid-flight; each ROUND advances every active request by one token
via ONE batched forward (nx_f32_llama_v4b); finished requests leave the
batch and free their pages. The socket seat wraps THIS; the correctness
property gated by nx_llm_sched_gate is BATCH-INVARIANCE: a request's
bytes are BIT-IDENTICAL to its solo run, regardless of co-tenants or
admission timing.
genealogy_id: yu_2022_orca_continuous_batching + kwon_2023_pagedattention
lineage_id: substrate_llm_sched_v1
dependencies 12 imports · 2 importers
diagram shows first 10 each side; +2 more imports, +0 more importers in the complete lists below.
imports: nx_syscalls.nxnx_tier.nxnx_prng.nxnx_bpe.nxnx_f32_llm.nxnx_f32_llm_v4.nxnx_f32_sampler.nxnx_reasoning.nxnx_kvcache.nxnx_f32_attn_paged.nxnx_f32_llama_v4p.nxnx_f32_llama_v4b.nx
imported by: nx_llm_batch_serve.nxnx_llm_sched_gate.nx
structs
| 30 | struct NxLlmReq |
| 46 | struct NxLlmSched |
consts
| 27 | const NX_SCHED_MAX_REQS: nx_int = 16 |
| 44 | const NX_LLM_REQ_BYTES: nx_int = 96 |
| 52 | const NX_LLM_SCHED_BYTES: nx_int = 32 |
functions
| 54 | func nx_sched_new(rc: *NxReasonCfg, pool: *NxPagedPool) -> *NxLlmSched |
| 63 | func nx_sched_req(S: *NxLlmSched, i: nx_int) -> *NxLlmReq |
| 67 | func _sched_stop(rc: *NxReasonCfg, t: nx_int) -> nx_int |
| 73 | func _sched_emit(rc: *NxReasonCfg, R: *NxLlmReq, t: nx_int) -> i64 |
| 91 | func nx_sched_admit(S: *NxLlmSched, toks: *i64, nt: nx_int, seed: i64, |
| 134 | func nx_sched_round(S: *NxLlmSched) -> nx_int called by 2: mainmain calls 7: sys_mmapnx_sched_reqnx_f32_llm_forward_v4bnx_f32_sampler_sample_top_k_sched_stop_sched_emit+1 |
| 186 | func nx_sched_release(S: *NxLlmSched, slot: nx_int) -> i64 |