code wiki / (root) / nx_llm_sched.nx

nx_llm_sched.nx

buildroot/runtime/nx_llm_sched.nx

6345 B193 linesdepth 15pulls 62 transitivereach 2 importersview sourcekind librarytopic llm
docsdependenciesstructsconstsfunctions

about

nx_llm_sched.nx -- sovereign CONTINUOUS-BATCHING scheduler (Orca / vLLM serving core). Requests ADMIT at any time (chunked prefill onto their own paged sequence, forked-nothing: independent prompts) and JOIN the decode batch mid-flight; each ROUND advances every active request by one token via ONE batched forward (nx_f32_llama_v4b); finished requests leave the batch and free their pages. The socket seat wraps THIS; the correctness property gated by nx_llm_sched_gate is BATCH-INVARIANCE: a request's bytes are BIT-IDENTICAL to its solo run, regardless of co-tenants or admission timing. genealogy_id: yu_2022_orca_continuous_batching + kwon_2023_pagedattention lineage_id: substrate_llm_sched_v1

dependencies 12 imports · 2 importers

nx_syscalls.nx nx_tier.nx nx_prng.nx nx_bpe.nx nx_f32_llm.nx nx_f32_llm_v4.nx nx_f32_sampler.nx nx_reasoning.nx nx_kvcache.nx nx_f32_attn_paged.nx nx_llm_sched.nx nx_llm_batch_serve.nx nx_llm_sched_gate.nx

diagram shows first 10 each side; +2 more imports, +0 more importers in the complete lists below.

imports: nx_syscalls.nxnx_tier.nxnx_prng.nxnx_bpe.nxnx_f32_llm.nxnx_f32_llm_v4.nxnx_f32_sampler.nxnx_reasoning.nxnx_kvcache.nxnx_f32_attn_paged.nxnx_f32_llama_v4p.nxnx_f32_llama_v4b.nx

imported by: nx_llm_batch_serve.nxnx_llm_sched_gate.nx

structs

30struct NxLlmReq
46struct NxLlmSched

consts

27const NX_SCHED_MAX_REQS: nx_int = 16
44const NX_LLM_REQ_BYTES: nx_int = 96
52const NX_LLM_SCHED_BYTES: nx_int = 32

functions

54func nx_sched_new(rc: *NxReasonCfg, pool: *NxPagedPool) -> *NxLlmSched
called by 2: mainmain calls 1: sys_mmap
63func nx_sched_req(S: *NxLlmSched, i: nx_int) -> *NxLlmReq
67func _sched_stop(rc: *NxReasonCfg, t: nx_int) -> nx_int
73func _sched_emit(rc: *NxReasonCfg, R: *NxLlmReq, t: nx_int) -> i64
91func nx_sched_admit(S: *NxLlmSched, toks: *i64, nt: nx_int, seed: i64,
134func nx_sched_round(S: *NxLlmSched) -> nx_int
186func nx_sched_release(S: *NxLlmSched, slot: nx_int) -> i64