Nishi Family › Compare › LLM Inference and Serving (SOTA)
LLM Inference and Serving — the full field, measured, sourced.
12 engines, 42 axes, 7 categories. Competitor cells sourced from each project's own docs (researcher-banked 2026-07-09); Nishi cells measured by our gates. Frontier ladder: /compare/llm/frontier
| Capability | Nishi | vLLM | SGLang | TensorRT-LLM | llama.cpp | LMDeploy | TGI | MLC-LLM | Ollama | ExLlamaV2 | CTranslate2 | DeepSpeed-MII | Aphrodite | Notes / source |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Speed and scale (quantitative) | ||||||||||||||
| Single-stream CPU decode tok/s (0.5B-class) | 2.40 measured | n/a GPU-first | n/a GPU-first | n/a GPU-only | ~20-60 | n/a GPU-first | n/a CPU subpar | ~10-30 | ~20-60 | n/a GPU-only | ~10-40 int8 | n/a GPU-first | n/a GPU-first | Nishi MEASURED 2026-07-09 (Qwen2.5-0.5B, 16T, ~123x cumulative from 51 s/tok); ~20-60 = cited same-class llama.cpp 4-bit CPU, so Nishi is ~8-25x behind same-hardware SOTA |
| Largest model served (published) | 0.5B proven | 670B+ DeepSeek-V3 | 670B+ (96xH100 EP) | 670B+ Blackwell | 670B GGUF | 1TB Intern-S1-Pro | 180B Falcon | 70B-class | 670B library | 120B-class multi-GPU | 20B-class enc-dec | 530B MT-NLG | 670B+ vLLM-parity | from each project README/model table (LMDeploy lists Intern-S1-Pro 1TB); Nishi = Qwen2.5-0.5B end-to-end proven |
| Weight-quant formats | 4 GGUF reads + Q8_0 repack | 10+ (FP8 MXFP4 NVFP4 INT8/4 GPTQ AWQ GGUF...) | 5+ (FP4 FP8 INT4 AWQ GPTQ) | 6+ (FP8 FP4 INT4 AWQ) | 25+ (1.5- to 8-bit GGUF) | 6+ (AWQ GPTQ W4A16 MXFP4 KV int8/4) | 7 (bnb GPTQ EETQ AWQ Marlin fp8 NF4) | 3-4 group-quant | GGUF set | EXL2 2-8 bpw | 3 (INT8 INT16 AWQ) | 2-3 | 10+ vLLM-derived | counts from each README quant list; Nishi = Q4_K/Q5_0/Q6_K/Q8_0 read + in-memory Q8_0 repack (measured arc) |
| Model architectures | 2 (Qwen2-class GGUF + nofloat custom) | 200+ | wide (LLM embed reward diffusion) | 50+ | 150+ GGUF | 60+ LLM + 40+ VLM | 20+ optimized | 40+ | 100s library | Llama-family+ | 20+ (enc-dec incl Whisper) | 20+ | vLLM-parity | vLLM README claims 200+; LMDeploy README model table; Nishi loads Qwen2-class GGUF plus its own trained nofloat transformer |
| Hardware targets | 1 (x86-64 ELF) | 10+ (NV AMD x86 ARM PPC TPU Gaudi Ascend Apple...) | 6+ (NV AMD Xeon TPU Ascend) | 1 vendor (NVIDIA all gens) | 10+ (CPU CUDA Metal Vulkan HIP SYCL MUSA CANN OpenCL RPC) | 3 (NV V100+ Ascend Jetson) | 6 (NV AMD Inferentia Intel Gaudi TPU) | 8+ (CUDA ROCm Vulkan Metal WebGPU iOS Android) | 4 (Mac Win Linux; NV AMD) | 2 (NVIDIA ROCm) | 2 (x86 CPU NVIDIA) | 1-2 (NVIDIA) | 4+ (NV AMD CPU Win) | backend lists from each README; llama.cpp is the portability king of the natives, MLC of browser/mobile |
| Production scale (self-reported) | 1 home box (RTX5080 NAS, live site) | 2000+ contributors, industry standard | 400k+ GPUs, trillions tok/day | NVIDIA enterprise stack | ubiquitous local standard | InternLM production | HF prod, now maintenance mode | browser+mobile niche | dominant local runner | enthusiast standard | prod MT/Whisper stacks | dormant since ~2024 | community serving (Kobold lineage) | self-reported README claims (SGLang: 400k GPUs; TGI: CAUTION maintenance mode) |
| KV cache and memory | ||||||||||||||
| Paged / blocked KV cache | Part | Best | Yes | Yes | Part | Yes | Yes | Yes | Part | Yes | No | Yes | Yes | vLLM invented PagedAttention (SOSP 2023); Nishi nx_kvcache BUILT + gate GREEN (paged==contiguous bit-exact, fork+COW) but not yet wired into serving; ExLlamaV2 paged via flash-attn; llama.cpp unified cache |
| Continuous batching | No | Best | Best | Yes | Yes | Yes | Yes | Yes | Part | Part | Part | Yes | Yes | SGLang zero-overhead scheduler + vLLM set the bar; LMDeploy persistent batch; MII Dynamic SplitFuse; CT2 dynamic offline batching; Nishi serves one request at a time (top gap) |
| Prefix caching / reuse | No | Yes | Best | Yes | Part | Yes | Part | Part | Part | Yes | No | No | Yes | SGLang RadixAttention (5x claim, banked blog list); TRT-LLM KV Cache Reuse; LMDeploy automatic prefix caching; llama.cpp prompt cache |
| Chunked prefill | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Part | Part | Yes | No | Yes | Yes | Nishi WIRED + measured 2026-07-09 (one m=n forward, ~17x fewer prefill passes, TTFT collapse); LMDeploy dynamic split-and-fuse; MII SplitFuse |
| KV-cache quantization | No | Yes | Yes | Yes | Yes | Yes | No | Part | Yes | Best | No | No | Yes | ExLlamaV2 Q4 cache pioneer (README); LMDeploy online int8/int4 KV (README); llama.cpp q8/q4 KV flags |
| Speed techniques | ||||||||||||||
| Fused quantized-read GEMM | Part | Best | Yes | Best | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Marlin-class W4A16 is the SOTA bar; Nishi CPU-Marlin: __f32_i8dot32 SSE intrinsic blessed, Q8_0 path 10.2x measured, Q5/Q4 unpack proven compute-bound on CPU (honest partial) |
| Speculative decoding | Part | Yes | Best | Yes | Yes | Part | Yes | No | No | Yes | No | No | Yes | SGLang DFlash/Spec-V2 (2026-06 blog = current frontier); TGI claims ~2x; Nishi BUILT prompt-lookup + bit-identical gate, 58 pct accept, wall 0.94x = no net win yet (honest) |
| Compiled / graph execution | Part | Yes | Yes | Best | Yes | Yes | Yes | Best | Yes | Part | Yes | Yes | Yes | TRT engine compilation and MLC whole-model TVM compilation are the two compilation identities; vLLM CUDA graphs + torch.compile; Nishi is AOT-native by construction but has no GPU graphs |
| Optimized attention kernels | Part | Best | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Part | Yes | Yes | vLLM README lists FlashAttention FlashInfer TRTLLM-GEN FlashMLA Triton; LMDeploy FlashMLA; Nishi has own GQA offset-causal attention, scalar+SSE, no flash-class tiling |
| Parallelism and distribution | ||||||||||||||
| Tensor parallelism | No | Yes | Yes | Yes | Part | Yes | Yes | Part | No | Yes | Yes | Yes | Yes | TGI NCCL TP (README); CTranslate2 tensor parallelism (README); llama.cpp row/layer split |
| Pipeline parallelism | No | Yes | Yes | Yes | Yes | Yes | Part | No | No | No | No | Part | Yes | vLLM/SGLang/TRT READMEs; llama.cpp RPC layer split |
| Expert parallelism / MoE | No | Yes | Best | Yes | Yes | Yes | Part | Part | Yes | Part | No | Part | Yes | SGLang large-scale EP (96xH100 + GB200 blogs) leads; LMDeploy DeepEP/eplb (README); Nishi has no MoE (remaining census opportunity) |
| Prefill/decode disaggregation | No | Yes | Best | Yes | No | Yes | No | No | No | No | No | No | Part | SGLang GB200 P/D blogs (3.8x prefill 4.8x decode); LMDeploy via DLSlime+Mooncake (README); TRT via Dynamo |
| Multi-node serving | No | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No | Part | Yes | Yes | llama.cpp RPC; LMDeploy proxy server multi-machine (README); MII replicas + load balancing (README) |
| CPU multithread scaling | Yes | Part | Part | No | Best | No | No | Part | Yes | No | Yes | No | No | Nishi MEASURED 7.34x q4k matmul on 8T + 16T decode (sovereign pool, no pthread); llama.cpp AVX/AMX/NUMA is the CPU king; CT2 MKL/oneDNN/Ruy |
| Model I/O and sampling | ||||||||||||||
| GGUF loading | Yes | Yes | Part | No | Best | No | No | No | Yes | No | No | No | Yes | llama.cpp owns the format; Nishi nx_gguf loader + tensor-type census (Q4_K/Q5_0/Q6_K/Q8_0/F32) measured; Ollama/Aphrodite GGUF native |
| Own tokenizer (no external lib) | Yes | No | No | No | Yes | No | No | No | Yes | No | No | No | No | Nishi sovereign BPE verified == Qwen2 digit-split behavior (nx_tok_probe); llama.cpp own tokenizer; the rest ship HF tokenizers |
| Sampling suite | Part | Yes | Yes | Yes | Best | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | llama.cpp deepest (min-p mirostat DRY XTC ecosystem); TGI logits warpers (README); Nishi greedy + temperature + top-k measured, no top-p/penalties yet |
| Structured output (JSON/grammar) | No | Yes | Best | Yes | Yes | Yes | Yes | Yes | Yes | Part | No | No | Yes | SGLang compressed-FSM 3x JSON decoding (banked blog list); llama.cpp GBNF grammars; vLLM xgrammar/guidance (README); Ollama JSON schema |
| Tool / function calling | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Part | No | No | Yes | vLLM tool calling + reasoning parsers (README); LMDeploy TOOLS CALLING (README); Ollama tool calling (README) |
| Multimodal (vision) serving | No | Yes | Yes | Yes | Yes | Best | Part | Part | Yes | Part | No | No | Yes | LMDeploy VLM table is the largest (InternVL home, README); llama.cpp multimodal; Ollama vision models |
| Embeddings / rerank serving | Part | Yes | Yes | Part | Yes | Part | Part | No | Yes | No | Part | No | Yes | llama.cpp embedding + rerank (README); SGLang e5/gte/reward (README); Nishi sovereign SGNS embeddings power live in-corpus search but expose no embed-API endpoint |
| In-stack training / autograd | Yes | No | Part | No | Part | No | No | No | No | No | No | No | No | Nishi sovereign autograd + stable Q16 integer training recipe (trained real models: reader, neural image codec); SGLang = RL rollout backend for external trainers; the field is inference-only |
| Serving and API | ||||||||||||||
| OpenAI-compatible API | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Part | Yes | universal in the field (each README); Nishi serve organ is bespoke HTTP + MCP-first plane instead - an honest drop-in-adoption gap |
| Anthropic Messages API | No | Yes | Part | No | No | No | No | No | No | No | No | No | No | vLLM README explicitly ships Anthropic Messages API alongside OpenAI + gRPC |
| gRPC API | No | Yes | No | Yes | No | No | Yes | No | No | No | No | Yes | No | TGI Rust gRPC core (README); MII gRPC (README); TRT via Triton |
| Agent-native MCP tool surface | Yes | No | No | No | No | No | No | No | No | No | No | No | No | Nishi live sovereign MCP plane (compare/status/search tools served to agents today); absent from every banked engine README |
| Token streaming | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Part | Yes | Yes | TGI SSE (README); Ollama stream (README); Nishi serve returns full completions (gap) |
| Metrics / tracing | No | Yes | Yes | Yes | Yes | Part | Best | No | Part | No | No | Part | Yes | TGI Prometheus + OpenTelemetry distributed tracing (README) = the production bar; Nishi has logs only |
| Turnkey local install | Part | Part | Part | No | Yes | Part | Part | Yes | Best | Part | Yes | Part | Part | Ollama one-command install is the consumer bar (README); Nishi = sovereign ELF drop, zero deps, but self-hosted only |
| Sovereignty and determinism (Nishi lane) | ||||||||||||||
| Zero third-party runtime deps | Best | No | No | No | Yes | No | No | Part | Part | No | Part | No | No | Nishi: own language + compiler to x86 ELF, own HTTP/TLS, no libc/BLAS/PyTorch/CUDA; llama.cpp minimal C/C++ but on OS toolchain; the rest sit on PyTorch/CUDA/HF |
| Bit-exact deterministic decode | Best | No | No | No | Part | No | No | No | Part | No | Part | No | No | Nishi gates prove bit-identical decode (spec==plain, paged==contiguous, cross-run); batched GPU float engines are nondeterministic by construction; llama.cpp reproducible only single-config |
| Model trained WITHOUT float | Best | No | No | No | No | No | No | No | No | No | No | No | No | Nishi integer-only Q16 training (grad-clip + LR-decay recipe, measured) - no engine in the field claims this |
| Full-stack auditability | Best | No | No | No | Yes | No | No | Part | Part | Part | Part | No | No | Nishi: every line from compiler to syscall is in-house readable; llama.cpp compact C++ auditable but sits on drivers/CUDA beneath |
| Offline / air-gapped | Best | Part | Part | Part | Yes | Part | Part | Yes | Yes | Yes | Yes | Part | Part | Nishi never phones home by construction; llama.cpp/Ollama fully local (model pull optional); HF-hub-default engines can be configured offline |
| Actively developed (2026) | Yes | Best | Best | Yes | Best | Yes | No | Part | Yes | Part | Part | No | Yes | from banked READMEs: TGI = CAUTION maintenance mode recommending vLLM/SGLang; MII dormant; vLLM/SGLang/llama.cpp are the active frontier |
Generated by nx_swcompare_sota from knowledge/compare/llm.sota — quantitative axes measured/sourced; researcher-fed (nx_swcompare_research). Zero JS, zero trackers.