LLM Inference and Serving

Nishi REMEASURED 2026-07-17 (F-004, Qwen2.5-0.5B i8, 16T healthy 17-thread build): decode ~15 tok/s bench / 23 warm -- the prior 2.40 was PREFILL mislabeled as decode; vs cited llama.cpp 20-60 = ~1-4x behind on decode (prefill ~9 tok/s remains the wall)

vLLM n/a GPU-firstSGLang n/a GPU-firstTensorRT-LLM n/a GPU-onlyllama.cpp ~20-60LMDeploy n/a GPU-firstTGI n/a CPU subparMLC-LLM ~10-30Ollama ~20-60ExLlamaV2 n/a GPU-onlyCTranslate2 ~10-40 int8DeepSpeed-MII n/a GPU-firstAphrodite n/a GPU-first

from each project README/model table (LMDeploy lists Intern-S1-Pro 1TB); Nishi = Qwen2.5-0.5B end-to-end proven

vLLM 670B+ DeepSeek-V3SGLang 670B+ (96xH100 EP)TensorRT-LLM 670B+ Blackwellllama.cpp 670B GGUFLMDeploy 1TB Intern-S1-ProTGI 180B FalconMLC-LLM 70B-classOllama 670B libraryExLlamaV2 120B-class multi-GPUCTranslate2 20B-class enc-decDeepSpeed-MII 530B MT-NLGAphrodite 670B+ vLLM-parity

counts from each README quant list; Nishi = Q4_K/Q5_0/Q6_K/Q8_0 read + in-memory Q8_0 repack (measured arc) [gptq2022] [awq2023]

vLLM 10+ (FP8 MXFP4 NVFP4 INT8/4 GPTQ AWQ GGUF...)SGLang 5+ (FP4 FP8 INT4 AWQ GPTQ)TensorRT-LLM 6+ (FP8 FP4 INT4 AWQ)llama.cpp 25+ (1.5- to 8-bit GGUF)LMDeploy 6+ (AWQ GPTQ W4A16 MXFP4 KV int8/4)TGI 7 (bnb GPTQ EETQ AWQ Marlin fp8 NF4)MLC-LLM 3-4 group-quantOllama GGUF setExLlamaV2 EXL2 2-8 bpwCTranslate2 3 (INT8 INT16 AWQ)DeepSpeed-MII 2-3Aphrodite 10+ vLLM-derived

vLLM README claims 200+ [vllm-docs]; LMDeploy README model table; Nishi loads Qwen2-class GGUF plus its own trained nofloat transformer

vLLM 200+SGLang wide (LLM embed reward diffusion)TensorRT-LLM 50+llama.cpp 150+ GGUFLMDeploy 60+ LLM + 40+ VLMTGI 20+ optimizedMLC-LLM 40+Ollama 100s libraryExLlamaV2 Llama-family+CTranslate2 20+ (enc-dec incl Whisper)DeepSpeed-MII 20+Aphrodite vLLM-parity

backend lists from each README; llama.cpp is the portability king of the natives, MLC of browser/mobile

vLLM 10+ (NV AMD x86 ARM PPC TPU Gaudi Ascend Apple...)SGLang 6+ (NV AMD Xeon TPU Ascend)TensorRT-LLM 1 vendor (NVIDIA all gens)llama.cpp 10+ (CPU CUDA Metal Vulkan HIP SYCL MUSA CANN OpenCL RPC)LMDeploy 3 (NV V100+ Ascend Jetson)TGI 6 (NV AMD Inferentia Intel Gaudi TPU)MLC-LLM 8+ (CUDA ROCm Vulkan Metal WebGPU iOS Android)Ollama 4 (Mac Win Linux; NV AMD)ExLlamaV2 2 (NVIDIA ROCm)CTranslate2 2 (x86 CPU NVIDIA)DeepSpeed-MII 1-2 (NVIDIA)Aphrodite 4+ (NV AMD CPU Win)

self-reported README claims (SGLang: 400k GPUs; TGI: CAUTION maintenance mode)

vLLM 2000+ contributors, industry standardSGLang 400k+ GPUs, trillions tok/dayTensorRT-LLM NVIDIA enterprise stackllama.cpp ubiquitous local standardLMDeploy InternLM productionTGI HF prod, now maintenance modeMLC-LLM browser+mobile nicheOllama dominant local runnerExLlamaV2 enthusiast standardCTranslate2 prod MT/Whisper stacksDeepSpeed-MII dormant since ~2024Aphrodite community serving (Kobold lineage)

vLLM invented PagedAttention (SOSP 2023) [kwon2023]; Nishi nx_kvcache BUILT + gate GREEN (paged==contiguous bit-exact, fork+COW) but not yet wired into serving; ExLlamaV2 paged via flash-attn; llama.cpp unified cache

SGLang zero-overhead scheduler + vLLM set the bar; LMDeploy persistent batch; MII Dynamic SplitFuse; CT2 dynamic offline batching; Nishi serves one request at a time (top gap)

SGLang RadixAttention (5x claim, banked blog list); TRT-LLM KV Cache Reuse; LMDeploy automatic prefix caching; llama.cpp prompt cache

Nishi WIRED + measured 2026-07-09 (one m=n forward, ~17x fewer prefill passes, TTFT collapse); LMDeploy dynamic split-and-fuse; MII SplitFuse

ExLlamaV2 Q4 cache pioneer (README); LMDeploy online int8/int4 KV (README); llama.cpp q8/q4 KV flags

Marlin-class W4A16 is the SOTA bar [marlin2024]; Nishi CPU-Marlin: __f32_i8dot32 SSE intrinsic blessed, Q8_0 path 10.2x measured, Q5/Q4 unpack proven compute-bound on CPU (honest partial)

SGLang DFlash/Spec-V2 (2026-06 blog = current frontier); TGI claims ~2x; Nishi BUILT prompt-lookup + bit-identical gate, 58 pct accept, wall 0.94x = no net win yet (honest)

TRT engine compilation [trtllm] and MLC whole-model TVM compilation are the two compilation identities; vLLM CUDA graphs + torch.compile; Nishi is AOT-native by construction but has no GPU graphs

vLLM README lists FlashAttention FlashInfer TRTLLM-GEN FlashMLA Triton; LMDeploy FlashMLA; Nishi has own GQA offset-causal attention [vaswani2017], scalar+SSE, no flash-class tiling

TGI NCCL TP (README); CTranslate2 tensor parallelism (README); llama.cpp row/layer split

vLLM/SGLang/TRT READMEs; llama.cpp RPC layer split

SGLang large-scale EP (96xH100 + GB200 blogs) leads; LMDeploy DeepEP/eplb (README); Nishi has no MoE (remaining census opportunity)

SGLang GB200 P/D blogs (3.8x prefill 4.8x decode); LMDeploy via DLSlime+Mooncake (README); TRT via Dynamo

llama.cpp RPC; LMDeploy proxy server multi-machine (README); MII replicas + load balancing (README)

Nishi MEASURED 7.34x q4k matmul on 8T + 16T decode (sovereign pool, no pthread); llama.cpp AVX/AMX/NUMA is the CPU king; CT2 MKL/oneDNN/Ruy

llama.cpp owns the format [llamacpp]; Nishi nx_gguf loader + tensor-type census (Q4_K/Q5_0/Q6_K/Q8_0/F32) measured; Ollama/Aphrodite GGUF native

Nishi sovereign BPE verified == Qwen2 digit-split behavior (nx_tok_probe); llama.cpp own tokenizer; the rest ship HF tokenizers

llama.cpp deepest (min-p mirostat DRY XTC ecosystem); TGI logits warpers (README); Nishi greedy + temperature + top-k measured, no top-p/penalties yet

SGLang compressed-FSM 3x JSON decoding (banked blog list); llama.cpp GBNF grammars; vLLM xgrammar/guidance (README); Ollama JSON schema

vLLM tool calling + reasoning parsers (README); LMDeploy TOOLS CALLING (README); Ollama tool calling (README)

LMDeploy VLM table is the largest (InternVL home, README); llama.cpp multimodal; Ollama vision models

llama.cpp embedding + rerank (README); SGLang e5/gte/reward (README); Nishi sovereign SGNS embeddings power live in-corpus search but expose no embed-API endpoint

Nishi sovereign autograd + stable Q16 integer training recipe (trained real models: reader, neural image codec); SGLang = RL rollout backend for external trainers; the field is inference-only

universal in the field (each README); Nishi serve organ is bespoke HTTP + MCP-first plane instead - an honest drop-in-adoption gap

vLLM README explicitly ships Anthropic Messages API alongside OpenAI + gRPC

TGI Rust gRPC core (README); MII gRPC (README); TRT via Triton

Nishi live sovereign MCP plane (compare/status/search tools served to agents today); absent from every banked engine README

TGI SSE (README); Ollama stream (README); Nishi serve returns full completions (gap)

TGI Prometheus + OpenTelemetry distributed tracing (README) = the production bar; Nishi has logs only

Ollama one-command install is the consumer bar (README) [ollama]; Nishi = sovereign ELF drop, zero deps, but self-hosted only

Nishi: own language + compiler to x86 ELF, own HTTP/TLS, no libc/BLAS/PyTorch/CUDA; llama.cpp minimal C/C++ but on OS toolchain; the rest sit on PyTorch/CUDA/HF. EVIDENCE (MEASURED 2026-08-13, and this cell carried none until 2026-08-14): a return-42 static ELF out of this toolchain is 188 bytes with --no-crash-guard and 5,105 bytes with the default crash guard. A binary of that size cannot contain a libc -- which states the claim as a number somebody can re-measure instead of as an adjective

Nishi gates prove bit-identical decode (spec==plain, paged==contiguous, cross-run); batched GPU float engines are nondeterministic by construction; llama.cpp reproducible only single-config

Nishi integer-only Q16 training (grad-clip + LR-decay recipe, measured) - no engine in the field claims this

Nishi: every line from compiler to syscall is in-house readable; llama.cpp compact C++ auditable but sits on drivers/CUDA beneath. EVIDENCE (this cell carried none until 2026-08-14): the chain is nx_cc -> nxasm -> ELF with no LLVM, GCC or libc beneath, and it compiles on the NAS over the mgmt API (build-over-API GREEN 2026-07-06). The compiler SELF-HOSTS, and nx_cc_equiv_gate -- a differential build-and-stdout-equality gate plus a self-host stage -- runs 10/10 GREEN before any toolchain promotion, so auditable-in-house is backed by a gate that rebuilds the reader itself rather than by an assertion about readability

DOWNGRADED Best -> Yes 2026-08-14, and the reason was sitting in this row's own note: it already said llama.cpp and Ollama are fully local. Four rivals hold Yes on this axis, so claiming Best over them was an overclaim that the evidence-counted liar-kill correctly refused to publish. Nishi never phones home by construction and that is a real property -- it is simply not a lead here. The honest fix for an unevidenced Best is sometimes to lower the grade rather than to go looking for a citation that would justify keeping it

from banked READMEs: TGI = CAUTION maintenance mode recommending vLLM/SGLang; MII dormant; vLLM/SGLang/llama.cpp are the active frontier