🚀 Feature Overview
Add support for TurboQuant KV cache quantization to enable memory-efficient long-context LLM inference with near-lossless quality.
TurboQuant is a novel online vector quantization method from the Google ICLR 2026 paper "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate" (https://arxiv.org/abs/2504.19874).
📊 Problem Statement
KV cache is typically the biggest memory consumer during LLM inference, especially for long contexts:
- A 70B model running 32k context can use 20-40GB of KV cache alone
- Current quantization options (FP8, INT4) use scalar quantization with MSE optimization
- These methods don't optimize for inner product preservation, which is critical for attention mechanisms
✨ Solution: TurboQuant
TurboQuant combines three key innovations:
1. Random Orthogonal Rotation
Pre-processes KV vectors with a random orthogonal matrix to spread energy evenly across dimensions, making the Gaussian assumption valid for optimal quantization.
2. Lloyd-Max Scalar Quantization
Uses learned codebooks (via Lloyd-Max algorithm) for near-optimal quantization of rotated vectors, beating uniform quantization on MSE.
3. Outlier-Aware Channel Allocation
- Auto-calibrates on first batch to detect high-variance channels
- Top 15% high-variance channels (especially RoPE-affected) kept at bf16
- Remaining channels quantized at 4-bit with Lloyd-Max codebooks
4. QJL Residual Correction (Optional)
1-bit Johnson-Lindenstrauss correction for unbiased inner product estimation.
📈 Expected Benefits
Based on benchmarks from the paper and vLLM implementation:
| Metric |
Improvement |
| Memory Savings |
2.69x - 4.4x vs bf16 KV cache |
| Inner Product Preservation |
>0.98 cosine similarity |
| Quality |
Needle-in-haystack PASS at all lengths |
| PPL Degradation |
<2.5% (7.53 → 7.69 on Qwen2.5-7B) |
| Throughput |
+3-5% prefill speedup (reduced memory bandwidth) |
🔗 Related Work
🛠️ Implementation Status
SGLang PR #21617 is a work-in-progress implementation with:
✅ Core quantization logic (turboquant.py)
✅ Lloyd-Max codebook generation
✅ Outlier channel detection and calibration
✅ Triton kernels for fused rotate+quantize
✅ SGLang-native integration (model_runner, FlashInfer backend, Triton backend)
✅ 42 unit tests passing
🚧 Remaining work:
- End-to-end benchmarking on actual models (GB10, Strix Halo)
- Performance tuning for Triton kernels
- Documentation and examples
📖 Technical Details
Architecture
KV Cache (bf16) → Rotate (R) → Outlier Split → Lloyd-Max Quantize → Encode
↓
Dequantize ← Merge Outliers ← Inverse Rotate (R^T) ← Decode ← KV Cache (uint8 + scales)
Configuration
# Command line
--kv-cache-dtype turboquant
# Config options (defaults shown)
polar_bits: 4 # Quantization bits for non-outlier channels
outlier_fraction: 0.15 # Fraction of high-variance channels kept at bf16
use_qjl: true # Enable 1-bit QJL residual correction
Files Modified
python/sglang/srt/layers/quantization/turboquant.py (core logic)python/sglang/srt/layers/quantization/turboquant_kernels.py (Triton kernels)python/sglang/srt/model_executor/model_runner.py (dtype config)python/sglang/srt/layers/attention/flashinfer_backend.py (hooks)python/sglang/srt/layers/attention/triton_backend.py (hooks)tests/test_turboquant_kv_cache.py (42 tests)
🎯 Use Cases
- Long-Context Inference: Run 100k+ token contexts on consumer GPUs
- Multi-User Serving: Serve more concurrent requests with same VRAM
- Cost Reduction: Reduce GPU memory requirements for production deployments
- Research: Enable experiments with longer contexts than previously feasible
📝 References
Labels: enhancement, quantization, kv-cache, long-context
🚀 Feature Overview
Add support for TurboQuant KV cache quantization to enable memory-efficient long-context LLM inference with near-lossless quality.
TurboQuant is a novel online vector quantization method from the Google ICLR 2026 paper "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate" (https://arxiv.org/abs/2504.19874).
📊 Problem Statement
KV cache is typically the biggest memory consumer during LLM inference, especially for long contexts:
✨ Solution: TurboQuant
TurboQuant combines three key innovations:
1. Random Orthogonal Rotation
Pre-processes KV vectors with a random orthogonal matrix to spread energy evenly across dimensions, making the Gaussian assumption valid for optimal quantization.
2. Lloyd-Max Scalar Quantization
Uses learned codebooks (via Lloyd-Max algorithm) for near-optimal quantization of rotated vectors, beating uniform quantization on MSE.
3. Outlier-Aware Channel Allocation
4. QJL Residual Correction (Optional)
1-bit Johnson-Lindenstrauss correction for unbiased inner product estimation.
📈 Expected Benefits
Based on benchmarks from the paper and vLLM implementation:
🔗 Related Work
🛠️ Implementation Status
SGLang PR #21617 is a work-in-progress implementation with:
✅ Core quantization logic (
turboquant.py)✅ Lloyd-Max codebook generation
✅ Outlier channel detection and calibration
✅ Triton kernels for fused rotate+quantize
✅ SGLang-native integration (model_runner, FlashInfer backend, Triton backend)
✅ 42 unit tests passing
🚧 Remaining work:
📖 Technical Details
Architecture
Configuration
Files Modified
python/sglang/srt/layers/quantization/turboquant.py(core logic)python/sglang/srt/layers/quantization/turboquant_kernels.py(Triton kernels)python/sglang/srt/model_executor/model_runner.py(dtype config)python/sglang/srt/layers/attention/flashinfer_backend.py(hooks)python/sglang/srt/layers/attention/triton_backend.py(hooks)tests/test_turboquant_kv_cache.py(42 tests)🎯 Use Cases
📝 References
Labels:
enhancement,quantization,kv-cache,long-context