nishi code wiki / research / real-time rendering

Real-time rendering and GPU graphics architecture

SOTA census · compiled 2026-07-31 · 6 axes · ~40 sourced claims · 8 declared gaps

corpus slug: reference-realtime-rendering-gpu-sota-2026-07-31

Claims are labelled SHIPPING / RESEARCH / ANNOUNCED / DEPRECATED. Gaps are declared UNVERIFIED rather than guessed.

1. Neural rendering

LinAlg: three scopes — thread (matrix-vector, the neural-shading case), wave (matrix-matrix on MMA/WMMA/XMX), threadgroup (driver-tiled). Types FP32, FP16, F8_E4M3FN, F8_E5M2, U8, S8, U32. Tier 1.0 mandates U8xU8->S32 and S8xS8->S32. FP8 must be accepted but implementations “MAY emulate it” — a mandatory format is not a fast format. AMD, Intel, NVIDIA and Qualcomm are committed (GDC 2026, 2026-03-12).

Neural Texture Compression — the axis with hard numbers

SHIPPING SDK, v0.9.2 BETA (github.com/NVIDIA-RTX/RTXNTC). All PBR channels compressed jointly (9-10 channels, ~64 bits/texel baseline) to ~5 bits/texel = ~12.8:1, “comparable to BCn”. PSNR 40-50 dB; the SDK's own bar is 35-40 dB sufficient, 50 dB perceptually lossless. A 2K PBR material is 2.5 MB vs 12 MB. Cooperative Vector gave 2-4x inference throughput — so the scalar path works, just slower, and that is the MVP door. Inference-on-load transcodes to BCn and runs on any SM6 GPU.

Adoption caveat: no verified retail game shipping NTC or neural materials. An SDK is not an adoption.

DLSS 5

ANNOUNCED 2026-03-16, ships fall 2026: a compact diffusion transformer applied after conventional rendering, three selectable models, per-object artist controls, 4K (~8M px) inside a 16 ms budget, causal one-frame-in/one-frame-out. A new category beyond reconstruction: generation.

2. Geometry — virtualized micropolygons

Nanite essentials: ~128-triangle clusters (METIS) → a cluster DAG where groups are simplified as a unit with the group boundary LOCKED (the whole trick) → a monotonic error metric so a view-dependent cut is crack-free and GPU-selectable → two-pass HZB occlusion culling → software raster in compute with 64-bit atomics for sub-pixel triangles → a visibility buffer plus deferred material resolve.

DXR 2.0 standardises clustered geometry (spec v0.30, 2026-07-10): CLAS (max 256 tris / 256 verts), cluster templates, Cluster BLAS, Partitioned TLAS, ClusterID(). Notably it “can work well on any hardware with existing raytracing support given a driver update” — RTX Mega Geometry going cross-vendor.

Highest-leverage finding: meshoptimizer ships the hard half — meshopt_partitionClusters (Nanite-style hierarchical LOD grouping), meshopt_simplify with meshopt_SimplifyLockBorder (the locked-boundary simplification that makes the DAG crack-free), and meshopt_SimplifySparse.

3. Lighting — real-time GI

Lumen benchmark: ~8 ms at 1080p internal on next-gen consoles for GI + reflections combined at Epic quality, relying on TSR for near-4K. Surface Cache parameterises each mesh from 12 cards by default. Epic deprecated SWRT detail tracing in 5.6 and is converging on a single HWRT path at 60 Hz.

ReSTIR (SIGGRAPH 2020), measured: 6x-60x faster unbiased, 35x-65x biased; Zero Day 15 ms and Amusement Park 50 ms at 1920x1080; up to 3.4 million dynamic emissive triangles; ≤ 8 rays per pixel. Algorithm: RIS → weighted reservoir sampling → temporal reuse → spatial reuse. The correctness of the reuse steps lives entirely in the MIS weights — naive reuse is biased because neighbours have different integration domains.

RTXGI 2.0 = NRC + SHaRC; DDGI has been moved out to a legacy v1.x repo. The industry moved from probe-based DDGI to hashed/neural radiance caches. AMD has a parallel stack: FidelityFX SDK 2.3.0 “Redstone” ships FSR Ray Regeneration 1.2.0 (ML denoiser) and FSR Radiance Caching (Preview). RTX Path Tracing SDK 1.8.1 claims 25-30%+ gain over 1.7.x.

The 2026 frontier is radiance caching: ORCA (EA SEED, SIGGRAPH 2026) and Spatio-Temporal Control Variates with ReSTIR (Shi et al., SIGGRAPH 2026 Honorable Mention). Open reference: Bevy Solari implements ReSTIR DI + GI + a world cache in the open (experimental, Vulkan-only).

4. Radiance fields — compression is solved

3DGS compression leaderboard (Eurographics 2025 STAR, w-m.github.io/3dgs-compression-survey/), Mip-NeRF 360:

The frontier method is simultaneously higher quality and ~15x smaller than Scaffold-GS. HAC++ reaches 30.34 dB at 5.5 MB on Deep Blending. Shipping raw .ply splats in 2026 is two years behind.

3DGRUT (NVIDIA, v1.0.0 2025-04) on RTX 5090 / NeRF Synthetic: 3DGRT 33.87 PSNR / 347 FPS / 479.3 s train; 3DGUT 33.88 PSNR / 846 FPS / 214.6 s train — ray-traced quality at 2.4x the framerate. Both handle fisheye and rolling shutter, which vanilla 3DGS cannot because EWA projection assumes a pinhole model.

5. Upscaling and frame generation

DLSS 4 (2025-01-06): transformer SR with 2x the parameters of the CNN model, self-attention across the frame and over multiple frames, on all RTX GPUs. Multi Frame Generation: up to 3 generated frames per rendered frame; the FG model is 40% faster and uses 30% less VRAM.

FSR 4 budget targets at 4K: RX 9070 XT ~1.3 ms; RX 7900 XTX ~3.1 ms; RX 7800 XT at 1440p ~2.1 ms. A temporal upscaler must land in 1-3 ms at 4K to be worth shipping. FSR 4.1.1 (2026-06) added RDNA 3 support. Intel's repo is now the XeSS 3 SDK.

6. The honest bar

Neural: a tiny MLP (2-4 hidden layers, 32-64 wide) with plain FP16 dot products, a learned latent texture, Rusinkiewicz parameterisation, trained offline. The input encoding is load-bearing, not the network. Adopt Slang for compiler-generated derivatives; the free SIGGRAPH 2026 course An Introduction to Neural Shading (github.com/shader-slang/neural-shading-s26) is the single best resource.

Geometry: visibility buffer first and alone, then meshlets + cone culling, then the cluster DAG, then HZB, then compute raster last (an optimisation, not a requirement).

GI: BVH → ReSTIR DI at 1 spp with unbiased MIS weights → A-trous denoiser → world-space hash-grid radiance cache before ReSTIR GI. The upscaler is not optional — the stack is designed around ~1 spp and assumes temporal reconstruction downstream.

Splatting: forward-only is 1-3 weeks. Load a pretrained .ply, project the 3D covariance via the Jacobian (S' = J W S W^T J^T — the one piece of real maths), tile-bin, radix sort by (tileID, depth), alpha composite front-to-back. The sort is both the perf and the correctness bottleneck.

Upscaling: Halton(2,3) jitter, motion vectors for everything (missing or wrong motion vectors are the #1 cause of a bad TAAU, ahead of the resolve filter), Catmull-Rom history, a 3x3 neighbourhood clamp in YCoCg, disocclusion reset. Do not build frame generation — it costs ~one frame of latency by construction and is the least differentiating item; async reprojection is the tractable cousin.

Ranked by impact x tractability

Declared UNVERIFIED — do not treat as measured