nishi product wiki / research / perceptual efficiency · ocean: /wiki/research

Render only what the brain will see

Compiled 2026-08-01 · product research brief · feeds perceptual-renderer (media-companion family) · forks off research_rtrender, research_voxelrender

The thesis in one line. The biggest speedups come from deleting work, not from accelerating it. A GPU renders every pixel of every frame, twice, ninety times a second, whether or not anything changed and whether or not the viewer can perceive it. The visual system does the opposite: it predicts, and spends effort mainly where prediction fails. Between a headset's optics and a person's awareness there is a large, measurable, and largely unexploited amount of computed light that is discarded before it is ever seen. The product opportunity is to stop computing it.

Evidence labels used in this brief

Label	Meaning
SHIPPING	In a product a customer can buy today.
RESEARCH	Peer-reviewed or preprint result; not a shipped product.
VENDOR	Manufacturer or marketing claim, not independently benchmarked.
OURS	An in-ecosystem measurement or design decision of ours.

1. Spatial asymmetry — foveation

Visual acuity falls off sharply away from the point of gaze, so peripheral pixels are computed at a fidelity the eye cannot resolve. This is the best-developed of the three asymmetries and it is already shipping.

Claim	Label	Source
VR-Splatting renders the periphery as cheap smooth 3D Gaussians and the fovea as expensive neural points with a CNN, hitting 90 Hz at 2016×2240 px per eye on a single RTX 4090. Frame budget is 11.1 ms; measured total 10.9 ms (peripheral splatting 4.9, foveal neural points 3.5, CNN 1.6, blending 0.5, tonemap 0.4). In a 12-subject study ~76% preferred it over the VR-GS baseline (p < 0.005).	RESEARCH	Franke, Fink & Stamminger, “VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points”, I3D 2025 / PACMCGIT 8(1), May 2025. arXiv:2410.17932 · doi:10.1145/3728302
The gaze-tracking overhead can eat the rendering savings. A3FR states the problem directly — “the computational overhead of the gaze tracking process itself can sometimes outweigh the rendering savings” — and fixes it by running the gaze DNN in parallel with rendering rather than before it. Reported: “up to 2×” end-to-end latency reduction. See the correction below on what that 2× is measured against.	RESEARCH	Xin, Wang & Zhang, “A3FR: Agile 3D Gaussian Splatting with Incremental Gaze Tracked Foveated Rendering in VR”, ICS '25, June 2025. arXiv:2507.04147 · doi:10.1145/3721145.3735112
Eye-tracked foveated rendering ships today. Meta reports GPU savings of 33–45% for eye-tracked foveated rendering vs 26–36% for fixed foveation at default resolution, rising to 36–52% at 1.5× resolution. Measured end-to-end eye-tracking pipeline latency in their UE4 test app: 46–57 ms.	SHIPPING VENDOR	Meta Horizon developer blog, “Save GPU with Eye Tracked Foveated Rendering”, 2023-02-15. link
FovealNet attacks the same tracking-overhead problem from the model side: event-based cropping discards >64.8% of irrelevant input pixels for ≥1.42× gaze-tracking speedup and ~13% system improvement with no accuracy loss.	RESEARCH	Liu, Duinkharjav, Sun & Zhang, IEEE TVCG, May 2025, doi:10.1109/TVCG.2025.3549577. arXiv:2412.10456

Correction to the conversation this brief came from. That conversation stated A3FR “had to parallelize tracking against rendering to get 2× end-to-end.” The “up to 2×” is a best case, not a typical result. On desktop-class hardware (Tesla T4, 1080p/1440p) the paper reports roughly 20% average speedup versus a sequential foveated baseline and ~40% versus full-resolution non-foveated rendering. The ~2× figure comes from a simulated edge GPU (Jetson Orin NX class) at 720p, measured against full-resolution non-foveated rendering (about 151 ms → 77 ms). Both numbers are real; citing the 2× without the measurement context overstates the gain against the baseline that actually matters here. The direction of the finding survives; the magnitude does not.

How wide is the high-acuity region, really?

The conversation used “about 1°.” The literature does not converge on a single number, because different fields draw the boundary in different places. A foveated-rendering survey puts the highest-acuity foveal region at ~2.5° of the visual field while noting that visual-cognition work often uses 0–0.5°/1° and other definitions run out to 10°; the survey states plainly that these boundaries are “not defined uniformly across various disciplines.” RESEARCH (Jabbireddy, Sun, Meng & Varshney, “Foveated Rendering: Motivation, Taxonomy, and Research Directions”, arXiv:2205.04529, 2022.) Use ~2.5° with the definitional caveat, not a bare 1°. The engineering conclusion is unchanged either way: the high-acuity region is a low-single-digit-degree patch of a >100° field.

2. Temporal asymmetry — event-driven

Conventional cameras and conventional renderers both work in whole frames at a fixed rate. Event sensors abandoned that on the input side a long time ago. The output side largely has not.

Claim	Label	Source
The first practical event camera (DVS128) is characterised in its own title: 128×128, 120 dB dynamic range, 15 µs latency, asynchronous temporal contrast — a measured, fabricated, peer-reviewed silicon result, not a marketing figure. Power ~23 mW.	RESEARCH	Lichtsteiner, Posch & Delbruck, IEEE J. Solid-State Circuits 43(2):566–576, Feb 2008.
Survey comparison of event vs frame cameras: latency <0.1 ms vs ~10 ms, dynamic range >120 dB vs ~60 dB, pixel bandwidth “on the order of kHz”, low power.	RESEARCH	Gallego et al., “Event-based Vision: A Survey”, IEEE TPAMI 44(1):154–180 (arXiv 2019, journal 2022). arXiv:1904.08405
Modern commercial parts: Sony/Prophesee IMX636 — DR >86 dB (5 lux–100 klux) or >120 dB (80 mlux–100 klux), pixel latency <1000 µs at 5 lux, 5 mW standby / 205 mW max. GenX320 — DR >140 dB, latency <150 µs at 1k lux, down to 36 µW in low-power mode. Note the illumination dependence: microsecond latency is a bright-light figure, not a flat guarantee.	VENDOR	Prophesee product datasheets: IMX636, GenX320
Event-driven Gaussian splatting for high-speed egomotion is real and active. Event3DGS reports +3 dB reconstruction quality and a 95% reduction in computational cost versus prior event-based dense 3D reconstruction, on synthetic and real high-speed egomotion data.	RESEARCH	Xiong et al., “Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion”, CoRL 2024. arXiv:2406.02972. Siblings: E-4DGS (ACM MM 2025, arXiv:2508.09912), EventSplat (arXiv:2412.07293, preprint).

Correction: the frequently-quoted “~1000× less data” is a manufacturer figure, and it is a range: Prophesee claims “10× to 1000× less data.” It is not a measured consensus number from the academic survey. Cite it as a vendor claim with the range intact, or measure our own on our own scenes.

Where the gap is. The input side of event-driven vision is a mature, commercially available technology with peer-reviewed silicon behind it. We could not find an equivalent output-side result: a renderer whose cost scales with change rather than with pixel count. That asymmetry — mature sensing, immature rendering — is the opening this product aims at. See the UNVERIFIED section: absence of evidence here is not proof of absence.

3. Perceptual asymmetry — the gaps

The third asymmetry is the least exploited and the cheapest to exploit, because it requires no new sensor: there are windows in which the visual system's sensitivity is strongly reduced, and pixels rendered into those windows are computed and then discarded by the viewer.

Claim	Label	Source
Saccadic suppression is real, centrally controlled, and outlasts the movement: “saccadic suppression is controlled centrally, and typically lasts for >100 ms even for saccadic movements of ~50 ms”. Suppression begins tens of ms before saccade onset and persists after it ends.	RESEARCH	Crevecoeur & Kording, “Saccadic suppression as a perceptual consequence of efficient sensorimotor estimation”, eLife, 2017-05-02. link
Typical saccade duration 20–40 ms for ~2° saccades, longer for larger amplitudes (the “main sequence” relation).	RESEARCH	Bahill, Clark & Stark, “The main sequence, a tool for studying human eye movements”, Mathematical Biosciences, 1975.
Saccade rate in natural viewing: ~2–3 per second, i.e. roughly 120–180 per minute.	RESEARCH	Standard oculomotor literature; see the survey context in Crevecoeur & Kording 2017 above.

Two corrections, one in each direction.
(1) “Functionally blind during saccades” is too strong. The primary source is explicit that sensory feedback “must be strongly reduced, but not completely suppressed, as observed experimentally.” It is a large reduction in contrast sensitivity and response gain, not a blackout. A renderer that emits nothing at all during a saccade is therefore a claim requiring measurement, not a free lunch — which is exactly why the experiment below has to be run rather than assumed.
(2) “Dozens of saccades per minute” understates the rate by roughly 3–6×. The real figure, ~120–180/min, makes the opportunity larger than the conversation claimed. Note the asymmetry in how these two errors cut: one overstated the effect per event, the other understated how often the event occurs.

4. The contested claim, handled honestly

The claim: “roughly 90% of the input to V1 is feedback from higher areas, not from the eye.” This is widely repeated. It conflates two distinct real findings and, as stated, it is wrong.

Finding 1 (about the thalamus, not V1): only ~5–10% of synaptic input onto LGN relay cells comes from the retina; the majority is corticothalamic feedback and brainstem modulation. Sherman & Guillery, PNAS 95:7121–7126, 1998-06-09; Van Horn, Erisir & Sherman, J. Comp. Neurol. 416:509–520, 2000.

Finding 2 (about V1 layer 4): “the major drive to the cat's visual cortex comes from a projection that forms a mere 5% of the excitatory synapses in layer 4.” Binzegger, Douglas & Martin, “A Quantitative Map of the Circuit of Cat Primary Visual Cortex”, J. Neurosci. 24(39):8441–8453, 2004. link

Why the popular version is wrong: the remaining ~95% is not mostly top-down feedback from V2/V4/MT. The same paper attributes ~34% of all excitatory synapses in area 17 to local self-innervation within the cortical layers, and the whole feedforward thalamo-recipient loop to only ~21%. Descending feedback from extrastriate cortex is a real but separate and smaller category (see Markov, Kennedy et al., J. Comp. Neurol., 2014).

The defensible statement: feedforward retinal/thalamic input is a small minority (~5–10%) of the synapses in V1's main thalamorecipient layer — but most of the remainder is local intracortical processing, not top-down feedback from higher visual areas.

Does the product thesis survive? Yes, and it should be restated more carefully. The engineering-relevant fact was never “the brain is 90% top-down.” It is that the sensory stream entering cortex is a small fraction of the computation the visual system performs, and the visual system reconstructs a stable percept from it rather than transporting a frame. Foveation, saccadic suppression and change blindness are all directly measured perceptual facts and none of them depend on the 90% figure. We keep the corrected anatomy and drop the slogan.

Predictive coding: a framework, not settled science

The canonical citations are Rao & Ballard, Nature Neuroscience 2:79–87, 1999 (a computational model that reproduced extra-classical receptive-field effects such as endstopping) and Friston, Nat. Rev. Neurosci. 11(2):127–138, 2010. RESEARCH But current reviews are explicit that “varied definitions and inconsistent empirical evidence have raised questions about its validity and explanatory scope” (Furutachi & Hofer, Annu. Rev. Neurosci. 49:471–494, 2026; see also the 2023 empirical-status review in Neurosci. Biobehav. Rev. 155:105459). Predictive coding is best described as an ongoing research programme, not a confirmed mechanism. This brief uses it as an analogy that motivates an experiment, and the experiment does not depend on it being true.

The 20-watt figure

The brain consumes ~20% of the body's resting caloric budget at ~2% of body mass (Raichle & Gusnard, “Appraising the brain's energy budget”, PNAS 99(16):10237–10239, 2002). RESEARCH “20 watts” is a derived conversion, not a figure printed in the primary source: it assumes a ~100 W whole-body basal rate. Accurate as an approximation; label it as one.

5. The product

Saccade-gated event renderer. Eye tracker detects saccade onset → rendering is suspended or degraded for the suppression window → a corrected frame is presented at fixation. Between saccades, render only the regions where a predicted frame diverges from the true frame beyond a perceptual threshold. Cost scales with change and visibility, not with pixel count.

Element	Commitment
Outcome (catalog row)	`media-companion → perceptual-renderer` — a renderer whose cost scales with perceivable change. Governance inherited: no-overclaim (a perceptual-equivalence claim must be measured, never asserted).
Customer-visible metric	Joules per perceptually-indistinguishable second — energy per second of rendering that a viewer cannot reliably distinguish from a conventional reference, on the same scene and hardware. Falsifiable by an outsider with a power meter and a forced-choice study.
Why this metric	Frames per second rewards computing pixels nobody sees. Energy per indistinguishable second only rewards work that survives to awareness. It is also the brain's own figure of merit.
Falsification condition	If, at matched perceptual indistinguishability, the saccade-gated event renderer does not beat a conventional renderer on joules per second on the same scene and hardware, the thesis is wrong and the rung is abandoned. Two specific ways it can fail: gaze-tracking energy and latency exceed the savings (the A3FR problem, measured and real), and suppression is partial rather than total so aggressive gating becomes visible.
Rig	An eye-tracked headset, a Gaussian-splatting pipeline, a power meter. OURS Both rendering halves already exist in our corpus — see the forks-off table.
Second harvest	Event cameras in, event rendering out: the same architecture serves the robot lane in both directions. A machine that computes only when the world changes idles near zero.

6. Declared UNVERIFIED

Listed rather than omitted. A brief with no declared gaps is a brief whose author did not look for them.

#	Gap
1	“Nobody publishes joules per perceptually-indistinguishable second.” This is a negative claim and we have not established it. We found no such metric in the foveated-rendering literature we surveyed, but we did not run a systematic search of power-measurement studies in VR. Treat as an unproven negative — it is the single most load-bearing unverified statement in this brief, because the product's novelty claim rests on it.
2	No output-side event renderer found. Same caveat as above: we did not find a renderer whose cost scales with change, but absence of a find is not proof of absence.
3	The “under $2k” rig cost from the originating conversation is unpriced. We have not costed an eye-tracked headset plus a power meter at 2026 prices. Do not repeat the figure until quoted.
4	Suppression thresholds are not engineering parameters yet. The literature gives durations and the fact of reduced sensitivity; it does not give us the contrast/spatial-frequency thresholds needed to decide what may safely be skipped mid-saccade. This must be measured on our own pipeline.
5	VR-Splatting's “1.45× vs the fastest related approach” is unconfirmed as to which baseline it names. The 90 Hz / 2016×2240 / RTX 4090 figures are solid; this comparative one was not verified against the paper's table.
6	Event-camera power comparison table not read directly. Per-sensor mW figures (DVS128 23 mW etc.) come from a secondary paraphrase of the Gallego survey; only the DVS128 title figures are primary. Independently plausible, not independently read.
7	PSVR2, Varjo and NVIDIA VRSS figures were deliberately excluded from the claims tables as vendor marketing or secondhand conference reporting. They are not evidence at this brief's bar.
8	Change blindness and inattentional blindness are asserted, not sourced here. The originating conversation invoked them; we source only saccadic suppression. They need their own claims before they can carry design weight.
9	The originating conversation is not fully captured. Its share link could not be machine-read, so this brief is built from the portion available plus fresh source-checking. Earlier turns may contain claims not represented here.

Forks off

Parent	What was inherited
research_rtrender — Real-time rendering and GPU graphics architecture	The 3D Gaussian splatting axis and the neural-rendering-in-the-pipeline framing that both VR-Splatting and Event3DGS build on; the discipline of labelling each rendering claim by evidence class.
research_voxelrender — Real-time rendering on a constrained stack	The constrained-stack posture: what to do when you cannot spend your way out with hardware — which is the same posture that makes deleting work more attractive than accelerating it.
research_standard — The brief standard	The claims-table skeleton and the authored-once rule this page follows.

What this brief fed: the perceptual-renderer product row in the media-companion family, and the shared event-architecture argument for the robot lane. What it is the first of: the product-research river at /product/research.