browser inverse rendering workbench
From image fitting to scene understanding
A convincing render can hide the wrong geometry, material, or light. These three investigations make the difference visible: fit a scene, change the evidence, and inspect what remains uncertain.
Executable studies that isolate one question from the main investigations.
Build intuition and connect the methods. These views do not run an inverse solve.
Controlled synthetic experiments. Targets are known for evaluation; each panel states what its solver receives and what its display illustrates.
01 / Representation
A silhouette can match while the unseen shape is wrong. Fit an implicit 3D scene to calibrated images, then challenge it with a different view, a topology change, or missing fine detail. The residual shows where the current fit disagrees with the evidence.
The default view ray-marches the fitted signed-distance field directly, preserving smooth topology changes during optimization.
Every number here is one draw. Fisher information is the usual alternative but is a local Gaussian approximation, and this lab has an exhibit it provably cannot see - the anisotropy branch, where two parameter sets give byte-identical images while the Fisher matrix reports full rank.
Every stochastic decision in this solver draws from one stream. A reported number is one draw from one seed - re-roll and re-run before treating a threshold as a property of the method. The manifest records the seed and a URL that restores this run.
Four calibrated views constrain side and rear geometry.
WebGPU is the preferred renderer and uses the same SDF normals, material model, ray-march budget, and cast shadows as the worker compatibility path.
Switch capture setup to inspect the images and camera poses used by the optimizer.
The planner evaluates held-out cameras and proposes the view with the largest residual-weighted information gain.
Fitting cube-like SDF parameters toward a smooth sphere target.
Fixed validation grid; adaptive search radius; best checkpoint retained.
shared inverse problem
Every lane receives the same filtered transport signal, target, initial shape and albedo, Adam update, and sample budget. Only the gradient backend changes.
A hard shadow edge dominates the geometry signal. Attached replay misses the decisive boundary term.
BRDF, light, throughput, and replayable path vertices while sampled visibility stays fixed.
Silhouettes, shadow boundaries, and occlusion events where path support changes.
Differentiate photon positions and smooth density kernels for difficult caustic paths.
03 / Measurement
A wall carries light from a scene the camera cannot see directly. Can that light recover a volume, an image, or only an angular profile? Change acquisition mode and follow the chain from the room geometry to sensor data to a measured reconstruction.
Occlusion adds no measurements; it makes the ones you have linearly independent.
The occluder losing to the edge here is a property of the 1D unknown, not of the occluder. Lifted to a 2D hidden scene it wins 70 usable directions to 16 — measured in the Coded Aperture lab.
This is the measured tensor, not a light-transport rendering. Time zero is rectified at the wall: the forward model measures only the wall-object-wall leg, so the sensor-to-wall legs never appear. The expanding front is the hidden object's response, not the outgoing laser pulse.
02 / Appearance
A highlight might come from shape, roughness, tiny facets, or the light itself. Fit a controlled procedural object from RGB, mask, and depth evidence, then freeze groups of factors to find out which explanation the measurements actually support.
physical participating media
Compare calibrated tomography, sparse-view regularization, and a single-view posterior on the same 3D density field. The reconstruction receives rendered measurements only; target voxels are retained solely for the forward simulator and evaluation.
measurement design workbench
Run four compact inverse problems with the measurements they actually require. Every lane exposes its progressive shot sequence, recovered quantities, residual, and local Fisher matrix instead of treating all photographs as equivalent RGB input.
| Auxiliary datum | Current estimate | Role |
|---|
Linear light stacks constrain normals and diffuse albedo; gloss and shadowing violate the baseline.
AoP and DoLP survive intensity changes, but normal branches and material class remain coupled.
Arrival time adds depth to radiance; a single sensor pixel still cannot recover lateral shape.
Exposure diversity separates radiance from a compact camera curve until motion or clipping removes evidence.
supporting experiment / geometry + appearance
Turn the fitted SDF into a measured triangle mesh, then explore a separate sparse-BSDF fitting experiment below. The direct-mesh, Gaussian-surface, and UV-atlas views illustrate possible pipelines; they are not additional solvers.
The first panel slices the current fitted SDF. Marching tetrahedra turns that same field into the middle mesh. Compare it with the target on the right. Both meshes use neutral shading so material differences cannot disguise geometry.
How the alternative pipelines work. These are interactive diagrams. The executable SDF extraction and measured metrics remain in the first tab.
A smooth analytic lobe plus a learned residual fitted from sparse angular samples.
Left to right: target, fitted GGX, and GGX with the learned residual. Each strip evaluates the angular functions plotted below. This is a coplanar material preview; the fit does not reconstruct a full 4D BSDF.
| lobes | RMSE | held-out | AIC | BIC | Gram cond |
|---|
On several of these presets the residual is flat across lobe counts, and the Gram condition number explodes above one lobe. That is the honest answer rather than a failed solve: a grid search over widely separated widths confirms two distinct lobes do no better than one. A mirror-centred GGX cannot reach a secondary lobe that sits at an offset from the mirror direction, so no arrangement of widths recovers it, and the large remaining residual is what the learned kernel is then for. The old hand-placed base could not show this because it was constructed to look right. Lobes are GGX projected onto the in-plane slice, not Gaussians in the mirror offset. That matters: the synthetic target is a sum of Gaussians in that offset, so fitting Gaussians would be a model-matches-generator cheat and every information criterion computed from it would be meaningless. The held-out band is contiguous rather than a random subset, because random holdout leaks through neighbouring samples under a smooth kernel. The fitted lobe count is a model-selection verdict under an assumed Gaussian noise model, not a claim that N physical lobes exist, and parameters are identifiable only up to lobe permutation. Where the Gram condition number exceeds 1e4 the reported weights are not meaningful individually even though the total residual is lower.
Fuse calibrated depth or correspondence into a point cloud, then reconstruct and texture a surface.
Fit an SDF through neural volume rendering, then extract the zero level set with marching cubes.
Optimize a deformable tetrahedral or dual-marching grid so topology and mesh quality can change together.
3DGS arrived first for view synthesis; later surface-aligned and 2D splats made mesh extraction more geometric.
Charts or coupled SDF-splat representations preserve global surface coherence and high-frequency appearance.
Differentiably extract and refine connectivity during Gaussian training instead of treating the mesh as post-processing.
illustrated guide / hidden-object capture
Time of flight encodes range; structured illumination encodes a reciprocal image; a passive edge encodes direction. Compare their measurements on the same hidden target. The last two panels illustrate the expected output of each acquisition regime.
Panels 1 and 2 compute a forward measurement. Panels 3 and 4 draw the known target with mode-dependent blur and offset; they do not invert panel 2. The illustration-quality readout responds to budget and noise sliders, not to measured reconstruction accuracy.
For executable reconstruction, open 3D NLOS Scene: confocal LCT, Hadamard dual photography, and a calibrated edge-penumbra inverse receive measurement arrays, with ground truth kept in the forward simulator and evaluation.
Pulsed or modulated illumination and time-resolved sensing isolate path length. Hardware is specialized, but range is much better constrained.
A projector emits many structured patterns while a camera records their responses. Helmholtz reciprocity turns the measured transport matrix into a virtual view from the projector.
Ambient illumination and an ordinary RGB video use an edge as an angular encoder. Moving targets become a 1D space-time reconstruction; two edges or stronger priors are needed for depth.
representation growth
Anisotropic Gaussians composited front to back, fitted by a hand-derived analytic adjoint. The population is not fixed: splats whose centre the loss keeps pulling are cloned if small and split if large, and anything that goes transparent is pruned. That adaptive density control is the part that is actually the contribution of Gaussian splatting — rendering ellipses was never the hard bit.
no densify pass yet
Press "Relight the recovery" after fitting. A splat carries a colour and no notion of a light, so whatever shading was present when it was fitted is baked into per-splat colour and the recovery simply does not change.
Scope. This is the two-dimensional case: no 3D covariance, no view-dependent spherical harmonics, no camera. Depth is an ordering key for compositing, not a coordinate, and the adjoint carries no term for it. Gaussians are cut at three sigma, which is a real approximation. The adjoint itself is checked against central differences to a worst relative error of 4e-9 across every parameter type, so the gradient is not the uncertain part.
On whether densification is worth it, measured rather than assumed: against a fixed population of the same final size, adaptive density control reaches a higher loss at equal iteration count. What it buys is cheap early iterations, so it only wins on wall clock — about 2.8x better at an eight-second budget, and a tie by twenty seconds. At this scale, with a simple target, knowing the right splat count in advance is worth more than placing them adaptively. That is not the regime the technique was designed for, and the honest reading is that this lane demonstrates the mechanism rather than vindicating it.
Computational cameras
A pattern can encode incoming direction or turn defocus into a distance cue. Explore the mask, follow the measurement, and compare the recovered image or depth with its evidence.
The actual mask · white transmits, black blocks
A distant point illuminates the sensor through a shifted copy of this mask. Every scene direction contributes its own weighted copy; the sensor records their sum. Reconstruction finds the angular image whose sum explains that exposure.
This view uses the selected mask with periodic boundaries. It recovers directions, not distance. Open Depth from coded blur for a lens-based camera that estimates range.
Try the vertical slit: it loses one image axis. A fully open mask records total light. Both can explain the sensor while leaving the scene undetermined.
Fine and coarse coding trade detail against noise. The best mask depends on the scene and measurement budget.
Calibrated periodic-mask camera. The unknown is angular RGB radiance, not metric depth.
Depth from defocus · a patterned lens aperture
The aperture turns defocus into a depth signature. Recover the distances of sixteen textured planar targets from their photographs: fit the blur footprint, then use the calibrated lens to convert its scale to distance.
Away from focus, each point spreads into a scaled copy of the aperture. A single photograph mixes unknown texture with unknown blur. An image prior helps compare distances; a second focus setting adds an independent measurement.
Each coloured cell is one estimated distance. Select a cell to inspect its blur signature and the competing depth hypotheses.
Calibrated focal length, focus distance and aperture diameter turn distance into a predicted blur footprint. The same code appears at different scales.
One photo uses a generic texture prior. With two focus settings, a candidate distance must explain both measurements using the same sharp texture.
Flat texture and noise can leave several depths plausible. Inspect the score curve and switch capture evidence instead of treating every coloured estimate as certain.
Controlled calibration chart: 4 × 4 independent planar image regions, with periodic blur within each region. The solver estimates one distance per region, not a dense curved surface or occlusion boundaries.
The single-image idea is motivated by Levin, Fergus, Durand & Freeman · Image and Depth from a Conventional Camera with a Coded Aperture (2007). This smaller experiment uses its own fixed mask and inference model.
Small examples · exact operator structure
These deliberately small 16 × 16 scenes expose rank and null spaces in a finite-plane camera. They use a different boundary model from the tiled-mask image recovery above.
| Aperture | Usable directions | Rank | Effective rank | ZNCC |
|---|---|---|---|---|
| solving | ||||
Why the single edge collapses, provably rather than empirically. Its visibility test is a comparison against where the ray crosses the aperture plane in x, and that crossing depends on azimuth alone. So two columns of the operator that share an azimuth are exactly proportional, every elevation difference lands exactly in the null space, and the rank is capped at the azimuth count — 16 of 256 here. The test asserts this to 3e-17, and separately measures a second cap nobody asks about: at a coarser sensor the rank falls to 9 rather than 12, because azimuth bins whose cuts land between two adjacent sensor columns produce identical rows. Sampling the hidden scene more finely than the sensor can distinguish buys nothing.
Full rank is not the same as a well posed problem. Two crossed edges, the finite occluder and the random mask all reach rank 256 — they can represent any hidden scene — and they are still far apart on how many directions survive a 1% noise floor: 74, 70 and 129. A panel that reported only rank would call these three equivalent. The undesigned random mask beating both structured apertures by nearly 2× is a lower bound on what aperture design buys, since it is not even a MURA.
The regularizer cannot buy an axis, and it is worth seeing that separately. Sweeping the smoothness weight across five thousandfold never lights up the edge's blind direction; the best setting reaches ZNCC 0.66 and over-regularizing gets worse rather than plateauing. More pointedly: the edge's recovered elevation profile does change a little between two different hidden scenes — by 4.9e-5 — and turning the prior off collapses that to 1.1e-13. Every visible trace of elevation in the regularized picture was put there by the prior, not by the data.
Scope. The hidden scene is directional: it lives at infinity and is parameterized only by direction, so there is no range axis and none is claimed. Occlusion is binary, the aperture is a single plane at a known position, and there is no time of flight, no transport and no wavelength — this is a visibility experiment. The anchor for the higher-dimensional version is Baradad et al. 2018, Inferring Light Fields from Shadows, which recovers a 4D light field from the shadow a known occluder casts on a diffuse wall by inverting exactly this kind of linear system.
experimental — accidental and engineered optics
Two focus settings turn blur into a distance cue. Capture a shaded scene, fit its distance using only the two photographs, and inspect the recovered sharp image. The aperture statistics and waterdrop experiments explore other ways optics encode information.
Capturing two focus settings…
Two calibrated focus settings; one target-plane distance, not a depth map.
Capturing the same scene through three apertures…
Optical intuition
These compact mask and point-spread tiles explain what each camera does to a point of light. Their optical sampling is independent of the image recovery and the small statistical benchmark below.
Small statistical benchmark · 64 × 64
A separate, fixed benchmark measures success across noise levels. These scores describe its small scene ensemble, not the photograph recovery above.
| Axis | Noise | Round | Coded | Annulus |
|---|---|---|---|---|
| solving | ||||
Depth from defocus asks which blur scale explains the picture, over an eight-rung ladder, so chance is 0.125. Two axes. Matched noise hands every aperture the same signal level, which is the comparison the coded-aperture framing usually shows and which is flattering to a mask that throws light away. Matched photons lets the mask's opacity cost it signal against identical sensor noise, which is what a fixed exposure actually does.
Measured, and it is not the pitch. Even on the flattering axis this mask does not beat a plain round hole; on the physical axis it collapses, from 0.891 to 0.500 at the hardest noise level while the round aperture — already fully open, so it has nothing to lose — does not move at all. The mask here is a fixed pseudorandom pattern rather than the one Levin et al. optimized, so this is a statement about this mask. What it does establish is that a round aperture is a far stronger baseline at pure scale discrimination than the usual framing admits, and that the light penalty is not a footnote.
| Scene | Scorer | Round | Coded | Annulus |
|---|---|---|---|---|
| solving | ||||
And here is where coding finally earns its place. The question is only near or far at a blur magnitude the estimator is told, so chance is 0.5. Three conditions are each necessary and none is sufficient, which is why the table has four rows and only one of them works.
An asymmetric aperture. Rays from a near point cross before the sensor and from a far point after it, so the aperture is imaged rotated by 180 degrees between the two cases. A round aperture and an annulus are their own rotation, so their two point spread functions are bit-identical — the test asserts a difference of exactly zero — and they sit at exactly 0.500 in every row no matter what else changes.
A non-Gaussian scorer. This was the surprise. Rotating a real point spread function by 180 degrees conjugates its Fourier transform, which leaves the magnitude untouched — measured at 3.6e-16 for the coded mask, whose two blurs plainly differ to the eye. A zero-mean Gaussian scene model gives a likelihood that depends on the aperture only through that magnitude, so the exact likelihood used everywhere else in this lane is structurally blind to the sign for every aperture. The cue lives entirely in the phase.
Read the near-chance cells as exactly chance rather than as measurements. The two hypotheses there are not hard to separate, they are mathematically identical, so their scores are usually bit-identical in double precision and where they are not the ordering falls to summation rounding. Two independent symptoms of that: the same cell reads differently under Node and in a browser from identical code, and it moved again when the trial seeding was changed without touching any physics. Neither reading carries information. The rounding-independent claim is the 3.6e-16 magnitude equality above.
A non-Gaussian scene. Swapping in a sparse-gradient scorer is still not enough, because a Gaussian field has Gaussian gradients however its spectrum is shaped — there is no sparsity to grip. Only with piecewise-constant scenes and a hyper-Laplacian score does the sign come back, at 0.927. Remove any one of the three and it returns to chance.
Scope. Monochrome, geometric optics, no diffraction and no chromatic aberration. Scenes for the scale experiment are drawn from exactly the 1/f Gaussian model the estimator assumes, which makes the likelihood exact and isolates the aperture as the only difference; real photographs are not that.
| Sensor samples across the drop | Depth error |
|---|---|
| solving | |
The ledger, and it is the finding. The drop's shape is known exactly here and correspondence is given — two gifts no real system receives — and depth is still wrong by more than half at a realistic number of pixels. That is the angular compression coming back as a bill: every sensor sample stands for nearly a degree of world direction, and a coarse ray direction is a coarse triangulation. The drop has to cover something like 1536 pixels before depth reaches single-digit percent, which is not a raindrop in an ordinary photograph.
| Assumed radius off by | Depth off by | Amplification |
|---|---|---|
| solving | ||
And the shape dependence, which is why the paper's actual contribution is recovering drop geometry by minimising surface tension plus gravitational potential energy rather than assuming a sphere. Every recovered ray direction comes from Snell's law at a surface whose curvature was assumed rather than observed, so a shape error becomes a ray error becomes a depth error — amplified about sixfold at a 1% radius error, and worse as the error grows, so it is not a fixed gain that could be calibrated away once.
Scope, and it is large. A two-dimensional cross-section: the window is a line, drops are circular arcs, the sensor is one-dimensional. The drop is modelled as a spherical cap, the zero-gravity equilibrium shape — real drops are gravity-flattened and teardrop-like, and the sensitivity above is exactly what says how much that matters. Correspondence is given rather than solved, which in practice is most of the difficulty. The glass slab is ignored, since two parallel refractions offset a ray without turning it. No Fresnel weighting, no dispersion, no absorption — geometry only.
layered appearance
One physical stack — a rough dielectric coat of thickness d and extinction sigma over a base of albedo rho — evaluated three ways. Naive lobe addition is what an inverse solver reaches for by default. Statistical adding-doubling transports energy and directional variance in closed form. The position-free stochastic walk is unbiased under the plane-parallel assumption, so it is the reference the other two are scored against rather than a third opinion.
The stochastic reference costs roughly 60 to 100 times the statistical operator, which is why it generates evidence in a worker rather than driving the solver. Compute the reference once, fit the cheap model to it.
| model | train RMSE | held-out RMSE | gap |
|---|
| albedo | base roughness | coat roughness |
|---|
What naive addition omits: internal reflection between the two interfaces, which is a per-channel geometric series rather than a tint; the 1/eta² solid-angle compression on the way in and back out; the roughness convolution that makes apparent base roughness a property of the whole stack; and absorption that depends on the refracted path length. Measured here, an albedo ratio of 4.00 between channels arrives as a response ratio of 6.74, and sweeping the true coat roughness inflates the recovered base roughness by about 1.5x while the true base never moves.
Boundaries. The statistical operator is an approximation: it degrades above coat roughness around 0.3 and over-blurs at grazing angles, which is why the stochastic reference is shown beside it rather than trusted implicitly — its visible scatter is a property of the estimator, not a rendering artifact. Both are plane-parallel: opposing-anisotropy stacks, non-plane-parallel geometry, spatially varying stacks, spectral rendering and polarization are not modelled. The naive baseline here is reciprocal, unlike the Weidlich-Wilkie construction it resembles; its failures are the ones measured above, not a reciprocity violation. And the held-out gap is real but modest at these settings — the literature reports far larger jumps on richer stacks, and the number shown is the one this lab measures.
volume lane
Volumetric inverse rendering fits density, emission, scattering, and lighting. The hard step is converting between a fuzzy density field, a signed-distance surface, and a mesh without losing the view-dependent evidence the photos provided.
Best for smoke, clouds, translucent objects, and NeRF-style radiance fields. It can explain soft silhouettes that a hard mesh cannot.
Extracting an iso-surface gives an editable shape, but the chosen density threshold can erase wisps, translucency, and nested structure.
Marching cubes makes a renderable asset. Gradients then need either differentiable rasterization, differentiable path tracing, or a proxy loss.
A perfect setup adds calibrated views until uncertainty drops; imperfect setups test single-view ambiguity, sparse coverage, and camera bias.
research map
The scene is deliberately small: a photo-sized target is matched by three differentiable ellipse primitives with color, opacity, position, radius, and rotation. That keeps the math visible while still showing the core inverse-rendering loop: render, compare, differentiate or search, then update the scene.
The tape records scalar operations and backpropagates through a stochastic pixel batch. It is flexible and close to modern differentiable renderers, but tape memory and visibility discontinuities become the hard parts.
The same renderer has a hand-written adjoint for every primitive. It is usually faster and cleaner for this parameterization, but each new BRDF, visibility event, or geometry class needs careful derivation.
Simultaneous perturbation estimates a useful gradient with two renders, no matter how many parameters exist. It is noisy, but it becomes attractive for black-box renderers and large parameter vectors.
The evolution-strategy lane samples candidate scenes and keeps elites. It parallelizes beautifully and survives nonsmooth objectives, but spends far more render evaluations as dimensions grow.
Tens of parameters, small images, soft silhouettes, color and opacity fitting, stochastic batches, and enough math to show why gradients help.
Thousands to millions of pixels or splats, batched candidate searches, SDF or Gaussian fields, mipmapped textures, analytic adjoints, and reductions on the GPU.
Multi-view calibrated capture, shape plus SVBRDF textures, environment lighting, visibility, shadows, indirect light, priors, and multi-stage coarse-to-fine optimization.
| Family | What it recovers | Why it matters | Possible browser implementation |
|---|---|---|---|
| Differentiable mesh extraction | Triangle topology, vertices, normals, sometimes adaptive cells | Produces asset-ready meshes while still optimizing through a field | SDF or tetra grid first; marching tetrahedra/FlexiCubes later |
| Neural surface reconstruction | Multi-view SDF, radiance, normals, camera poses | Recovers detailed zero-level surfaces from RGB images without input depth | Hash-grid SDF optimization offline; extracted mesh streamed to the browser |
| Gaussian surface extraction | Surface-aligned 3D/2D splats plus mesh or implicit field | Combines fast view synthesis with geometry suitable for export and editing | WebGPU splat preview with Poisson, TSDF, or coupled SDF extraction |
| Mesh-splat joint optimization | Connectivity, vertices, Gaussians, texture and normal attributes | Keeps the rendering representation and downstream triangle asset consistent | Alternating WASM mesh updates and WebGPU splat/raster losses |
| Feed-forward sparse-view mesh | Camera, depth, point maps, mesh and material latents | Amortizes reconstruction so a few images can initialize a usable asset quickly | External model inference followed by browser-side geometric refinement |
| Texture-space differentiable rendering | High-resolution albedo, roughness, metallic, normal maps | Directly optimizes texels instead of only primitive color | WebGPU texture arrays plus mip-aware adjoints |
| Sparse measured-BSDF fitting | Analytic lobes, learned residuals, and sampling distributions | Turns a few angular measurements into a compact relightable material | WGSL evaluation with WASM fitting and physical projections |
| Gaussian inverse rendering | Geometry, view-dependent appearance, material fields | Fast capture-like assets with editable relighting targets | WGSL splat rasterizer or ray-traced Gaussian kernels |
| Diffusion/material priors | Plausible albedo, normals, depth, roughness, metallicity | Breaks ambiguities that pixel loss alone cannot resolve | External model call now; distilled tiny model later |
| Polarized / Stokes inverse rendering | AoP, DoLP, normals, dielectric or conductor reflectance factors | Adds orientation evidence that is absent from ordinary RGB intensity | Rotating-analyzer capture now; neural incident-Stokes fields offline |
| Response-aware LDR inverse rendering | HDR illumination, PBR material, geometry, and camera response | Stops unknown tone curves and clipping from being baked into material | Bracketed response fit now; joint scene factorization offline |
| Propagating-light inverse rendering | Geometry and reflectance from time-resolved multi-bounce transport | Illumination motion reveals paths and occluded structure beyond static images | Transient histograms now; time-resolved neural caches offline |
| Progressive large inverse models | Shape, material, and view-dependent radiance from a growing view set | Amortized priors turn each added image into a progressively refined asset | External inference followed by explicit browser-side audits |
| Path-space differentiable rendering | Materials and lighting consistent with shadows and GI | Forward render and gradients agree under global illumination | Hybrid WASM optimizer with WebGPU path kernels |
| Volumetric inverse rendering | Density, absorption, scattering, phase functions | Handles fog, smoke, translucent media, medical and optical media | Ray-marched fields first; differential tracking later |
| Active transient NLOS | Hidden range and reflectance from time-resolved photon arrivals | Path length turns indirect light into geometric constraints | Confocal transient cube plus Wiener Light Cone Transform |
| Structured inverse transport | Projector-camera transport matrices and reciprocal virtual views | Recovers rich indirect appearance without picosecond timing | Hadamard patterns, compressed capture, and transport transpose |
| Passive edge-resolved NLOS | Hidden angular structure, motion, and sometimes coarse depth | Works with an ordinary camera and uncontrolled illumination | Edge-transfer inversion with explicit uncertainty |
Optimizes silhouettes, colors, curves, strokes, sprites, and layers directly in image space.
Optimizes vertices, pose, UV textures, normals, BRDF maps, and lights through rasterization or path tracing.
Optimizes an implicit shape or radiance field sampled by rays, often with neural or grid parameters.
Optimizes splat positions, covariance, opacity, features, and increasingly material attributes.
Optimizes density, extinction, scattering, phase, emission, and heterogeneous media fields.
| Representation | Optimized unknowns | Gradient route | Failure mode | Validation signal |
|---|---|---|---|---|
| 2D layers | Coverage, transform, color, opacity, texture atlas | Analytic raster/SDF derivatives, soft masks, SPSA | Looks right only from the input camera | Pixel loss, mask IoU, perceptual loss |
| 3D mesh | Vertices, pose, UV maps, normals, BRDF, lights | Differentiable rasterization plus path-space estimators | Topology, occlusion edges, baked lighting in albedo | Novel view, relighting, geometry scan error |
| SDF / neural field | Implicit surface, density, radiance, material grids | Ray marching, eikonal terms, volumetric rendering | Floaters, slow training, weak material disentanglement | Novel view, extracted mesh, normal consistency |
| Gaussian field | Means, covariance, opacity, features, normals, BRDF | Splat rasterization, ray tracing, path-traced gradients | View-dependent baked color masquerading as material | Novel view, relighting, editing consistency |
| Volume | Density, absorption, scattering, phase, emission | Ray marching, differential tracking, score estimators | High variance and ambiguous density/material tradeoffs | Multi-angle capture, transmittance, relit transport |
Pixel L2 is the baseline. Research systems add mask loss, gradient/edge loss, perceptual loss, multi-scale pyramids, normal consistency, eikonal terms, BRDF priors, and relighting losses.
Single image is most ambiguous. Multi-view reduces geometry uncertainty. Multi-light or OLAT capture is the real unlock for separating albedo, roughness, specular response, and illumination.
Soft rasterization is biased but stable. Edge sampling and reparameterization handle visibility. Path replay and differential tracking make physically based gradients usable for path and volume transport.
The inverse problem needs taste: smoothness where physics wants smoothness, sparsity where assets should be simple, priors where the image evidence is underdetermined, and uncertainty where answers are non-unique.
Good evaluation asks whether the result survives new cameras and new lights. Stanford-ORB, OpenIllumination, controlled lighting datasets, and urban multi-illumination sets are more informative than training-view PSNR alone.
Keep the visible app interactive: small live demos in WebGPU, heavier benchmark presets as queued WASM/GPU jobs, and exported configs that can be run by Mitsuba, Dr.Jit, redner, or native research code.
Old demos often optimized color per vertex or per primitive. Newer pipelines optimize UV-space or surface-attached texture maps for albedo and SVBRDF parameters, usually with regularization and priors so the result relights instead of just matching pixels.
The hard part is disentanglement: many combinations produce the same image. Modern systems combine multi-view capture, lighting changes, physical constraints, learned priors, and differentiable visibility to separate geometry, BRDF, and illumination.
3D and 2D Gaussian approaches make reconstruction fast and detailed, but relighting needs more than baked color. The frontier adds spatially varying material attributes, ray tracing, and indirect illumination.
Volumes are differentiable-friendly in topology but expensive in transport. Useful targets include density, emission, absorption, scattering, and phase parameters; serious gradients need careful free-flight and path sampling estimators.
Per-pixel rendering, residuals, reductions, candidate batches, raster primitives, SDF fields, and analytic gradient kernels are natural GPU work. Dynamic tape construction is less GPU-friendly unless the graph is compiled ahead of time.
WASM is the pragmatic lane for CPU SIMD, image preprocessing, small autodiff tapes, custom solvers, and browsers without WebGPU. It is also easier to share code with native renderers.
A serious version would keep render and loss kernels on WebGPU, compile analytic or generated adjoints where possible, and use WASM for optimizer orchestration, image IO, and fallback kernels.