How I know the
pictures are right

01 — Why looking is not a test

A wrong estimator converges too

The most dangerous failure in a renderer is not noise. It is a clean, stable, plausible image of the wrong number.

When the correlated batch mode was built, there were two candidate normalisations: divide by the number of primitives drawn, or by the number that actually crossed the camera ray. The second is wrong. It was built anyway, behind a compile flag, fired, measured, and then deleted, because a trap you have watched go off is one you will not ship.

It reads four and a half times the right answer. It is also stable: over a sixteen-fold range in pass count the wrong film moves by 1.5 %, exactly the way a converging estimator does.

The inflation factor is N / #crossed, which is 1/P(cross): measured at 2.1–6.8× on frame energy and 6.6–14.2× per lit pixel across five preset/strategy pairs. It is not a bug that shows up as an artefact. It shows up as a caustic that is simply brighter than the truth, in an image nobody has a reference for.

The consequence for the rest of this page. A stable image can be off by a factor of four and a half, so no acceptance number on this site is an image. Every one is a division: one estimator over another that shares no code with it, in a format built to be divided. That is the whole apparatus, and it is four things — a linear film, an independent second opinion, a statistic chosen to suit the estimator's tail, and a byte comparison to catch what statistics cannot.

02 — The primary instrument

Two independent estimators of one integral, reported as a ratio

Nothing in this project is accepted on the strength of its own output. Every estimator has to divide against something it shares no code with, and land on 1. The table below is what landing looks like.

The pairs are chosen so that the two sides agree on the transport and disagree on everything else: one samples a photon and sweeps it into an analytic sheet, the other fires a stochastic beam and gathers it with a kernel; one connects to a light in scene.lights[], the other traces a ray until it hits an emissive entity. When two constructions that different land on the same number, the number is the integral — there is no shared code left for a common error to hide in.

Degenerate limits are free exact tests

Where a new estimator becomes an old one, the comparison costs nothing and pins the measure convention rather than the value. Driving the refractive index to 1 collapses the hourglass onto the photon cone, which was validated a month earlier against next-event estimation; the density has to match term for term, not merely in expectation. A limit that agrees is worth more than a Monte Carlo run that agrees, because it removes the sampler from the question entirely.

What a ratio cannot test

A shared factor cancels and is therefore invisible. Strategies 20 and 22 deliberately share an emission convention (pp.vhEmit = 0, radiance L_e/cos_y) so that 22/20 is a like-for-like test of the Jacobian — which means it is not a test of the emitter model. That model is one of three in this codebase and the disagreement between them is unresolved; see section 08.

Preset 54, equal passes, same camera. Read the last row first: the two estimators agree on the total to within 4 %, and that agreement is what makes the first three rows a measurement of sharpness rather than of brightness. A caustic 5.3× higher and 9× narrower at the same energy is the kernel's blur removed — the wide kernel's. A carries r = 0.70 against B's 0.15, and with the beam at the pair's own radius its waist is 16.1 px, a factor of 2.0 at matched radius; the rest of the 9 was the competitor's knob. B is the pair and carries a kernel of its own, and at r = 0.05 its waist is 7.5 px, so the width is the sheet's while the kernel shows in the energy (the ladder on Caustics without a kernel). Without the energy row any of it would be indistinguishable from an exposure error.

Three of those four rows are a correction, and the fourth is why they could be corrected. The energy ratio had a producer and reproduced to five digits when the films were regenerated. The peaks and the widths did not: they shipped as 0.0299 / 0.1313 / 57 px / 7 px, read off the films by hand in a session that did not record which row it read, and the films measure neither peak nor 57 px at any row. docs/tools/waist.py is the producer they should have had. It states its rule before it reports a number — lower-half row where B's row peak is largest, band = the contiguous rows within 10 % of that peak, luminance on the raw film, half-maximum crossings interpolated on both sides — and reports the band median with the spread across its seventeen rows, because one row of a 6,000-pass film is noisy at the percent level: peak 0.0229 (0.0206–0.0250) against 0.1231 (0.1158–0.1276), ratio 5.3× (4.7–6.2), width 70 px (68.5–73.1) against 7.7 px (6.1–10.4), ratio 9.2× (6.8–11.7). The direction and the size of the claim survive; two of its digits did not.

03 — The measurement surface

A film format you can divide

A tonemapped PNG is not evidence. Every number on this site comes off a raw, linear, unclamped dump.

The format is deliberately trivial: the four bytes PGR1, three little-endian int32 for width, height and pass count, then w·h·3 little-endian float32 in linear HDR. No tonemap, no exposure, no gamma, no clamp. It is written by three hooks — the PP film, the CPU integrators and the GPU path tracer's accumulation texture — so that all three render paths land in one comparable container.

The property that matters most is the least glamorous. The stored values are a running 1/N mean, not a sum, so a 400-pass dump and a 4000-pass dump divide directly with no normalisation step and therefore no place to put a normalisation bug.

Every render that produced a number here was launched from a JSON config through a dot-path loader — 54 live pp.* keys, including the strategy, the preset, the bounce range, the guard, the MIS mask and the dump pass. There is no interactive step in any measurement, so every one of them re-runs from a file.

The tool

docs/tools/filmstats.py is 99 lines and lives in the repository on purpose, beside the numbers it produced: the instrument behind the film ratios on this site belongs under version control, and the scratchpad it started in has been cleared more than once.

• stats — mean, p99, max, non-zero fraction. The first thing run on any dump, because "mean 0, 0 non-zero px" is a different problem from "mean 0.03, max 18744".
• ratio — per-pixel median ratio over pixels above the reference's 40th percentile. The default acceptance statistic.
• diff — total-energy ratio plus where the energy moved, listed as the five largest per-pixel gains. Used to confirm that an added transport lands, and where.
• chroma — per-channel totals and saturation percentiles. A dispersive caustic must show channel-dominant pixels; a grey one must not.
• png — tonemap to look at it. This is the only way to look at a headless film, because the application has no screenshot function at all: F12 logs "Snapshot requested" and writes nothing (Engine.cpp:373).

One diagnostic worth stealing. In a diff, summed negative delta == 0.0000 means a change only ever added energy. That is the signature of a gate that stopped rejecting things it should reject, and it is visible in one line of output long before it is visible in an image.

04 — The independent model

A Python oracle that does not share a line with the renderer

Fifteen rules about where each estimator's variance comes from, each one cited to a run — and the study that produced them never launched the renderer once.

docs/variance/primitives.py is a line-by-line numpy port of the C++ integrator, validated against a finite-difference evaluation of the co-area Jacobian to a median of 1e-10. Four studies were run on it: the plane family, the direction sweeps, the medium dependence, and the dielectric hourglass. Sixteen thousand estimator/ray pairs were compared against a composite Gauss–Legendre quadrature self-converged to 5.2e-7, and a hundred and eighty-four direction-sweep rows against a tan-substituted Simpson × GL quadrature converged to 1e-13.

The caveat belongs in the same paragraph as the result. The renderer was never run for any of it. This is a strong statement about the mathematics of the estimators and about the numpy port, not about the shipped C++. Where the two have been compared, it is through the ratios in section 02 and nowhere else.

The disagreements are published, not filtered

Section 6 of that study exists to list every place a Monte Carlo mean failed to match the analytic integral, so that nobody rediscovers them. Three examples of what it records against itself:

• Over 16 758 plane pairs the mean ratio is 0.9995–1.0010 per estimator, but |z| > 3 in 0.91 % of pairs against the Gaussian expectation of 0.27 %, with pooled z standard deviation 1.076. The batch-mean standard error of a 1/J-tailed estimator is optimistic by about 7 %, and its sample mean sits below the truth more often than above.
• The photon cone reads a real −0.70 % ± 0.15 % deficit on rays that cross the light axis. Traced to the [−0.9999, 0.9999] clamp discarding a polar sliver those rays traverse, and confirmed by removing it: 0.99384 ± 0.00045 with the clamp, 1.00080 ± 0.00089 without.
• Strategy 23's running mean on one ray is +31 % at 4×10⁵ samples and settles to +3 % at 6.4×10⁶. A fold-tailed estimator's sample mean is skewed at any sample count a render will reach, which is a fact about the statistic and not about the estimator.

Predictions registered before the run

The study closes with twelve in-engine experiments, each with a quantitative prediction that would falsify the numpy model if it came out wrong — the plane band following sin(γ)−2.9, the 3-vs-12 crossover between light edge 0.5 D and 1 D, the cone's clamp deficit appearing on axis-crossing pixels only and vanishing one pixel to the side. Writing the number down before the run is what makes the run capable of being wrong.

The correlated batch mode closed that loop end to end: a design document predicted what each primitive would look like at sixteen samples, and the report of what actually appeared was written against it, prediction by prediction. Two did not survive, and both misses are worth more than the hits.

The light-size series closed two more, the other way round. Both crossovers the variance study had written down — the planes overtaking the direction trio once the emitter subtends 35° at the ball, the vertex hourglass overtaking the light-seeded one at 35–41° — were rendered on one rig at five emitter sizes out to 77°, every cell timed and its energy checked against the camera walk, and neither happened inside the range. The trends ran the predicted way and the numbers did not: the trio's variance floor is a third of what the study assumed, and the light-seeded hourglass keeps its coverage where the study had it collapsing. Both corrections stand in the study beside the predictions they replace, and the rig found a missing shadow ray in the trio on the way, which is what a rig is for.

The second miss is sharper. Strategy 3 translates the light quad without rotating it, so a ceiling quad stays horizontal at every distance and a level camera sees every member near edge-on. The tangency is not "a few blinding slivers among ordinary patches" as predicted — it is a single incandescent line across the frame at the camera's own eye height, and the entire estimator's variance lives on that one scanline. That is a better fact than the prediction was, and it was only available because a picture of the sampling geometry existed to be looked at.

The other oracle: an acceptance harness inside the process, at five named pixels

Everything above is off-line, and its caveat is that the renderer was never run. The complement runs inside the renderer, behind one checkbox with a matching config key — pp.gateLog, so a headless run can turn it on and a measurement stays re-runnable from a file. It is four logs, and each one checks the shipped C++ against a reference that shares no code with the estimator being checked. What follows is a source read of the harness, not a new measurement; the runs it produced are the ones already quoted on this page, and the one it caught is named at the end.

A row this page has to contradict, on the page where contradictions belong. The inventory grades the furnace / energy-ratio log measured, which is the grade carried in the table above because that is the inventory's grade and this page does not silently regrade anything. The source disagrees, and where they disagree the source wins: furnaceLog is declared, exposed and assigned, and read nowhere. The defect class is the one this site keeps finding — a control that draws its own promise. Two things it must not be confused with. There is a real furnace test in this codebase, and it is a different feature under a different flag: with pp.surfaceBSDF on, the same binary that renders runs a reciprocity check over 10⁴ random direction pairs and a white-furnace hemisphere integral at eight incidence angles per material arm, and prints a table. And the energy ratios quoted throughout this site came off filmstats.py and the gate logs above, not off this checkbox. Nothing on the site is retracted by this; one control is.

What the harness has actually caught. One case is recorded in the source at the line it fixed, and it is the same shape as blind spot four below. The area-light NEE was too bright by 1/cos θL, because the emitter's own cosine had been folded into the MIS density and not into the contribution. Gate A could not see it: its point light has no surface orientation, so that cosine is 1, and the gate's pixel is the centre one, where the connection is nearly head-on and the error is nearly nothing. Gate C saw it, as a near-surface over-brightness on preset 34 — a whole-image metric on a scene with an emissive quad and grazing connections, which is exactly the pair of conditions the analytic gate does not have. That is the argument for keeping four gates instead of the best one: each is blind somewhere, and the blindnesses are in different places.

05 — Acceptance conventions

The statistic has to match the estimator's tail

The default acceptance criterion is a per-pixel median ratio converging to 1.000. It is the wrong criterion for about half the family, and using it anyway is how you get a confident wrong number — so the family has three acceptance statistics, one per tail shape, each matched to the estimator it judges.

Median rather than mean, because a single firefly cannot hide a systematic error behind itself. But a median over pixels assumes the two films light the same pixels, and for the sparse members they do not.

Why medians mislead for sparse estimators

Strategy 22 lights roughly 45 % as many pixels as strategy 20 at equal passes. Its per-pixel median against 20 is therefore coverage-limited: it reads low, and it reads low for a reason that has nothing to do with bias. The acceptance statistic for the sparse vertex members is consequently a total-energy ratio over a shared support, which is why the table in section 02 reports 1.007 and 1.020 for 22/20 rather than a median.

Why totals mislead for fold members

Every co-area member whose Jacobian vanishes like a square root — strategies 7, 9, 20, 22, 23, 24 — has weight tail P(W > w) ~ w−2, measured Hill index 1.96 and 2.00. Its second moment is logarithmically divergent, so the sample total is dominated by the largest draw and a raw energy ratio is a lottery. Strategy 9's energy ratio over five growing batches, and the same films with the top 0.1 % of pixels clipped:

The 8.44 is one pixel. Strategy 8 is the one member of the direction-sweep family with bounded weights, which is exactly what makes it the clean unbiasedness gate for the whole correlated mode: if the normalisation is right, 8 says so without any robustification at all.

Why guard-on and guard-off totals must never be compared

The grazing guard discards contributions whose Jacobian falls below a floor. It is tempting to price it by differencing the two film totals. For a fold member that number is meaningless: the guard-off total is just the largest sample that happened to be drawn, and neither film measures the truth at the affected pixels. What can be compared is the footprint — which pixels differ at all, and whether the rest are bit-identical — and that is how the guard was actually measured (section 07).

Which statistics converge and which do not

Six seeds of one identical hourglass configuration, same sample count:

Two consequences are worth stating flatly. Error bars come from batch means and never from per-sample variance: two seeds of the same 10⁶-photon beam run gave ratios of 0.920 and 1.004 with a quoted standard error of 0.013–0.02, so the two runs sit four to six standard errors apart while each one reports itself as converged. And the per-pixel variance of a fold member is not an error estimate, so any adaptive sampler that treats it as one will happily concentrate effort on the pixels where its own estimate is least trustworthy.

06 — The safety net

Byte-identical is the strongest test available

A statistical check asks whether two films agree within noise. A byte comparison asks whether the second run drew the same random numbers as the first, and that question catches a class of defect nothing else does.

An extra random draw shifting the stream, or a gate that quietly became more permissive, will both hide inside Monte Carlo noise indefinitely and neither will survive a hash. Several of the real bugs in this project were found only that way.

Two recent changes, both of which should have been invisible to existing renders:

• Correlated batch mode. Twelve films pinned from the pre-change binary and re-rendered from the final one at 300 passes each — preset 41 strategies 1 and 9, preset 47 strategies 3, 7, 8, 9, 13, 18, 20 and 21, preset 48 strategies 20 and 23. All twelve byte-identical, checked twice: once with the deliberately-wrong debug trap compiled in, and once after removing it. The mode works by re-seeding the RNG immediately after the camera ray is built; with the mode off it re-seeds nothing, so no call moved.
• A new scene preset. Eight films pinned against a clean HEAD binary rebuilt from a stash, re-rendered after the change — presets 48/20, 47/22, 52/21, 35/1, 49/21, 41/9, 48/18, 51/20. All byte-identical. The new occluder list defaults to empty, so the four existing showcase presets add no entity, no material and no random draw.

The instrument produces false alarms, and that has to be written down too. One apparent difference on preset 52 / strategy 21 in the first pinning batch was a bad capture: the clean HEAD binary reproduced the post-change hash three times out of three. A byte comparison is only as reliable as the capture that produced the baseline, so a mismatch is a reason to re-capture before it is a reason to investigate.

The same tool is also the strongest positive result on the site. When the grazing guard's footprint was measured, the finding was not only that it touches 309 pixels out of 55 800 — it was that every pixel it does not touch is bit-identical between guard on and guard off, on all eight runs. That is what turns "it does not dim the caustic core" from an impression into a measurement.

The other kind of identity: two algebras, one image

Byte identity asks whether a change moved anything. The estimator panel carries the complementary instrument, and it is a dual-form cross-check: one toggle, LightSurface EXCLUDED against INCLUDED, switching between two algebraically equivalent ways of writing the same emitter hit. The excluded form is the short one — the emitter area, the emission cosine, the π of the Lambertian lobe and the intersection determinant all cancel on paper, so the code omits them and returns power times transmittance. The included form refuses the cancellation and computes every one of those factors explicitly, letting them cancel numerically instead. Both must produce the same image, and the moment one of the four is folded in at the wrong place or counted twice, they stop doing so.

It is worth naming beside the byte harness because it catches the opposite failure. A hash catches a change of behaviour that noise would hide; this catches a bookkeeping error that was always there and that no amount of convergence will reveal, because a consistently mis-folded constant converges beautifully to the wrong number. The two together are why a factor in this estimator has to be wrong in two independent ways before it reaches a film.

The seed lock underneath a byte-identical film

Both batches above are photon-primitive films, driven from headless pp.* configs where the integrator's own RNG is deterministic by construction. The megakernel has one equivalent instrument, a toggle in Technique ▸ Tracer ▸ Seeds & Animation, and no result on this page was taken through it — but it is the thing a reader would reach for to reproduce a GPU frame, so what it does is worth stating exactly. This is a source read, not a measurement.

One line of it is a case study for the audit rule this project runs on. The tracer toggle reaches the GPU through a reference alias declared beside it, bool& fixedNoiseSeed = fixedTracerSeed;, so a search for the control's own name finds the checkbox and no consumer at all. Ask only "does this identifier have a reader" and the answer is no and the grade is wrong. Ask "is the feature reachable, by any route" first — which is the rule adopted after a light tracer was called dead while shipping as two menu entries — and the alias is the route, one line below the declaration.

07 — The corrections

Seven claims this project made, and the seven measurements that overturned them

Seven in two days, each overturned by a measurement, each written into the document that carried the wrong claim, beside it. This is the section the rest of the site earns its credibility from. A record that holds only confirmations is not a record, and the reason a ratio elsewhere on these pages can be believed is that this table exists and nothing was taken out of it.

The habit that produced all seven is now a rule. Reasoning about why an estimator is dark does not work; instrumenting it does. One glass-occlusion investigation ran for several rounds on careful source reading, reached a confident wrong conclusion, and recorded it in a commit message as settled; a single counter in the actual code overturned it in one run. That happened three times in one session. The written rule is now: instrument first, and treat a mechanism you can only argue for as a hypothesis with a pending measurement.

This is a section and not an appendix because it is the strongest evidence on the site that the other numbers are real. Two of the seven were corrections to claims that had already driven a design decision elsewhere in the project, and each was caught by one measurement that cost less than the decision it overturned.

08 — The limits of the instruments

What this apparatus cannot see

A page about how you know things is exactly where the limits of your instruments belong. Seven of them, in rough order of how quietly they fail — and number five is one this page closed, which is the only reason it is written up as an instrument failure rather than as a result.

1. The regression harness never loads two presets in sequence

The headless runner writes a JSON config, launches the executable, waits for the film dump, kills the process, and moves the file aside — one fresh process per render. It therefore cannot see any state a preset writes and no preset resets, and byte-identical films prove nothing about that entire class of defect. The focal-length leak in section 07 is the one instance that has been found. The class has not been swept. The rule added to the behaviour document is: when adding a preset, check what it writes outside its own scene, not just whether the films still match.

2. Every validated ratio for the dielectric members was measured with the guard on

So some part of the hourglass's 2.4 % solo deficit is the guard's discard rather than the missing internal-reflection chains it is attributed to. The two have not been separated, and section 05 explains why the obvious separation — differencing the two totals — is not available for a fold-tailed estimator.

3. Two suspected double transmittances, with a decisive test that has not been run

One in the GPU shader, where a pass-through multiplies by an absorption term after the free flight was already sampled analog against the full extinction; one in the CPU emitter term, where a transmittance is applied on top of analog survival. Both are invisible on today's presets because the fog is nearly pure-scattering, and both have an unambiguous test written down — a purely absorbing slab must render at Tr, not Tr². Neither has been run. They are recorded as suspected, not as confirmed.

4. The two render paths disagree about a directly-seen emitter, and it is unresolved which is intended

On preset 0's ceiling quad viewed obliquely, the CPU integrator returns L_e exactly and the GPU applies a view cosine, a ratio of 2.96. Both NEE paths agree, so only the direct-emitter class differs. The consequence is a standing rule that every CPU-vs-GPU ratio must mask directly-seen emitter pixels until the question is settled. The reason it went unnoticed for months is the sharpest item on this page: the acceptance test that should have caught it views the quad head-on, where the cosine is 1. A gate can pass because of where it looks.

5. A reference left at its defaults is not the reference you configured

The composite mode shipped with the radiometry of its specular continuation — the code that draws a glass ball by refracting the camera ray through it — argued rather than measured. The check that should have settled it read the CPU integrator at 0.10× the GPU on plain diffuse wall pixels, which nobody believed, so it was recorded as inconclusive and shipped anyway. The cause was not a convention: the GPU reference had been given a bounce limit through a field that the current bounce model ignores, so it silently ran naive path tracing with full global illumination and eight fog scatters against a composite that has neither. The number was real; the denominator was a different integral.

Closed by moving the test off the fogged scene it could not live on and onto the fog-free Cornell box the existing surface reference already uses, where one shader detail — a glass transmission does not spend a surface bounce, so a one-bounce cap still admits the whole refracted chain — makes the two sides computable to the same integral. The diffuse half lands at 1.0008 and the glass ball at 0.9949, which rules out the specific error the design was afraid of: a squared refractive index applied inside the ball would have moved that number by up to 2.25×. What is still argued is narrower and is stated as such: no fog-free scene in the project contains a mirror, so the mirror arm of the same code is checked by reading both sources rather than by dividing two films.

The measurement also named a real convention difference and priced it: the CPU integrator does not tint a refraction by the glass’s own colour and the GPU does, so on a ball whose glass is 0.95 grey the two disagree by exactly 1.108 in red and green and by nothing in blue — predicted from the two source lines, then measured at 1.1029 and 0.9949. It is converted in the comparison harness rather than fixed in either renderer, and it is inert on every scene the composite mode actually ships.

6. The per-pixel variance tool does not exist

The twelve registered predictions in section 04 are all stated in per-pixel relative variance, top-1 % energy share or max-over-median, computed from K independent film dumps at equal passes. filmstats.py has no sub-command that does this. The study names it as the one missing tool and it has not been written, so those twelve predictions are currently unfalsifiable in the engine — which is to say they are not yet evidence of anything.

7. Some of this is built, checked, and never looked at

Everything below is built and passes its headless or numeric check; three of the strategies listed are validated estimators with an energy ratio to their name. None has been displayed, and the grade says so: the check verifies the number, and the picture is the next thing to take.

• never viewed  Clean view, split A/B compare, and the hourglass controls — mirror relays, the guard and Jacobian-floor slider, include-photon-point
• never viewed  Strategies 22, 23 and 24. All three validated by energy ratio; none ever displayed
• never viewed  Showcase presets 49–52, including the spectral prism fan
• never viewed  Presets 38, 39 and 43 — emissive sphere, cube and ball oracles with no menu entry, reachable only from a config key

This is tracked as its own category in the behaviour document rather than folded into "done", because the failure mode of a headless-only check is specific and known: it verifies the number and says nothing about the picture.