MRSSE

For BC6H encoding in Oodle Texture, we needed a sensible error metric to use in the encoder core. BC6H is HDR which is more challenging than LDR data since we need to handle vast differences in magnitude.

BC6H internally essentially treats the float16 bits as 16-bit integers (which is a semi-logarithmic mapping) and works with those integers throughout. It’s a bit more complicated than just grabbing the float bits, but not in a way that really matters here. The most straightforward approach is to do the same thing in the encoder, pretend we care about those 16-integers, and just work with squared errors etc. on those, which is easy to do and also quite fast.

This works reasonably well with white(-ish) light and not very oversaturated colors, but degrades horribly especially with bright, colored emissive light. My colleague Jon Olick wrote a post showing some examples back when we originally released Oodle Texture. The problem with working on log-space RGB data boils down to this: say we have some bright red glowing thing and the RGB tuples for a pixel (in whatever scaling we choose) are something like (2.34, 0.02, 0.01). Working with the logarithms per-channel in this scenario tells us that getting the value in the green channel off by 0.001 would be about as bad as getting the value in the red channel off by 0.1, so we end up producing large net brightness shifts to avoid imperceptible hue shifts, a bad choice.

We still would prefer something squared error-ish in the encoder core: the derivative of a squared error is linear, so minimizing a squared error by finding its critical points turns into solving a linear system. We find ourselves doing this hundreds of times for millions of pixels each, so a simple linear system is good.

I’ll cut to the chase, the main distance metric we use in Oodle Texture for HDR data is what we internally call MRSSE, “Mean Relative Sum of Squared Errors” (somewhat unfortunately different from the similar but distinct notion by the same name used in statistics). In essence we wanted something like our usual squared error (aka SSE, Sum of Squared Errors, or SSD, Sum of Squared Differences), but normalized to be more like a relative than absolute error, so we used

The numerator is self-explanatory, the denominator was chosen to avoid divisions by zero if either vector is nonzero (in our case, there’s also an extra tiny bias term on the order of the smallest normalized float16 to avoid divisions by zero) and to make the difference symmetric. The latter is not required, but convenient. In our case, x and y are 3-component vectors. The normalization here is somewhat arbitrary, it would be perfectly sensible to throw an extra factor of 2 in the numerator to normalize by the average squared length of x and y, not the sum of their squared lengths, but we only ever care about comparing errors with each other, an overall scaling of 2 changes nothing, so we went with the simpler form.

Since this is a BC6H encoder, most commonly one of the two inputs, say x, is held constant (coming from the source image) while we try many candidate choices for y. We do this a lot, and in this case we expect anyway (after all making the two close is our objective), so we use the asymmetric relative squared error

instead (again, with a tiny bias term in the numerator to avoid division by zero). This has the huge advantage that since x is fixed, we can compute all the values for a block once up front, and then just have a regular weighted squared error (with a per-pixel weight factor) per pixel. That’s what’s used in our inner loops. It makes “minimize this error”-type optimization problems actually linear, which is extremely handy.

We did not attempt to do a full formal evaluation of different error metrics. We did, however, try something like 8 or 9 different metrics in the early stages of writing the BC6H encoder (this would’ve been mostly 2019/2020) and had a surviving 3 metrics (one using ICtCp-ish variant using the SMPTE 2084 PQ transfer function, one using the “natural” albeit problematic semi-log2 encoding implied by BC6H, and this one) carried through all the way into shipping Oodle Texture, until we ultimately decided to ship with just the one always on because we never found an image that really benefited from anything else. Of the metrics we evaluated, this ended up being computationally one of the cheaper ones and simultaneously the best in terms of visual result quality in informal but careful testing; it was not close. The semi-log2 encoding is the cheapest by far, but its problems with highly saturated bright colors ultimately disqualified it.

This one was good, still fairly simple, and gave us a result quality that (in my opinion anyway) is appreciably better than all other BC6H encoders I’ve seen.