Why does EXR 16b half float not work for displacement (but it should)?

[HenrikBC]

This is not a V-Ray issue per se, but I would really appreciate some more eyes/minds on this:

If I make a displacement map and save is as 32b Full Float EXR and use it in max - it generally works great.
But if I take that same map and convert it to EXR 16b half-float, my displaced render will suddenly have visible steps.

"Duh! You should always use 32b Full Float for displacement!" I can hear you all say, but...

If I take the original 32b EXR and convert it to a 16b PNG - the PNG will work just fine (albeit with a gamma correction to produce the same height), without no artifacts or visible steps.

Can someone shed some light on this? Does something else happen when making a half-float EXR that I am not aware of?

Below are some rendered examples illustrating the issue mentioned above.





32b full float




16b half float




16b png - with inverse gamma 0.51

[HenrikBC]

My own theory right now is that 16 bit half float skips steps between color values, so instead of saving 0,1,2,3,4,5,6 it simply saves 0,2,4,6.
This would explain what is happening in my examples above.

I always understood that 16 bit half float was a lossy format, but I guess I thought the lossy part was focused in the longer decimal values, and not the values found within the lower 0-65k (16 bit) range.

If anyone can confirm my theory - or has further insight, it would still be greatly appreciated. Knowing is half the battle.

[hermit.crab]

How are you converting the maps? I did a simple test with a noise texture, saved it as 32-bit, and converted it through EXR-IO in PS, however, I'm getting identical results. Could you provide the textures from your scene?

[^Lele^]

Something indeed looks lost in conversion.

[HenrikBC]

Thanks, guys! It's good to hear that there may be hope yet!

My two EXR files are in the Dropbox link below. 32b full float and 16b half float - both exported as mono channel from 3ds max from the material editor through "Render Texture" (but I get identical results when exporting 16b half float EXR for displacement from After Effects - ugly steps instead of smooth curves).

https://www.dropbox.com/s/n0wds97wa2...Float.exr?dl=0
https://www.dropbox.com/s/pabkfl8x7q...Float.exr?dl=0

[^Lele^]

So, something is definitely afoot in the way the image is converted.
If i take the average value of all the pixels, i get 0.49673 for the original, and 0.49700 for the 16 bpc.
Converting to 16bpc from inside nuke, instead, returns 0.49674 (we can chalk this down to rounding.).

That the 16bpc you converted rounds to a 3 digits is a suspicious coincidence (color picking i see no sign of any stepping in values, and they all go to the fifth decimal place), however in my test i get absolutely no stepping with any of the 3 images.

[HenrikBC]

^Lele^ Are you saying that the 16bpc file that I sent was used fine to render one of your images - without any stepping?
I have tried rendering/converting from 3ds max, After Effects and Blender to 16bpc EXR - and in every case, when I test it for displacement in max using V-Ray, it just fails.

[^Lele^]

I'm afraid so, yes.
The central image is done with your 16bpc exr. Which i also looked at in detail in nuke, without finding any oddity, besides the strange pixel color average with 2 zeroes at the 4th and 5th decimal places.
I tried also extremising the displacement so to better see any stepping, but nothing.
I may be missing some info, or may be doing something wrong, i can't quite imagine what now.

I'll sleep on it, and maybe the morning will give me new ideas. (or someone cleverer will find out.)

[HenrikBC]

That's ...super weird. And I'm guessing you did your displacement test rendering in V-Ray in max? I'm in max 2021 and V-Ray 5.10.03 b00001.

[hermit.crab]

I did some testing and I managed to reproduce it - the banding happens even when directly exporting 16 bpc from 3ds Max (just needs a falloff tex to be visible). The same thing is happening in Arnold. I presume the issue is related to 16bpc .exrs specifically since ineed it does not happen with .pngs.

[HenrikBC]

hermit.crab So right now it seems that ^Lele^'s result is the odd one out here?

I'm still not sure if it's the EXR format - or where exactly the trains leaves the tracks in terms of this not working as good as PNG.
I found this in the OpenEXR documentation, but I lack the smarts to read this at a sufficient level to determine anything:
https://openexr.readthedocs.io/en/la...half-data-type

[^Lele^]

I have managed to repro this too, with any form of conversion to 16bpc fp EXR: from nuke too (i had to change the shader and lighting to see it.).
Identically to Alex, a 16bpc int PNG shows no stepping.

Doing a difference of the 16 and 32 bpc EXRs (yours, or the nuke converted one) results in 0.00009 (avg) and a max difference of 0.000244, which we should consider enough to show stepping.
The difference with the 16bpc int PNG, instead, returns a 0.0000079 (avg) and a max difference of just 0.00003, or about ten times less, clearly enough to not show stepping.

The empirical result seems to indicate one should use 32bpc fp EXRs, or 16 bpc int PNGs to displace, and avoid 16bpc EXRs converted from 32 bpc.
The theory part is that with 16bpc EXRs, one only gets 10 bits for the mantissa while the 16bpc int PNG has 65536 values to express the same range (or, 2^16).

[HenrikBC]

^Lele^ Then I did read the EXR documentation correctly after all. Thanks for the clarification. A shame though, but it makes sense that file sizes that are too good to be true, turns out to in fact be too good to be true.

[^Lele^]

Yes, you did, and my bad for not mentioning that!
I on the other hand had to talk to Vlado to get to the bottom of it. XD

edit: it should be mentioned that they are mighty fine, f.e., for images that don't contain data (i.e. beauty renders, color REs/AoVs) and even for data where the slight loss of precision is acceptable (f.e. normals).
It blows my mind that we can see the issue with precision at the fourth decimal place, as well.
The difference image i tested with is pure black, in other words, until it's multiplied by around 1000 times. (attached, the format's loss of precision, times 1000).

[HenrikBC]

Yes, you did, and my bad for not mentioning that!
I on the other hand had to talk to Vlado to get to the bottom of it. XD

Haha - I appreciate that!

edit: it should be mentioned that they are mighty fine, f.e., for images that don't contain data (i.e. beauty renders, color REs/AoVs) and even for data where the slight loss of precision is acceptable (f.e. normals).
It blows my mind that we can see the issue with precision at the fourth decimal place, as well.
The difference image i tested with is pure black, in other words, until it's multiplied by around 1000 times. (attached, the format's loss of precision, times 1000).

Agreed, I have been very happy with half float for everything else - especially for its above white and below black value capacity. I suppose the part that trips me up is that it's called "16 bit" where it might be more appropriate to call it "10 bit" if one were to compare the precision to eg. 16bit PNG.

And thanks for that image - it really hammers home what is happening. And now that I know, I can stop falling into this particular trap. Much appreciated, all of you in the thread - thank you!