Floating Point precision in VraySamplerInfo

[bjornar_moland]

I'm trying to use VraySamplerInfo to generate an elevation grid raster, as used in GIS applications. However, even if I save as 32-bit EXR, the actual pixel-values seem to have a very low precision? This is also reflected directly in the Vray Frame Buffer, where values up to around 1000 are shown with only 1 decimal point, and values above are always shown as integers. The values shown in the frame buffer are not identical to the actual values saved in the EXR; while values in the frame buffer could be like 174.1, 174.2, 174.3 etc (or just integers for larger values), the values in the EXR might be 174.125, 174.25, 174.375 etc. for values in this range. Needless to say, the actual geometry is a smooth gradient (a piece of road in the attached image).

I have turned off any filtering/antialiasing, as that would result in erroneous height values along edges. Also, although this is GIS-data that normally belong in coordinate-systems with very large x/y coordinates, the geometry has been moved closer to the origin, to accommodate the general precision available in 3D Studio MAX. The largest x/y coordinates in the example image are in the range of a few thousand (less than 10000).

Why am I not seeing greater precision in these values?

[hermit.crab]

The VFB, indeed, shows only 4 digits, however, when saving 32-bit EXRs the values seem correct from what I tested. Are you certain your files are exported as 32-bit? When saving through the "V-Ray raw image file" option you need to tick the "EXR/VRST 32-bit output" checkbox located in the file browser window (check attached screenshot).

[bjornar_moland]

Thank you, you are correct. The reason I'm not seeing more precision is that I'm looking through the eyes of Photoshop; my bad! The Gimp will however show me the floating point values in all their precise glory. As will probably several other EXR-reading application (Fusion reads them fine for example). My main GIS app for rasters (Global Mapper) will not, though, nor will it read 32-bit TIF (SGI LogLUV) written from 3DS MAX. I tried exporting 32-bit TIF from the Gimp (I'm only interested in the blue elevation channel, so I remove red and green, and save the blue channel as monochrome), and these read fine in my GIS app, but there is an issue with the scale and/or gamut of the elevation data?

It's a bit hard to tell what exactly is happening under the hood here, but some observations; When I save out an elevation raster as so-called GeoTIFF from Global Mapper, I specify that the elevation unit should be meters. When I look at this file in the Gimp information panel, it will report monochrome Pixel values that match the elevation in meters. It will also report RGB values that are given as percentages that are x100 of the elevation, so a Pixel value of 150 gives RGB values of 15000%. In the Image\Precision menu, the image is by default set to "Perceptual gamma (sRGB)".

If I open an EXR from 3DS MAX, the Pixel values are also giving the correct elevation in meters, but now the percentages are different (approx. x6 for a value around 140 meters), and in the Image\Precision menu, the image is now set to "Linear light".

I don't expect you to be an expert on GIS/GeoTIFF formats, but any ideas about how to handle this correctly would be appreciated.

[bjornar_moland]

...from the Linear vs Perceptual gamma settings you'd assume that the Linear one would be...eh...linear, but it seems to be the other way around; the RGB percent is always exactly x100 in the sRGB case, while it varies with scale in the Linear case.

[^Lele^]

Yes, you'd need an application able to maintain a 32bpc FP pipeline, to be able to make proper use of those data channels (especially if you are indeed stretching the precision limit!).
LWF (i.e. the perceptual) is re-linearised, whereas Linear has an inverse gamma applied (as the monitor's not been inverted.).
It's confusing because it talks of the *display* (i.e. if that was linear, or sRGB), but the data is fine regardless (the picker ought to also always show the *data* values without correction).

Using Fusion or Nuke (or equivalent software.) will make your life a *lot* easier.

[bjornar_moland]

What I perhaps didn't state clearly is that when I export the 3DS MAX produced EXR to a 32-bit TIF, this results in wrong elevation values when imported into my GIS app (a max height of around 200m becomes around 9,5m). Interestingly, if I open this TIF into Gimp again, it's default is then set to Perceptual gamma, and the max height becomes 9,5 as in the GIS software. If I then change the mode back to Linear, the correct elevations are shown... I can't quite wrap my head around this. Is there some conversion or color profile that must be applied in the Gimp before exporting?

[^Lele^]

I have unfortunately no knowledge of GIMP.
All i can tell you is that the Exr is saved linearly by V-Ray (no gamma or CC is applied. "real" data is stored.), so any issue you may have is either with format conversion (to Tif. It may expect a gamma.) or with it being read back (by either GIMP or your GiS application.).

Maybe someone else used to these roundtrips may be of more help.

[bjornar_moland]

OK, thanks. I'm pretty sure the GIS app also saves (and reads) linear data, anything else just doesn't make sense, so I suppose it's the Gimp that must be complicating matters.