tiled exr's are HUGE!

[super gnu]

ok so im investigating ways to efficiently render massive textures (100's of sq km, 8k x 8k texture per sq km) and tiled exr's look like a good solution.. however i just converted my first exr, and the 8k x 8k image that was 20 mb as a jpeg, is now almost 400 meg as a tiled exr at default settings. as a comparison a png of the same file was 100 meg.


now what concerns me is network overhead if im trying to load 10's of these textures over the network to various machines on the farm.

can someone clarify where the benefit lies? is it simply less ram usage allowing larger/more textures to be used? or does it only fetch data from the network for parts of the file?

basically any speed / memory advantage will be negated if all the renderboxes have to drag 10gb of textures over the network...

[instinct]

I doubt the difference is to be explained by tiled vs. non-tiled but rahter by jpg vs. png vs. uncompressed

That depends on the compression used in the EXR. Also it does make a big difference what bitdepth you save to. 32bit float obviously needs more space than 8bit jpgs.

A tilebased EXR has the advantage that each of the tiles can be arbitrarily accessed without reading the rest of the file. This allows to a) only keep in memory what is currently
beeing WRITTEN to and b) only loads parts of an image that are currently needed. This does not necessarily result in more speed. Only if you are hitting paging. On the other
hand it also means that the renderer does not have to grab all the images as a whole, but can pull what is currently needed. This in turn might mean that parts are read
multiple times if they are needed at multiple times, so the double network hit might actually cause a slowdown.

As with proxys i would not suggest considering it as a method to speed things up, but as a method to render things that would not render anymore due to ram limits.

Regards,
Thorsten

[super gnu]

thanks Thorsten, yes i rather stupidly used the word "speed" in my post, i didnt actually plan to get any speedup out of it, its definitely the aim to render the unrenderable. however if the speed penalty is too great due to network access or other issues, i might have to consider other options. i guess ill just have to grab a bunch of big textures and test it out.


i had assumed that the network overhead would be -less- due to it not needing to load the whole map if its not needed, or a low res version if its distant.. however i didnt consider the fact that jpegs are actually incredibly efficient when it comes to network usage, and the compression in these tiled exrs seems a bit... crap..

actually, there are a number of compression options.. any tips on which will give me the smallest files? since the source is only an 8 bit image, any way to make the tiled exr 8 bit too? it (the tiledexr utility) defaults to 16 bit, with an option to do 32.. i dont need either.

[super gnu]

another option i guess, would be to copy all the maps locally to each renderbox before the big render..

[instinct]

I am not really uptodate in regards to the maketiled utility. EXR as a file format offers various lossy compression / lossless compression schemes. I have no idea which is used with maketiled.
8Bit is not an option with EXR, you can have 32bit float, 16bit half and UINT 32. 16bit half would be the smallest. As for compression schemes you might wanna check the EXR docs.

Regards,
Thorsten

[super gnu]

ok so i did an initial test on a scene with one 8k x 8k texture in. i rendered it with a jpeg first, then swapped to my 400 meg tiled exr.

i reopened max between tests.

with the tiled exr it took longer to open the file, that suggests its loading some large part of the exr with the file, maybe to do the material preview? i dont have textured displayed n the viewport (how does that work with tiled exr?)

rendering, the ram usage was similar, circa 2.4- 2.6 gig but a little higher with the tiled exr. ?! (unless i enabled summed area filtering for the jpeg, in which case the rendering used 3.4 gig)

rendertimes were similar.

one odd thing, i have the bitmap in both the diffuse and bump slots. the rendering with the tiled exr had defined shadows from the bump map, in the foreground where the pixels of the map were huge with lighter pixels in the map casting clear shadows, whereas in the jpeg version the shadows were not present.

this is with all filtering modes in both bitmap types.

anyone fancy trying to explain my results? at what point would i see the benefit of using tiled exrs for my textures? does the fact they are now 16 bit and my jpegs are 8 bit, offset the benefit? if so .. that sucks

[instinct]

Well i don't know how VRay or Max is handling tiled EXRs. I assume for VRay to make use of them you have to load them via VRayHDRI. In addition keep in mind that rendering wise you are looking at the same RAM usage usually. No matter how small the JPG is on disc it has to be decoded/uncompressed for rendering of course. So the only benefit is network traffic not ram. You are not going to see any difference at all unless you will actually hit the limit with JPGs. Until then VRay will always load them all into RAM as this will always be faster to access than disc or network.

Regards,
Thorsten

[super gnu]

ahh ok, so the tiled exr will always load the full quality image unless it hits ram limits? i was assuming that it would generally use less ram, because parts of the texture that were not visible would not be loaded, and distant parts of the map would be loaded from lower resolution tiles.

ive just done a further test, where the whole map is visible, tiling into the distance, this time i used the vray hdri loader for both the exr and the jpg.


with the exr, it used 3.46 gig of ram to render (average) and took 6 mins 47.

with the original jpeg, it used 2.88 gb of ram and took 5 mins 58 to render.

so.. hmm.

[super gnu]

also it seems i get a -much-nicer looking bump map with vrayhdri as the map type than bitmap, irrespective of the file type im using.