.ext Vs .tx

[BenDel]

Hi,

is there a real benefit to use .exr versus .tx for 16b map ?

thank you

[Dariusz Makowski (Dadal)]

Tiled EXR etc should help with ram usage and so on... yea go EXR

[BenDel]

Do you have more informations about it please ?
unexpectedly, I have some some surprises with .tx
it seems to go faster with some renderfarm... (windows XP) and slower with others... (wondows7)
I continue my tests, but if someone has specifications about memory usage with EXR and TX....
I had allways worked with .exr, but 16b seems to be too much for my current renderfarm
thank you

[Dariusz Makowski (Dadal)]

The EXR did u use vray convener to make them tillable EXR ?

There is app in vray folder from chaos that u can use and convert images to tiled EXR...

[BenDel]

img2tiledexr.exe from Chaos to convert into .exr
and
maketx.exe from oiio to convert in .tx

both are mipmapped.
this is my first time with .tx in Vray. I use it because I have some issues with the memory charge.
I use .tx for 8b texture (img2tiledexr.exe does only 16b)

[Dariusz Makowski (Dadal)]

Humh interesting. Soo what issues are they? Are the textures really big or you just trying to run something on ur smartphone?

[BenDel]

Yes, textures are really big.... in fact no.... average is 6K, but I have a lot
1/4 are 16b...
as my current renderfarm is mixed (different kind of machines... different OS...) it's hard to get some real stats.
But on some nodes, .tx seems to go really faster than .exr .... so I'm surprise and i'm wondering what is the difference between .tx mipmap and tiled exr in the memory management...

[vlado]

The 8-bit .tx files are more memory efficient; OpenEXR has only 16- or 32-bit support so it has at least twice the memory footprint as an 8-bit .tx file.

Best regards,
Vlado

[BenDel]

Hi Vlado, and thanks for your reply.

I understand the difference between 8b and 16b.
What I'm wondering is: are the 16b .tx (mipmap) and the 16b .exr (tiled) acting in the same way during the computation ? I mean, memory usage, network access caching, loading/unloading etc... currently is there a real advantage to use tiled .exr vs 16b mipmap .tx ?
Does the compression option for tiled .exr affect the memory usage? I mean, uncompress the file for each access... does it slow down the render in case of huge quantity of maps ?
is the tile accessed at the bucket level ?
In fact, I'm trying to drastically optimize scenes with massive maps usage... So I try to really understand what depends of memory, network, CPU etc...
Maybe could you give me a link to get some specifications.... I can't find technical documentation on this, specifcally for VRay.

Finally, I have scripted my optimisation: all 8b textures are mipmap 8b .tx, and others are tiled .exr
Why don't you ditribute a a kind of maketx.exe for 8b maps? This solution seems to be unknown by VRay users... or maybe it's just me
The fact that this tool is not given with VRay is not really intuitive.

thanks in advance for your time!

[snivlem]

The 8-bit .tx files are more memory efficient; OpenEXR has only 16- or 32-bit support so it has at least twice the memory footprint as an 8-bit .tx file.

Best regards,
Vlado

Vlado, do these textures also help out the sampler (speed) or is it purely about memory efficiency?

[vlado]

It's only about memory.

Best regards,
Vlado

[stephane.bonnot]

Following your replies, I had a look at BenDel tests to see if high render times with tiled exr files were due to some bottleneck of our renderfarm.

Warning , tech details ahead !

The differences between 16bit uint tx and 16bit half exr are definitely linked to io and bandwidth limitations of our server and the way different Oses ( WinXp64 and Win7x64) are accessing it.

When only a few machines access the server, we don't see big differences between tx and exr tests.

This is quite different under heavy load. Under heavy load, tx scenes slow down a bit but keep a consistent output rate between machines. This is not the case for exr tests where Win7 machines eat up the bandwidth, keep a consistent rate , but slow down Xp machines by
a huge factor.

So on our renderfarm, for a scene with very high texture count/size, tx are much more efficient ( between 5x and 15x for the test scene ). To investigate a bit more , we ran several tests with BenDel and here are tech details that we found :
. IO read bytes are much higher on scenes with exr files ( very low on tx , memorymap ? )
. the render log seem to show that exr are entirely read on first hit, whereas tx are only 'mapped' ?

[log details and end of this post in next post ( too many characters)]

Stephane.

[stephane.bonnot]

Log with Tif files

Here is a part of the beggining of the logs of the same render scene with tif files, tx files, and a mix of tx and exr (all 16bit (uint for tx , half for exrs), tiled , mipmapped)

scene with TIF files :

[2013/Mar/15|11:51:28] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_color_HIGH_03.tif" loaded.
[2013/Mar/15|11:51:48] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_HIGH_OPA_03.tif" loaded.
[2013/Mar/15|11:52:08] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_color_HIGH_02.tif" loaded.
[2013/Mar/15|11:52:29] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_HIGH_OPA_02.tif" loaded.
[2013/Mar/15|11:52:50] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_facade_COL_03.tif" loaded.
[2013/Mar/15|11:53:02] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_facade_OPA_01.tif" loaded.
[2013/Mar/15|11:53:23] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_fenetresTranches_COL_01.tif" loaded.
[2013/Mar/15|11:53:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements_COL_03.tif" loaded.
[2013/Mar/15|11:54:05] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements_OPA_02.tif" loaded.
[2013/Mar/15|11:54:28] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements2_COL_01.tif" loaded.
[2013/Mar/15|11:54:50] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements2_OPA_02.tif" loaded.
[2013/Mar/15|11:55:13] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_fenetres_COL_03.tif" loaded.
[2013/Mar/15|11:55:19] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batD_f2/Df2_facadeB_color_v01.jpg" loaded.
[2013/Mar/15|11:55:23] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batD_f2/Df2_facadeB_alpha_v01.jpg" loaded.
[2013/Mar/15|11:55:46] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df2/Df2_BFacade_COL_02.tif" loaded.
[2013/Mar/15|11:55:59] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df2/Df2_BFacade_OPA_01.tif" loaded.

Note the time for each line, each texture is loaded sequentially, the time between log is proportional to tif file size. This seems like a normal behaviour as files are loaded when hit.
This is the slowest scene.

[stephane.bonnot]

Log with tx files

scene with TX files :

[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_color_HIGH_03.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_HIGH_OPA_03.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_color_HIGH_02.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_HIGH_OPA_02.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_facade_COL_03.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_facade_OPA_01.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_fenetresTranches_COL_01.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements_COL_03.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements_OPA_02.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements2_COL_01.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements2_OPA_02.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_fenetres_COL_03.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batD_f2/Df2_facadeB_color_v01.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batD_f2/Df2_facadeB_alpha_v01.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df2/Df2_BFacade_COL_02.tx" loaded.
[2013/Mar/19|11:09:44] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df2/Df2_BFacade_OPA_01.tx" loaded.

Note the time for each line, there is no pause when hitting the files, looks like only the requested tile is loaded/mapped.
This is the fastest scene.

[stephane.bonnot]

Log with exr and tx files

scene with tx and exr files
[2013/Mar/19|18:20:17] V-Ray: Preparing scene for frame...
[2013/Mar/19|18:20:17] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_color_HIGH_03.tx" loaded.
[2013/Mar/19|18:20:17] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_HIGH_OPA_03.tx" loaded.
[2013/Mar/19|18:20:17] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_color_HIGH_02.tx" loaded.
[2013/Mar/19|18:20:17] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batG_b3/Gb3_HIGH_OPA_02.tx" loaded.
[2013/Mar/19|18:20:21] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_facade_COL_03.exr" loaded.
[2013/Mar/19|18:20:21] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_facade_OPA_01.tx" loaded.
[2013/Mar/19|18:20:26] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_fenetresTranches_COL_01.exr" loaded.
[2013/Mar/19|18:20:31] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements_COL_03.exr" loaded.
[2013/Mar/19|18:20:32] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements_OPA_02.tx" loaded.
[2013/Mar/19|18:20:32] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements2_COL_01.exr" loaded.
[2013/Mar/19|18:20:32] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_ornements2_OPA_02.tx" loaded.
[2013/Mar/19|18:20:36] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df1/Df1_fenetres_COL_03.exr" loaded.
[2013/Mar/19|18:20:36] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batD_f2/Df2_facadeB_color_v01.tx" loaded.
[2013/Mar/19|18:20:36] V-Ray: Bitmap file "Z:/...../Modeling/wip/ben/batD_f2/Df2_facadeB_alpha_v01.tx" loaded.
[2013/Mar/19|18:20:39] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df2/Df2_BFacade_COL_02.exr" loaded.
[2013/Mar/19|18:20:39] V-Ray: Bitmap file "Z:/...../Textures/Facades/Df2/Df2_BFacade_OPA_01.tx" loaded.

Once again, note the timed log : all tx files create no pause in rendering, exr files take some time to load at this point ( proportional to their size, doesn't look like only a tile is loaded)
This is kind of mixed result, much faster than tif on faster Win7 machines, slower than tif on slower WinXp64 machines, but all in all much slower than tx (due to the higher load on our server and the "pauses" on exrs).