Profiling network for best mipmap and vray proxy performance?

[joconnell]

Heya Folks!

We've had our first job in where we've had a tonne of very large environment extensions and the odd all cg establishing shot. Since there's so much detail we've proxied everything to make our main working scenes smaller. We've used a lot of forest and railclone for layout and have some very large cache files for pirate ships. The supplied ship models have a tonne of udims which we've cut down to about 20 tiles and the textures were converted into multires exr files. As you'd expect, rendering a single frame locally is okay but your frame time jumps massively when you send it to the farm and have 40 machines all trying to pull the same cache files at the same time. Rendering a frame is around 20 - 30 gigs of memory. Have you guys done any internal profiling of setups like this and come up with any guidelines on how to profile and configure a network and storage for best performance with this?

On slightly related notes, if you use light cache as a secondary gi method, it'll calculate GI for the entire shot in one go. I presume this kind of removes one of the benefits of the vray proxy since it can't be lazy loaded as needed, likewise with all the textures for a scene needing to load in right at the start and thus making everything very sluggish at the start?

Lastly, what type of penalty is expected for mip mapping? I presume it checks every texture space every frame for accuracy, would this cause a lot of overhead for a scene where you'd have hundreds of proxies using mip mapping or is it a fairly fast process? Or is it more like the older days of the vray proxy where mip mapping is a facility to render shots that you couldn't otherwise fit into system memory but causes significant slow downs?

Cheers!

John

[king_max]

My best guess for this kind of proxy handling would be to locally copy all the proxies to each machine and give a local path to it. (i.e. c:\proxy). It should be pretty straightforward with any freeware copy tool, or use windows command line xcopy. I cannot see any alternative to it other than investing a lot of money in building a 10G network. (good luck with that).

By "mipmapping" you refer to texture filtering?

[joconnell]

Yea, by mipmapping I mean the multi res tif or exr facility where the renderer can have a look at how far an object is from camera and then load an appropriate resolution of texture for a frame. If you use the vray img to multires exr converter you'll get a single file which contains your original image, a half res, a quarter res and so on, all the way down to a size you've specified. You load this using the vray hdri loader and it'll do all the nice multires things, the caveat is I'm not sure how much overhead this comes with though.

[king_max]

I actually never tried that feature. I guess Vlado would know more about the internal v-ray handling of such feature. What do you get out of it anyway? A bit faster render times for distant objects? I mean, if you used the single high-res map (instanced) it get loaded one single time and I guess the multires occupies much more RAM and a greater number of those files surely bottlenecks the network.

[joconnell]

Yep - it's mainly a way to either use one map file for both close and far shots or to be able to get around memory limitations of your render nodes.

[Morbid Angel]

what king_max said is very valid for the large data sets across networks. But its not so much the 1 Gigabit network vs 10 Gigabit network, rather your storage. You see, when you have 1 gig network, each machine can read up to 120 mb/s, so that's pretty fast, you can read a several gigs in a few minutes. However, if your switch as well as your storage cannot accommodate this high volume then it won't matter the speed of your network. The switch is far more crucial in this case and it can be expensive $20-$50 thousand dollars I worked on an isilon, a very advanced server, but it folded very quickly when we had a lot of data and everything crawled to a halt.

Anyways, your best bet is sync local. We used command script called robocopy on windows which would compare files by date and update / copy from network to nodes. So you have to do some environment stuff, so you can switch between local / network path easily.

But you quickly run into a problem, since most of our render machines had small HDD's they run out of space very quickly.

For the mipmapping, this is actually a good way of going about large amounts of textures. We had up to 5500 textures on a single model and couldn't render it in vray (originally setup by Ilm and render man). Have a look at open io and make tex. For example new maya comes with arnold that has the make tex built in, it might give you an idea of how it works.

[sharktacos]

We use mipmaps for everything. Makes things much faster, as it only loads into RAM what it needs based on the resolution. We use a Python script in Maya to automatically translate all our textures to mipmaps. You could probably adapt it to work in Max:
http://docs.sharktacos.com/texture/mipmap.html

[joconnell]

Yep - I used multires exr's for all of the feature assets if they were close up - not as nuts as your model dmitry but still around 1500 textures. Again I'm mainly wondering about the overhead of the renderer checking if the mipmapping is at the correct level for each texture at each frame, that could be a lot of loops of processing and if it's swapping between levels of resolution frequently it probably drops into the swamp of our network speed

Sync looks like the way to go in future then, cheers for the feedback!

[sharktacos]

John be aware that tiled exr textures, because they are always 16bit, are significantly slower to load into memory than 8bit mipmaps. In contrast, mipmap .tx files can be either 8bit or 16bit as needed.

My understanding is that the lag time is not in the renderer checking if the mipmapping is at the correct level for each texture at each frame, but in needing to read all the textures into RAM which would adversely affect network speed. So if it can see that it only needs to load a 512x512 8bit file for that frame, that is faster than loading a 4k file.

[Morbid Angel]

yeah, if I understand correctly, not entire texture is always loaded but only smaller part of it, so you rarely read the full size .tx in ram. Also .tx can be a 32 bit full float as well, so its a perfect format (biggest bang for buck) type thing.

[joconnell]

Sure - in our case we had everything as the multires exr's so it would have been pulling quite low res versions of the original 8k textures. As you mentioned, vlado had said that tx was more efficient for textures that were originally made as 8 bit, I must do a check to see what percentage benefit we get!

The scene itself was a bit nuts, nearly everything was a proxy so there was a huge draw on resources in here. Lele had previously mentioned when he was working on oblivion that they were able to configure the storage for the project with the optimum size of disk chunk for the size of file it would be regularly serving, I was wondering if there was something similar for vrmesh files where you could find out how big each "packet" of the file was.

Fun things

[king_max]

what king_max said is very valid for the large data sets across networks. But its not so much the 1 Gigabit network vs 10 Gigabit network, rather your storage. You see, when you have 1 gig network, each machine can read up to 120 mb/s, so that's pretty fast, you can read a several gigs in a few minutes. However, if your switch as well as your storage cannot accommodate this high volume then it won't matter the speed of your network. The switch is far more crucial in this case and it can be expensive $20-$50 thousand dollars I worked on an isilon, a very advanced server, but it folded very quickly when we had a lot of data and everything crawled to a halt.

Anyways, your best bet is sync local. We used command script called robocopy on windows which would compare files by date and update / copy from network to nodes. So you have to do some environment stuff, so you can switch between local / network path easily.

But you quickly run into a problem, since most of our render machines had small HDD's they run out of space very quickly.

For the mipmapping, this is actually a good way of going about large amounts of textures. We had up to 5500 textures on a single model and couldn't render it in vray (originally setup by Ilm and render man). Have a look at open io and make tex. For example new maya comes with arnold that has the make tex built in, it might give you an idea of how it works.

When I mentioned the 10Gb network, I was assuming that all the components can actually handle the 10Gb speeds. That means 10G network cards for all the render nodes + fast SSD on every node (best would be enterprise PCIe M.2 ssd) and you should be pulling all the assets from a 10Gb NAS running enterprise level SSD in RAID 10. Besides the switch, in your case (40 machines) you would also need SFP+ trancievers modules for each node etc... The fiber connections price grows exponentially with the cable length, so unless you're sitting in a small room with 40 machines, it gets really, really expensive. Anyway, I would really love to try a 10Gbe setup.