No doubt the data has to be moved at some point. But moving the data once with the push to the nodes has proven more effective here. I hear ya that the load is either when the scene starts, or throughout the scene, and that the throughout the scene likely put less load on the server. Either way, we found the pushing to the local nodes works best for most animations.
As a developer, I know what memory mapping is (I am referring to the mmap() call). I don’t know what if any of the VRay code does this (I haven’t disassembled it.. I’m not allowed to due to my license agreement :p, right.). However, dynamic loading is very similar, as you know. That is why I said “memory mapped (or tiled in some internal method).” All have similar network bandwidth profiles-- continuous usage during render. The formats we use (exr and tx) are BOTH tiled AND MIP mapped.
Pushing the files to the nodes requires an update before rendering, but usually only one or two assets changed. However, the read from local or remote disk for all the assets happens every time a render happens.
Staggering the start of the tiled/cached assets like with Deadline doesn’t help much with the load. It helps with non-tiled/cached assets, sure, but not so much with the tiled stuff.
I do agree with the overall premise… using tiled textures is a good thing, and typically much faster in terms of total time.8)
well i can definitely say that swapping to TX files has made a dramatic improvement to scene responsiveness here. my main concern was slowing down rendering, farm costs you see! anyway the files feel usable again and i have far fewer “ill go and make another cup of tea then” moments.
still have some quite long delays on render end but much better than before.
and the rendertimes, if anything, have gone slightly down . yay!
It had saved my bacon (albeit as exrs back then) a couple of times over in production, and so late into them to not have any business to do so.
Glad to hear it worked for you too!
Cheers for that @_Lele - i’d say there’s zero chance of being able to change our storage format since it’s shared between all departments - we do a lot of sneaky behind the scenes stuff with volumes but I don’t know if we’d be able to go to that level. I totally get yourself and JoeLaff’s experience with that though, in my previous place we were getting 30 minute a frame times for a lookdev turntable on a scanline or pixo supplied asset but when it went to the farm, the 30 or 40 machines that were pulling data down totally ground to a halt and made things 3 times slower!
i wonder if there has ever been a cgi project which did NOT involve continuous problem solving and unexpected software/hardware behaviour. i regularly find myself lurching from one wierd problem to another via 3 failed plugins and a random undeletable object or two.
at least it means we are unlikely to be replaced by AI quite yet.