infuriating delays in working

[^Lele^]

ive just converted all textures to TX (and resized what i can)

No need to resize if you make them tiled TX.
It will make a world of difference also in interactivity with the material editor and scene, as the full sized texture is highly unlikely to ever load.
Have a read here.
Memory management will be all the better for it too, so you'll never worry about that again (at most, it'll render slow, but not run out of RAM.).
As for not clearing the cache between renders, it's done smartly enough that if the memory limit is reached it'll purge the right stuff first.
If the output is big, and with many REs, rendering bufferless will also provide for substantial ram savings.

[joconnell]

Here's a semi related question Lele - say we have a single high res texture and a mip mapped version that's being used in multiple places in a scene. If the high res texture is called once in shot is it then held in memory and doesn't take up any more regardless of how often it's called from then on? If we then take the same render but use a mip map, say the texture is again used from in front of camera to the far horizon, do we get any penalty from now using the high res version and all the various smaller res versions as it goes away from camera? As in say there's 10 levels of mip mapping, would vray see this as 10 different textures loaded into ram rather than our original high res map? Would we get different benefits later on in the process where the renderer is doing a lookup on the further away and smaller texture instead?

[^Lele^]

Heya John!

I am unsure as to how the code behaves exactly (and as we're nearing deadlines, anyone who would know is snowed under with work.).
In my experience, however, moving from full-res to tiled *always* had a major, positive impact on speed and memory, be it in stills, the material editor, or long sequences, with or without a good chance for reuse of a given mipmap tile.
Particularly in complex networking environments (say, a farm pulling the same-ish data from a central location, while artists work on the same scene to improve it), the speed at which high-res stuff can be pulled, compared to tiled, is a no-context.

One thing to be wary of, when you atomise the reads like so (even more so with many, small, or very voxelised proxies) is to ensure the network and disks are set up to deal efficiently with small reads.
One production i worked on slowed to a crawl because we had xMeshed complex rigged stuff, and the files were often a Kb or less.
The system, able to easily pull 100+MB/s with huge batches of data would literally be unable to hit 10 KB/s with the minute stuff.

Now, Proxies and Tiled TX should not suffer *exactly* in the same way, but you may want to ensure they behave, either taking some time to test on the network, or providing the slaves with a local cache (which then can likely sit on slower drives, as the data amounts will be a lot less.).

Feel free to mail me if you are running tests and want some company.

[Joelaff]

Just watch out for any of the memory mapped (or tiled in some internal method) assets like tiled textures (exr or tx) or VRay Proxies. They all put a heavy load on the server/network during render if you have a lot of render nodes. With 20-40+ render nodes we are talking a pretty saturated connection. We push all those assets to each render node before rendering, and they are pulled from each node's local drive during render. Much faster.

[^Lele^]

TL;DR: Proxies and .TX files will provide for a *much improved* load on any hardware, compared to original versions of meshes and textures.

Long version:
You make it sound as if the load is due to the Proxies and TX textures, but that isn't the case *at all*.
We're comparing pulling the same texture and meshes in non-atomic fashion (i.e. full res textures and meshes inside max), which would essentially front-load the *much higher* data amounts at max file load (Proxies offer also disk-space benefits, as the data is nicely packed, compared to even a zipped max file of the same explicit data.).

So, while proxies and textures are loaded *during* the rendering (particularly if one has read the guide i linked, and knows how to extract the most from the concepts, as explained in there.), they will provide for a *much* easier time on the disks and the network, rather than a harder one, by virtue of their design.

If one has Deadline, or similar tool, the farm can easily be managed so that clients start slightly staggered, and with different bucket orders from one another, thereby giving the network and file server an easier time.
Local caching is great, but it isn't always feasible: the copying *will* hobble the network, as one's just fronting the work, and the cache could become invalid many times over, during a production, and so needing updates.

One has to have very good, reactive IT managing it, or truly solid hardware and software pipelines which can deal with that stuff intelligently, and in the background (both of which John does have, right now. He works in a nice company.).
Not like this is pertaining to local caching only, either: the same goes for preparing and maintaining versions of the assets in proxy or .tx format, to be sure.

So, sure, cache locally for finals, and long sequences, but just moving to tiled and proxies will make the infrastructure's workload a lot lighter, compared to not doing so.

Memory mapped and tiled, for the record, isn't the same thing: we don't do memory mapped, that was MentalRay, we just do tiled, mip-mapped.

[Joelaff]

No doubt the data has to be moved at some point. But moving the data once with the push to the nodes has proven more effective here. I hear ya that the load is either when the scene starts, or throughout the scene, and that the throughout the scene likely put less load on the server. Either way, we found the pushing to the local nodes works best for most animations.

As a developer, I know what memory mapping is (I am referring to the mmap() call). I don't know what if any of the VRay code does this (I haven't disassembled it.. I'm not allowed to due to my license agreement , right.). However, dynamic loading is very similar, as you know. That is why I said "memory mapped (or tiled in some internal method)." All have similar network bandwidth profiles-- continuous usage during render. The formats we use (exr and tx) are BOTH tiled AND MIP mapped.

Pushing the files to the nodes requires an update before rendering, but usually only one or two assets changed. However, the read from local or remote disk for all the assets happens every time a render happens.

Staggering the start of the tiled/cached assets like with Deadline doesn't help much with the load. It helps with non-tiled/cached assets, sure, but not so much with the tiled stuff.

I do agree with the overall premise... using tiled textures is a good thing, and typically much faster in terms of total time.

[super gnu]

well i can definitely say that swapping to TX files has made a dramatic improvement to scene responsiveness here. my main concern was slowing down rendering, farm costs you see! anyway the files feel usable again and i have far fewer "ill go and make another cup of tea then" moments.

still have some quite long delays on render end but much better than before.

and the rendertimes, if anything, have gone slightly down . yay!

[^Lele^]

It had saved my bacon (albeit as exrs back then) a couple of times over in production, and so late into them to not have any business to do so.
Glad to hear it worked for you too!

[joconnell]

Cheers for that ^Lele^ - i'd say there's zero chance of being able to change our storage format since it's shared between all departments - we do a lot of sneaky behind the scenes stuff with volumes but I don't know if we'd be able to go to that level. I totally get yourself and JoeLaff's experience with that though, in my previous place we were getting 30 minute a frame times for a lookdev turntable on a scanline or pixo supplied asset but when it went to the farm, the 30 or 40 machines that were pulling data down totally ground to a halt and made things 3 times slower!

[super gnu]

Cheers for that ^Lele^ - i'd say there's zero chance of being able to change our storage format since it's shared between all departments - we do a lot of sneaky behind the scenes stuff with volumes but I don't know if we'd be able to go to that level. I totally get yourself and JoeLaff's experience with that though, in my previous place we were getting 30 minute a frame times for a lookdev turntable on a scanline or pixo supplied asset but when it went to the farm, the 30 or 40 machines that were pulling data down totally ground to a halt and made things 3 times slower!

i wonder if there has ever been a cgi project which did NOT involve continuous problem solving and unexpected software/hardware behaviour. i regularly find myself lurching from one wierd problem to another via 3 failed plugins and a random undeletable object or two.

at least it means we are unlikely to be replaced by AI quite yet.

[^Lele^]

hear hear!