V-Ray forgetting assets mid sequence

[jonathandeblok]

Running into an issue where were V-Ray is forgetting it's assets. (version info is in the log) Below are the logs of two consecutive frame (117 en 118 which are in the middle of larger sequence). 1-117 render fine and then suddenly 118 and beyond renders out empty frames.. these are 100% static scenes with only an animated camera.

logs for both frames look different, see below, any clues to what could be going on?

edit: forgot to mention, this happens at random, re-queuing the same job using the exact same files "fixes" it, or not but then failing at some another frame. And we're running multiple vray rtx sessions at the same time on the same machine. (all very lightweight scene that easily fit into s 3090)

log.txt

[hermit.crab]

What kind of network storage are you using? Could you try storing the assets locally and see if that makes any difference?

[jonathandeblok]

What kind of network storage are you using? Could you try storing the assets locally and see if that makes any difference?

Yeah that is the next thing on the list.. it's quite a slow NAS and we're rewiring things for this project to have a dedicated machine with fast storage that isn't used by the entire company at the same time. Most failed frames happen overnight when things like backup maintenance jobs hit heavy on the NAS as well. So there is a big chance that is the root cause.

That being said.. I was sort of under the impression/hoping the when you tell Standalone to render out a sequence it will load things in the beginning and keep them in memory for the next frames.. but I'm not really sure that is the case anymore looking at various forum posts and logs. Does it reload all geo and textures for each frame?

And is there a way to set longer timeouts for this before it gives up? And/or atleast a way to enable throwing errors for this? it failing silently make me feel a bit uneasy

[hermit.crab]

When rendering with GPU, textures are loaded/unloaded on every frame. Regarding the geometry, it is a bit complicated since it depends on the dynamic memory limit or if there's animation present - in such cases geometry is reloaded. Otherwise, there is a known issue with QNAP NAS storages, where proxies were not always loaded and your case might be similar. I suggest you check out the topic where it was discussed.

[jonathandeblok]

When rendering with GPU, textures are loaded/unloaded on every frame.

Ohh that's a bit of a derailment.. as in having 10 machines read a few 100 assets from the NAS is ok if it's done once per large batch of frames.. but pumping a few GBs of the same data over and over again for each frame to 10 machines from a single NAS is not going to fly well. How does this even work in the bigger studios with even larger assets and animations? I've searched the docs, there isn't a build in to localize the assets right? (for non-DBR)

So to solve this I need to code some things that pre-syncs all the assets to all the slaves locally and then repath the vrscene files on the fly?

[hermit.crab]

Most likely. You may try localizing the assets by archiving the scene from the host-platform.