We have 10 PCs/rendernodes on a farm. We often light our scenes with HDRI. Occasionally some rendered scenes have frames that look darker. When we look closer we can see that the frames that are darker simply have the dome light either turned off or missing the HDRI map (not really sure which). If this happens, it will only be one of the PC’s on the farm that does this, so luckily we can tell Deadline (that we are using to manage the farm) to re-render all the frames done by that PC. But I’d still like to resolve the problem as it wastes time.
What is really weird is that this doesn’t happen on every scene, and when it does happen is can be any PC/rendernode, so its not like one of the PC’s is having this problem, it can be any of them.
Because it doesn’t happen on every job and when it does happen its on a different rendernode each time. I’m really struggling to diagnose what could be causing the problem.
Here are some thoughts that I have on what could be causing it.
1. The Scene is taking up too much RAM on the render node and therefore is not loading in the HDRI
2. The network is being oveloaded with traffic and so the rendernode can’t access the HDRI file
3. There is some kind of licencing problem with VRAY
4. There is some abnormally with the network paths or location of the HDRI
Most of the above, I think would not be the problem, as things like licencing issues would simply stop the rendernode from rendering, not make it unable to see a HDRI map?
If anyone has any thoughts suggestions or advice on trying to find out what is going on, that would be great.
How do you manage your assets?
Do you have a library that each machine has to reach and find the HDRI maps?
If you are using relative paths to the assets, try using absolute paths.
Many thanks for the fast reply
We have the assets on a NAS (qnap) on the network.
Not sure we are using relative or absolute paths, how do we set this, and do I need to set it on all of the render nodes, or just the workstation that submits the job?
Thanks for the reply.
We have the assets on a NAS (qnap) on the network.
The HDRI is 21MB .hdr file
not sure if the network storage uses Samba or not (I’m not great with IT). But its a QNAP NAS. How would I find out?
That would indeed be Samba based. You should try disabling oplocks either globally, or at least for large files type that your render farm may access. Note that one would THINK that oplocks should only be an issue if one of the clients is opening the file for writing (which render nodes should not be). But for some reason this does indeed come into play.
Somehow you need to pass an option like this to your Samba config file. There is typically some GUI way to add such options. Alternately, there may just be a global switch to disable oplocks.
Note the syntax is case sensitive. So even the above would not work correctly if you named your files “myfile.Hdr” as only *.hdr and *.HDR are in the list. You can add all case variants if you want. It just gets long. But the safest test (and honestly not too much disadvantage) is just to disable oplocks. Then you can refine them later if need be with a veto_oplock_files list.
Honestly, a 21MB files doesn’t seem like the type of file to suffer from this (so maybe this is not your issue), but I haven’t used HDR much in years (use EXR). So I don’t know how susceptible they are to the issue. We actually don’t have EXR in our veto lists, and have never had a problem, even with very large ones. However the issue is exactly as you describe-- missing assets on various render nodes, completely at random (or loosely related to the order in which the nodes are started rendering the scene).
Sorry for the late reply on this. this is great advice. I’ve been super busy, but finally got around to doing this today. I turned off oplocks on the shared folder (we have one shared folder for work that has everything in it). Before we could really test if it worked, we found that it caused a really off side effect. After effects ran horribly when trying to playback footage from that share folder, as in played 1 frame every 30 seconds. so I turned it back on and After effects was fine again. So now i’m looking for a way to setup a separate shared folder with oplocks off just for the HDRI files to see if that works. Man this is complicated :-). Thanks for all the help though
Weird. No issues here with AE or other compositing apps running poorly. Of course I am using the veto oplock files option above, and we run TrueNAS on enterprise hardware. But no issues also with the backup server which is TrueNAS on prosumer hardware.
Did you reboot the machine running AE? Perhaps some vestigial oplocks remained or something and it was confused. Could try to disconnect the share from the client and remount it as well.
Hmm, that sounds like it could have been it. It was a colleague that reported it so I’m not 100% sure. We have managed to setup a separate share though now specifically for the HDRI files, so fingers crossed that will work. Thank you all for your help and advice
Finally got around to testing this. I setup a partition of the NAS and set it to ‘oplocks off’. Then copied all our HDRI files into that and sadly a machine will still not use the HDRI in the frames it does for that job. This is such an odd problem. Its super frustrating. I’m going to try to see if I can get ‘deadline’ to fail the frames when there is a missing map. Then at least another node can pickup the frames on the node that is dropping the HDRI file
I believe I am running into this bug… I have an EXR HDRI in my dome light, and it is flickering because it is failing to load on some nodes. Not a networking issue, since Everything else, and all other scene load fine. I think the dome light is very buggy right now. I have run into all kinda of issues. (Turn off LC and you get weird changes in the lighting that aren’t in the texture for one).
