Render nodes occasionally render frames without HDRI map

[clay_creative]

I believe I am running into this bug... I have an EXR HDRI in my dome light, and it is flickering because it is failing to load on some nodes. Not a networking issue, since Everything else, and all other scene load fine. I think the dome light is very buggy right now. I have run into all kinda of issues. (Turn off LC and you get weird changes in the lighting that aren't in the texture for one).

Did you ever find a workaround?

Ugh.

Sorry, just seen this. No I've not found a solution yet. though it sounds like your problem is different. glad you fround a solution though

[Vizioen]

Good old 3ds max.

[Joelaff]

OK. It seems it wasn't the poor little Dome Light. It was some geo that needed to be collapsed for... reasons.

[Joelaff]

NOOOO....

It is still doing it. Unrelated! Any clues?

[Joelaff]

Update:

Doctor, heal thyself... Somehow EXR had been removed from our oplock veto list... Added it and at least the textures don't have the loading issue anymore.

[Joelaff]

I believe I am running into this bug... I have an EXR HDRI in my dome light, and it is flickering because it is failing to load on some nodes. Not a networking issue, since Everything else, and all other scene load fine. I think the dome light is very buggy right now. I have run into all kinda of issues. (Turn off LC and you get weird changes in the lighting that aren't in the texture for one).

Did you ever find a workaround?

Ugh.

[Joelaff]

Interesting, and very frustrating. Are you actually getting errors for missing maps?

With our issues with oplocks it have no errors. Of course our issue was with assets other than maps (.aur files).

Have you tried EXR files instead of HDR files? if you are getting error messages they could prove useful of course.

[clay_creative]

Finally got around to testing this. I setup a partition of the NAS and set it to 'oplocks off'. Then copied all our HDRI files into that and sadly a machine will still not use the HDRI in the frames it does for that job. This is such an odd problem. Its super frustrating. I'm going to try to see if I can get 'deadline' to fail the frames when there is a missing map. Then at least another node can pickup the frames on the node that is dropping the HDRI file

[clay_creative]

Let us know if this solves it. I am very interested. It does sound like exactly like the issues we had that were solved by this.

Will do, cheers

[Joelaff]

Let us know if this solves it. I am very interested. It does sound like exactly like the issues we had that were solved by this.

[clay_creative]

Weird. No issues here with AE or other compositing apps running poorly. Of course I am using the veto oplock files option above, and we run TrueNAS on enterprise hardware. But no issues also with the backup server which is TrueNAS on prosumer hardware.

Did you reboot the machine running AE? Perhaps some vestigial oplocks remained or something and it was confused. Could try to disconnect the share from the client and remount it as well.

Hmm, that sounds like it could have been it. It was a colleague that reported it so I'm not 100% sure. We have managed to setup a separate share though now specifically for the HDRI files, so fingers crossed that will work. Thank you all for your help and advice

[Joelaff]

Weird. No issues here with AE or other compositing apps running poorly. Of course I am using the veto oplock files option above, and we run TrueNAS on enterprise hardware. But no issues also with the backup server which is TrueNAS on prosumer hardware.

Did you reboot the machine running AE? Perhaps some vestigial oplocks remained or something and it was confused. Could try to disconnect the share from the client and remount it as well.

[clay_creative]

Sorry for the late reply on this. this is great advice. I've been super busy, but finally got around to doing this today. I turned off oplocks on the shared folder (we have one shared folder for work that has everything in it). Before we could really test if it worked, we found that it caused a really off side effect. After effects ran horribly when trying to playback footage from that share folder, as in played 1 frame every 30 seconds. so I turned it back on and After effects was fine again. So now i'm looking for a way to setup a separate shared folder with oplocks off just for the HDRI files to see if that works. Man this is complicated . Thanks for all the help though

[joaquim_montserrat1]

That would indeed be Samba based. You should try disabling oplocks either globally,

This is the solution.

Note that on QNAP it's easy to disable the Oplock. Control Panel > Shared Folders > Edit Properties > Uncheck "Lock Files (Oplocks)."
https://docs.qnap.com/operating-syst...1C3ADADD6.html

This setting can be enabled / disabled per folder shared.

[Joelaff]

Thanks for the reply.
We have the assets on a NAS (qnap) on the network.
The HDRI is 21MB .hdr file
not sure if the network storage uses Samba or not (I'm not great with IT). But its a QNAP NAS. How would I find out?

Hey,

That would indeed be Samba based. You should try disabling oplocks either globally, or at least for large files type that your render farm may access. Note that one would THINK that oplocks should only be an issue if one of the clients is opening the file for writing (which render nodes should not be). But for some reason this does indeed come into play.

Somehow you need to pass an option like this to your Samba config file. There is typically some GUI way to add such options. Alternately, there may just be a global switch to disable oplocks.

veto oplock files=/*.aur/*.AUR/*.vdb/*.VDB/*.max/*.MAX/*.max/*.mov/*.MOV/*.tx/*.TX/*.ifl/*.IFL/*.tga/*.TGA/*.abc/*.ABC/*.prt/*.PRT/*.tyc/*.TYC/*.hdr/*.HDR/*.png/*.PNG/*.exr/*.EXR/

Note the syntax is case sensitive. So even the above would not work correctly if you named your files "myfile.Hdr" as only *.hdr and *.HDR are in the list. You can add all case variants if you want. It just gets long. But the safest test (and honestly not too much disadvantage) is just to disable oplocks. Then you can refine them later if need be with a veto_oplock_files list.

Honestly, a 21MB files doesn't seem like the type of file to suffer from this (so maybe this is not your issue), but I haven't used HDR much in years (use EXR). So I don't know how susceptible they are to the issue. We actually don't have EXR in our veto lists, and have never had a problem, even with very large ones. However the issue is exactly as you describe-- missing assets on various render nodes, completely at random (or loosely related to the order in which the nodes are started rendering the scene).

Further reading:
https://www.smallnetbuilder.com/nas/...cks-and-nases/