Render nodes occasionally render frames without HDRI map

[clay_creative]

Hi,

This is a really strange problem.

We have 10 PCs/rendernodes on a farm. We often light our scenes with HDRI. Occasionally some rendered scenes have frames that look darker. When we look closer we can see that the frames that are darker simply have the dome light either turned off or missing the HDRI map (not really sure which). If this happens, it will only be one of the PC's on the farm that does this, so luckily we can tell Deadline (that we are using to manage the farm) to re-render all the frames done by that PC. But I'd still like to resolve the problem as it wastes time.

What is really weird is that this doesn't happen on every scene, and when it does happen is can be any PC/rendernode, so its not like one of the PC's is having this problem, it can be any of them.

Because it doesn't happen on every job and when it does happen its on a different rendernode each time. I'm really struggling to diagnose what could be causing the problem.

Here are some thoughts that I have on what could be causing it.

1. The Scene is taking up too much RAM on the render node and therefore is not loading in the HDRI
2. The network is being oveloaded with traffic and so the rendernode can't access the HDRI file
3. There is some kind of licencing problem with VRAY
4. There is some abnormally with the network paths or location of the HDRI

Most of the above, I think would not be the problem, as things like licencing issues would simply stop the rendernode from rendering, not make it unable to see a HDRI map?

If anyone has any thoughts suggestions or advice on trying to find out what is going on, that would be great.

Many thanks

Luc

[vladimir_krastev]

Hi, thanks for posting.

How do you manage your assets?
Do you have a library that each machine has to reach and find the HDRI maps?
If you are using relative paths to the assets, try using absolute paths.

[Joelaff]

Are the HDRIs on a file server? How big are they? Are they tiled (MIP mapped) EXRs ? What format?

Is the file server Samba based? What version?

[clay_creative]

Hi, thanks for posting.

How do you manage your assets?
Do you have a library that each machine has to reach and find the HDRI maps?
If you are using relative paths to the assets, try using absolute paths.

Many thanks for the fast reply
We have the assets on a NAS (qnap) on the network.

Not sure we are using relative or absolute paths, how do we set this, and do I need to set it on all of the render nodes, or just the workstation that submits the job?

Many thanks

Luc

[clay_creative]

Are the HDRIs on a file server? How big are they? Are they tiled (MIP mapped) EXRs ? What format?

Is the file server Samba based? What version?

Thanks for the reply.
We have the assets on a NAS (qnap) on the network.
The HDRI is 21MB .hdr file
not sure if the network storage uses Samba or not (I'm not great with IT). But its a QNAP NAS. How would I find out?

Many thanks

Luc

[vladimir_krastev]

If you have a server and all render nodes look for the HRDI in the same place it makes sense to use absolute path for it.

This is how absolute path looks in 3ds max's asset tracker:


This is how relative path looks like:


Check Autodesk's help pages as well - Paths Menu (Asset Tracker)

This may not be the reason for your issue, I am sharing it just as a good practice.

[Joelaff]

Thanks for the reply.
We have the assets on a NAS (qnap) on the network.
The HDRI is 21MB .hdr file
not sure if the network storage uses Samba or not (I'm not great with IT). But its a QNAP NAS. How would I find out?

Hey,

That would indeed be Samba based. You should try disabling oplocks either globally, or at least for large files type that your render farm may access. Note that one would THINK that oplocks should only be an issue if one of the clients is opening the file for writing (which render nodes should not be). But for some reason this does indeed come into play.

Somehow you need to pass an option like this to your Samba config file. There is typically some GUI way to add such options. Alternately, there may just be a global switch to disable oplocks.

veto oplock files=/*.aur/*.AUR/*.vdb/*.VDB/*.max/*.MAX/*.max/*.mov/*.MOV/*.tx/*.TX/*.ifl/*.IFL/*.tga/*.TGA/*.abc/*.ABC/*.prt/*.PRT/*.tyc/*.TYC/*.hdr/*.HDR/*.png/*.PNG/*.exr/*.EXR/

Note the syntax is case sensitive. So even the above would not work correctly if you named your files "myfile.Hdr" as only *.hdr and *.HDR are in the list. You can add all case variants if you want. It just gets long. But the safest test (and honestly not too much disadvantage) is just to disable oplocks. Then you can refine them later if need be with a veto_oplock_files list.

Honestly, a 21MB files doesn't seem like the type of file to suffer from this (so maybe this is not your issue), but I haven't used HDR much in years (use EXR). So I don't know how susceptible they are to the issue. We actually don't have EXR in our veto lists, and have never had a problem, even with very large ones. However the issue is exactly as you describe-- missing assets on various render nodes, completely at random (or loosely related to the order in which the nodes are started rendering the scene).

Further reading:
https://www.smallnetbuilder.com/nas/...cks-and-nases/

[joaquim_montserrat1]

That would indeed be Samba based. You should try disabling oplocks either globally,

This is the solution.

Note that on QNAP it's easy to disable the Oplock. Control Panel > Shared Folders > Edit Properties > Uncheck "Lock Files (Oplocks)."
https://docs.qnap.com/operating-syst...1C3ADADD6.html

This setting can be enabled / disabled per folder shared.

[clay_creative]

Sorry for the late reply on this. this is great advice. I've been super busy, but finally got around to doing this today. I turned off oplocks on the shared folder (we have one shared folder for work that has everything in it). Before we could really test if it worked, we found that it caused a really off side effect. After effects ran horribly when trying to playback footage from that share folder, as in played 1 frame every 30 seconds. so I turned it back on and After effects was fine again. So now i'm looking for a way to setup a separate shared folder with oplocks off just for the HDRI files to see if that works. Man this is complicated . Thanks for all the help though

[Joelaff]

Weird. No issues here with AE or other compositing apps running poorly. Of course I am using the veto oplock files option above, and we run TrueNAS on enterprise hardware. But no issues also with the backup server which is TrueNAS on prosumer hardware.

Did you reboot the machine running AE? Perhaps some vestigial oplocks remained or something and it was confused. Could try to disconnect the share from the client and remount it as well.

[clay_creative]

Weird. No issues here with AE or other compositing apps running poorly. Of course I am using the veto oplock files option above, and we run TrueNAS on enterprise hardware. But no issues also with the backup server which is TrueNAS on prosumer hardware.

Did you reboot the machine running AE? Perhaps some vestigial oplocks remained or something and it was confused. Could try to disconnect the share from the client and remount it as well.

Hmm, that sounds like it could have been it. It was a colleague that reported it so I'm not 100% sure. We have managed to setup a separate share though now specifically for the HDRI files, so fingers crossed that will work. Thank you all for your help and advice

[Joelaff]

Let us know if this solves it. I am very interested. It does sound like exactly like the issues we had that were solved by this.

[clay_creative]

Let us know if this solves it. I am very interested. It does sound like exactly like the issues we had that were solved by this.

Will do, cheers

[clay_creative]

Finally got around to testing this. I setup a partition of the NAS and set it to 'oplocks off'. Then copied all our HDRI files into that and sadly a machine will still not use the HDRI in the frames it does for that job. This is such an odd problem. Its super frustrating. I'm going to try to see if I can get 'deadline' to fail the frames when there is a missing map. Then at least another node can pickup the frames on the node that is dropping the HDRI file

[Joelaff]

Interesting, and very frustrating. Are you actually getting errors for missing maps?

With our issues with oplocks it have no errors. Of course our issue was with assets other than maps (.aur files).

Have you tried EXR files instead of HDR files? if you are getting error messages they could prove useful of course.