Problems and crashes when using Nvidia denoiser with Threadripper 1950x in distributed rendering mode

[Alex_M]

I'm getting problems and crashes when I use Nvidia denoiser with a couple of Threadripper 1950x render slaves. I'm not sure if it's related to the problem, but both render slaves have very basic GPUs - Nvidia 210 512 mb which are just for display output and nothing fancy. Could the problem be related to this?

Here are the issues in more detail:

Issue 1: When rendering with the 1950x slaves with Nvidia denoiser turned on and using the bucket sampler, Vray doesn't construct the Nvidia denoiser's channels properly which leads to improperly denoised image after the rendering is finished. Please check the screenshot below. All buckets with pure purple color are the Threadripper 1950x slaves.



Issue 2: When rendering with the 1950x slaves with Nvidia denoiser turned on and using the the progressive sampler, Vray crashes right away. I have the 3dsmax minidump file of the crash if you need it.

Both issues happen in every scene I've tested so far so I don't think that the problem is related to the contents of the particular 3ds max file. Also, there are no crashes if I use a Threadripper 3970x machine as a DR slave, but I should also point out that this machine also has a different GPU - Nvidia GTX 980.

Software used: Max 2021.2 + Vray nightly 5 (5.00.30 build 30324)

[nikoleta.garkova]

Hi Alex,

Thanks for the detailed info. Please note that since update 2.1, Nvidia AI denoiser works with Maxwell and newer GPU architectures only. So yes, the issue is most likely linked to the GPUs used. For further information, you can check here: https://docs.chaosgroup.com/display/...noisingEngines

Regarding issue 2: Are you able to reproduce this on the Nvidia GTX 980 machine? If so, please do attach the dump file so we can check.

[Alex_M]

Thanks for the reply, Nikoleta. So to confirm, if we want the render slaves to render an image that uses the Nvidia denoiser, we would have to install compatible GPUs on them, is that correct? Essentially, the denoising is done on each slave as well, not just on the render host?

To answer your question, no, we don't see Issue No.2 on the GTX 980 machine (Threadripper 3970x + 64GB DDR4 RAM). Both issues happen only on the render slaves with Threadripper 1950x + GeForce 210 512mb + 32GB DDR4 RAM.

[nikoleta.garkova]

Apologies for misleading you there - actually the denoising should be happening on the render host only, so maybe this is a completely different issue (not related to the denoiser itself) as it also seems there is a difference in the buckets. Can you make a few tests to confirm this? Are you sure the software versions are unified on all the machines? Would it be possible to also forward the crash dump files? Thanks!

[Alex_M]

Yes, I can do a few tests for you. I'll be waiting for more information on that.

The Vray plugins are the same version across all machines, both the host and the render slaves - Vray 5 nightly (5.00.30 Build 30324).

I've just sent you an email with the scene and crash dump file. The support ticket number is: 320-089-8097.

[nikoleta.garkova]

Thanks, we'll review and get back to you ASAP.

[nikoleta.garkova]

Hi Alex,

Thanks for the provided scene files! We've managed to reproduce this - it seems that the slaves with old GPUs don't recognize that NVIDIA AI Denoiser is set, and VRayDenoiser Render Elements are constructed instead - essentially causing these discrepancies. We've reported this in our internal bug-tracking system (ID number: VMAX-10372) for developer analysis.

[Alex_M]

Good to hear.