Nvidia denoiser creates jagged edges in the alpha channel AFTER render is stopped

[Alex_M]

I found an odd bug with the denoiser. If I use Nvidia denoiser and I stop the rendering through the "Stop" button in Max'es render progress window, the alpha channel becomes jagged and aliased. The alpha channel is perfectly fine and smooth before I stop the render. However, if I stop the render from the VFB window, the alpha channel remains perfectly anti-aliased and smooth even after rendering was stopped. Please check the screenshots below. This problem doesn't occur with Vray's own denoiser.

[Philip Shounev]

Hello,

Actually, the result is the same, without stopping the render at all - the edges appear to be a little bit uneven at the end. When it comes to stopping it, using the above-mentioned ways, here's the difference:
- stopping it through the max's render progress window - stops rendering, applies the denoiser and outputs an image
- stopping it through the VFB window - stops rendering, without applying the denoiser, no output image is written

Best regards,
Philip

[hermit.crab]

Is the image saving correctly, however? It is possibly because of a difference in previews. Is there a reason for using the Max frame buffer in such a manner?

[Alex_M]

Hello,

Actually, the result is the same, without stopping the render at all - the edges appear to be a little bit uneven at the end. When it comes to stopping it, using the above-mentioned ways, here's the difference:
- stopping it through the max's render progress window - stops rendering, applies the denoiser and outputs an image
- stopping it through the VFB window - stops rendering, without applying the denoiser, no output image is written

Best regards,
Philip

Hi Philip,

Thanks for the reply. However, I'm not too sure what you are trying to say. Are you saying that everything works fine? To me it doesn't. When I stop it through the render progress window, I get an alpha channel that's unusable for post production. The edges are clearly a lot more jagged and aliased. They are nowhere near as good as the alpha when stopped through VFB.

Please see the compositing examples below. One of the images was composited with the alpha channel taken when render was stopped through the progress window, and the other image was composited with the alpha taken when the render was stopped through VFB. I think you'll agree that the render stopped through VFB is clearly superior in quality and most importantly proper looking.

[yavor.rubenov]

Hello,

First of all - there's indeed an issue with the Nvidia AI denoiser - when it denoises the alpha it causes the jagged edges. We'll have to check if we can avoid it somehow or should we report it to Nvidia.

As for the difference when stopping the render through the Max stop button and the frame buffer stop button - as Philip tried to explain - pressing the stop button in the VFB aborts the rendering and doesn't apply the denoiser at all - so you get the proper rendered alpha. If you press the Stop button in the Max Render Window or just leave the rendering to finish - the denoiser is applied so you get the troublesome Nvidia behavior.
So if you want to use the alpha for compositing and you need denoising - you'll have to use the default V-Ray denoiser.

Best regards,
Yavor

[Alex_M]

Thanks for clearing that up, Yavor!