VRay matte shadow makes noisy alpha

[TubeSmokeGuy]

Hi!

I have the feeling that when you render a shadow matte into the alpha channel, the shadow is super noisy.
Someone else noticed that?

The RGB pass looks already almost grain free but the alpha is still noisy.
How is that possible?
Is there a different way how VRay checks if the noise threshold for the alpha has been reached?

Without denoiser, to be clear.

Thanks and Greetings!

[hermit.crab]

How exactly are you testing (comparing) this? From what I've tested, the shadow noise pattern matches the RGB noise pattern.

[TubeSmokeGuy]

A colleague of mine who pointed me to that issue will prepare an example and elaborate more on that topic. As I said, I have no doubts that mathematically everything is correct and also measurement wise the values may be correct, but it's more a feeling. Looking at the RGB pass, I can say that it's almost noise free, but looking at the matte shadow in the alpha, I can say that it looks very grainy.

[julius_may]

Hello everyone, I am that mentioned colleague.

Here is an example of that problem: I rendered an image with an alpha based on direct Shadow from a sphere onto a matte plane. For the purposes of demonstration, I used very noisefree settings. But on a white background, you can see how the shadow, because of the alpha is pretty noisy.

200% zoom, so you can better see the grain:

200% Zoom Alpha:


However, the shadow does become very smooth, if you raise the min subdivs. I took a screenshot of that, as well and you can see how the shadow is basically noise free. So it seems like the system always uses min subdivs when calculating the alpha of the shadow, regardless of noise threshold settings
Here is the shadow at 200% again and even zoomed in, it looks very smooth.

[hermit.crab]

Could you attach the scene from the screencap here? I cannot produce such noisy results with those render settings, so there might be something else going on.

[julius_may]

Sure thing, its not the exact same scene, but it has the same results.

Max version is 23.3.0.3201
Vray version is 5.20.02 build 00001

You can see the shadow best when putting a white background into the Frame Buffer Background.

[hermit.crab]

I see. Sorry to say, this is a common issue with shadow catchers. Though it's barely visible, the alpha noise pattern still matches the RGB, though it's barely visible (check RGB values). Using a VRayDenoiser should give you the desired look without the render time cost of increasing the image sampler settings.

[julius_may]

Ah, alright then, that would explain it, thanks.
So far, our solution has been denoising and/or high min subdivs, too.

[caypiranha]

Well, I think if you may try settings like this, it could serve its purpose also being a bit faster and without using denoising.
What I never saw before, was a strange gap in the shadow in your original scene (left side), which turned out to disappear while using more samplings.

Decreasing the NoiseTheshold to 0,02 e.g. gives a even faster rendering (11,3 sec), while leaving the shadow even not to noisy.
If you now turn a denoiser on that ( I tried the Intel one) you get a ultrasmooth shadow.

[TubeSmokeGuy]

And well, using VRayDenoiser with Default Settings gives results like this: https://forums.chaos.com/forum/v-ray...disturbs-alpha

So, no clean and smooth matte shadow in alpha possible with VRay, without tweaking or postpro? I wish Chaos would make a more handy solution that gives smooth alpha shadow mattes directly / instandly.

[Joelaff]

The defaults are meant to make RGB look good without absurd render times.

If you tweak the settings as shown you can get those shadows with relatively reasonable render times. It's good to keep in mind the weird backwards nature where a higher quality often renders faster in these cases.

I could see this being an issue if you are trying to render part of your RGB with the catcher (like to get an object and its shadow in a single pass). Most of the time people will render the shadow as a completely separate pass with the objects set to unseen by camera.

Perhaps VRay could default to more samples for the catcher when using a shadow catcher (either the VRayMtlWrapper or the shadow matte options in the VRayProperties dialog)??

[caypiranha]

And well, using VRayDenoiser with Default Settings gives results like this: https://forums.chaos.com/forum/v-ray...disturbs-alpha

So, no clean and smooth matte shadow in alpha possible with VRay, without tweaking or postpro? I wish Chaos would make a more handy solution that gives smooth alpha shadow mattes directly / instandly.

Regardless of what Software you use: Maya, Max, or renderer like vRay and sure - many others: The desired "default" settings are sometimes not satisfying in the first place. But as we see on the thread you linked here or the solution which might be taken here, it could be possible to archive something, that´s quite nothing left to moan about.
High Quali shadows never were fast to render. And good quality pictures always took a lot of time wich never are coming out of the Box. Although nowadays, we render everything with global illumination, it´s not self-evident, that all this is also now rendering fast like hell and being high-quality.
It´s today maybe more like: the defaults give you a fast and timesaving start and you tweak it up until your needs are almost there. For first renderings and getting everything in place, it´s serving it purpose pretty well.
I find this almost better than turning first the HQ settings down to work faster on the basics and then afterward increasing them up again for final renderings. Then once it comes to getting final, just tweak e.g. the subdiv multiplier, and then it´s good to go.

[seandunderdale]

Hi quality shadows in the beauty, with denoiser are almost instant. Its the matte shadow -1 alpha shadows that are ignored by the denoiser, in my experience. Im doing some tests now, and if I disable matte shadow and render the floor, its instantly smooth...turn on matte shadow and the shadow is grainy and takes forever to smooth out, as if Im not using a denoiser at all. Even though in the render element it has "denoise alpha ticked.

Is there a workflow mistake going on here? How should we be rendering shadow catchers since they cant be denoised in render?

[hermit.crab]

Is there a workflow mistake going on here? How should we be rendering shadow catchers since they cant be denoised in render?

They can and you said it (that it took longer to denoise). If you have a scene, where the Shadow Catcher is not being denoised, do attach or send it. Also, are you using CPU or GPU? Note that if you're using progressive GPU, you need to update the VRayDenoiser layer manually to see the effect.

[seandunderdale]

Heres a scene. The cube denoises instantly in RT, but the matte shadow floor does not. If I do an offline render, it does denoise at the end, but during an RT render, I cant see any denoising to the shadow. Im using CPU rendering, and I am also using Maya...I just didnt want to start a whole new thread for it.