Opacity broken in VRayLightSelect in VRayEnvironmentFog

[beat_reichenbach]

When using light selects or VRayLightMix in combination with VRayEnvironmentFog and Matte objects the opacity in the light select channel does not match the opacity in the RGBA channel. This means it's not possible to use those light selects.
Use cases for these scenarios are creating atmosphere renderpasses with trees. The leaves will have opacity maps and will be held out as matte object in the atmosphere pass.
Even with the opacity mode set to clip in the middle, the transparent areas are visible.

Below is a screenshot with all three opacity types. RGBA at the top, and plused together light selects at the bottom. As you can see any values that are gray and not either black or white will not have the correct opacity in the light selects.

[hermit.crab]

The alphas actually do match, but because the VRayEnvironmentFog affects the alpha channel, the results are not as you'd expect. Currently, you cannot disable the alpha contribution, I'm afraid. However, we do have an improvement request logged (internal bug-tracker id: VMAX-9793) about the matter. I'll add this thread as well. You can work around it by rendering the Atmosphere in a separate render.

[beat_reichenbach]

Thanks for the info!

The alphas actually do match, but because the VRayEnvironmentFog affects the alpha channel, the results are not as you'd expect.

I see so basically we're looking at a unpremultiplied output when looking at the lightselect? Even if we premult the light selects, there are still weird behavior of the alpha where the areas behind the plane with alpha=0 are brighter than expected. (See attachment). Additionally when premultiplying the light selects, and recomposing (plus) them in Nuke, doesn't give back the beauty result. So might there be additional problems?

You can work around it by rendering the Atmosphere in a separate render.

Could you elaborate on this please? This render is the atmosphere rendered separate there is no geometry visible to camera. Or do you mean, render each light separate?

[hermit.crab]

Actually, you are right - the issue is a separate one - the lights are indeed brighter, but not because of the alpha. I've logged it (internal bug-tracker id: VMAX-11906) for developer investigation. Fortunately, the results from rendering in Standalone are as you'd expect. Use it as a workaround.