Problem with Reflectance / Refractance in Glass with Layers Render workflow

[Otaner-state]

I'm trying to split up a shot for a film we are working on, out a window with glass and rain on it. we've got it all looking how we like it and now trying to split it up into layers to have more control in the comp in nuke but encountering some issues with the window layer.

Issue Description

using Maya's render layers with V-Ray, splitting the scene into three layers:

Foreground – Spaceship/Cockpit
Midground – Clouds
Background – Mountains

However, when using this standard approach, I'm experiencing an issue with the refraction/reflection of the glass window (see Image 1).



Instead of rendering only the refracted parts of the glass in the cockpit layer, i'm getting a completely black output in the glass areas. When compositing the three layers in Nuke, the cockpit glass appears solid black, preventing me from seeing through it as expected (see Image 2).



My Question Is there a way to correctly handle refraction and reflection in Maya's render layers using V-Ray while maintaining accurate compositing results?

[Morbid Angel]

I think you must modify the way your shader's refraction is output through aovs. In particular if I understand the issue correctly, your alpha of the glass is rendering solid black. You must set affect channels in refraction - advanced tab to color+alpha, or all channels, depending on if you want the refraction's alpha to be propogated to other aovs. It also seems your glass's reflection/refraction is not a simple one with roughness in there, for that reason your bg has to be primary visibility off, but yet still be visible in refraction in order to be refracted/reflected in the raindrop and glass.
Lastly, by layering multiple refractive/reflective objects such as water + glass you have to set your reflection/refraction depth to 16 or higher in order to properly propagate the raytrace.

[hermit.crab]

What Morbid Angel said. Also, note that when the reflection/refraction affects All channels, the VRayReflection/VRayRefraction should be excluded from the comp. Otherwise, you'll get double of each.