I have a cube, vraymtl with a layered texture in diffuse, layered texture is 2 colors with a checker alpha.
Scene is lit by ambient light.
Try to bake and the layered texture is ignored. (see screenshot)
Also is it possible to bake out just the Diffuse or just the reflection without any lighting information, so basically just flattening all the texture layers etc?
I had do the same setup, and I think I figured out whats going wrong for you.
First of all, you can add a Diffuse render element, and you can use that as the unlit version. Or any other render element for that matter. For example a Raw Total Light to only get the lighting information.
Now, whats going on with your cube not getting the red checker. I set it up the same, and if I do a normal render, not a bake. The cube renders flat green too. So its not the bake that gets it wrong.
If you go into the checker, and turn on Alpha is luminance, it will render correctly! Its pretty strange however that in the viewport, it shows it being masked correctly, but if you check the connection in the hypershade, you can see that its the out alpha of the checker that gets connected to the layered texture, which is pure white unless you check Alpha is luminance.
You would need the “alpha is luminance” enabled in the checker texture > color balance. You’ll need to do the same for other procedural maps and for bitmaps that don’t have alpha (like jpg).
you can map outR, of your color texture instead of outAlpha, then you don’t need to check alpha is luminance. Or you can run a Rgb to Luminance node, which will essentially turn all channels into single channel.