In theory the mask would be ok because mask clips values > 0 and opacity clips values > 0.5 afaik. So the mask would be correct but only if those objects do not occlude each other. And you need a base rendering with clipping turned on of course. So all in all not feasable. Only solution is the ShadowCatcher workaround so far.
But it seems like when using regular methods and using png or other formats you will get edges - if you are not using the method I posted above. You’ll get some kind of edges and you’ll need to remove those with certain methods in AE or Nuke.
I’m not convinced alpha is working correctly (and by extension the masks aswell). I set up a scene with a tree that has a black background and a 1.0 white self-illumnated material on the tree(no opacity maps), with tonemapping disabled and the image filter set to none. In my screen caps from the VFB you can see the difference between the two… Perhaps I’m misunderstanding something, but in this scenario shouldn’t they be identical? What’s stranger is that the alpha shows it actually encroaches 1px into the beauty with a black background. If you do the opposite scenario, 0 albedo tree with white background, the alpha expands past the bounds of the beauty by 1px, producing the halo effect. Peculiar.
I think you have to apply gamma 2.2 to alpha or 1/2.2 to the beauty and then compare the result - it should match (don’t use JPG doing so), maybe you can check this.
But once again @sebastian___ and RomasNoreika: The original issue is NOT about alpha or premultiply or rendering on non-black or whatsoever. It’s the Corona mask which does not take opacity fully into account, appart from any post problems that may arise. The comparison I posted (https://forum.corona-renderer.com/Corona masks RGB no clean edges why?#msg115947) has a white alpha everywhere (BG and FG is rendered).