Hi guys,
I always get this and usually fix it in PS up to a point. Is this normal and no way around it? I now use Next, but it was the same with 3.6. My objects all have terrible edges when rendered, no matter what are subdivs or noise settings. Example attached.
Hi, Render Mask has no anti-aliasing by design, to avoid issues in compositing. If you want to have anti-aliasing you can use the alpha as a mask, or use a MultiMatte, VRayObjectID or VRatMtlID render element. Using VRayObjectID/VRayMtlID you have to set the output type to Color (with AA).