When I render selected objects using the selected object mask (Progressive as well as Bucket), I get AA issues. Basically, it looks like any AA is ignored when using rendering selected objects. I’m using 3.50.04.
This is obvious in the beauty as well as as Alpha and other channels. It’s not scene specific and get it with all scenes
I also have problems with edges when using Render Selected BUT only with progressive sampler. Bucket seem to be working fine for me with Render Selected. The devs have acknowledged this issue with Progressive but it is still not resolved. Maybe your case is a bit more specific and manages to make bucket fail too? Can you post screenshots? Please check this thread.
It looks like I will need some additional information after all since I am not able to replicate that here, maybe my scene is way to simple?
Here are RGB and Alpha with Render Mask disabled: