I’ve been trying to use the Render Mask feature but most of the time it produces nasty rough edges. Please check the image below to see what I mean. Notice the ugly edges around the blue hull of the boat. In this image I’ve used Adaptive sampler, antialiasing filter VrayLanczosFilter with size of 2.0 and BF+LC for GI. Is there something I can do for the edges to appear smooth?
Can you get me some way to reproduce this? A part of the scene will be helpful. This generally works fine on simpler scenes, but maybe there is something going on.
Best regards,
Vlado
Sure. Please let me know where I can upload the file securely and I’ll send the archived scene.
If you email our support guys to support@chaosgroup.com they will provide you with FTP info where you can upload the archive.
Best regards,
Vlado
Actually, no need to send this particular scene. I managed to figure out the exact steps to reproduce the problem which can be tested even with a simple scene. 
Please download the attached scene and try the following:
- Render the whole image with no Render Mask (don’t adjust any settings)
- Enable Render Mask with the option Selected
- In the VFB enable Glare Effect from Lens Effects
- Select the box and render again
The result is rough black outline around the object that appears only after the render finishes completely. So looks like this happens when Render Mask and Lens Effects are used in conjunction. I hope you can manage to reproduce it. If not, let me know and I will try to make a video of the steps.
rough_edges_render_mask_lens_effects.zip (590 KB)
We reproduced the issue and logged in our bug taking system. As soon as we get any update we will let you know.
I’m not using glow and lens flares but I still get weird fringing/edge around the objects that are rendered with Render Mask. Any way to avoid this or is it a bug? I’m using the progressive sampler and BF+BF. Tried bucket too but still the same.
Any chance to send us that scene as well, it might be the same issue but it might be something else?
Sure, I just sent you a stripped down version at support@chaosgroup.com Everything is set, you just need to make a small region render around the slatted ceiling. I’ve also provided a vrimg file to load into the VFB before making the region render so you have a point of reference.
Let me know if you need anything from me or if any questions. Thanks for looking into it!
Not trying to push you guys, I know you’ve got a lot of work around the launch of the new SP but I just wanted to know if there is any progress made on fixing the problem with using Render Mask with Lens Effects? You know, the black outlines that appear around the selected objects as seen in my first post in the thread. Is there a workaround that maybe I can use temporarily?
Thanks in advance!
The developers are currently working on it. It should be fixed soon.
Neat!
…
Hi Alex,
We managed to fix this bug for the next SP release.
Thanks so much for looking into this and fixing it!
seems like this was not fixed
It is fixed actually I just double checked it but it may be something else.
Can you send us a scene for investigation?
was sent in the file you guys have from me last, try to render any scene from it, some partitions in the file on layers
I still have problems with jagged edges/stepping/aliasing when using Render Mask with Progressive Sampler. If I render it with Bucket Sampler instead of Progressive the edges are perfectly smooth.
Please see the screenshot below and notice the jagged edges on the left curtain compared to the right one. Vray version - check signature. This is not related to the use of Lens FX, I don’t have them enabled in this image.
Hello,
We haven’t yet come with a nice solution for the render mask edges and the progressive sampler. For now there is only a warning about the problem.
Best regards,
Yavor
Ok, thanks.