I have read some old posts in this forum, from 2007 -08, regarding poor antialiasing using the Multimatte render element. I am experiencing this in some areas of my multimatte render (very jagged edges here and there). In the old posts people said to check off the Object outline checkbox" if using Adaptive Subdivision" Image sampler. But I am using Adaptive DMC and Vray 1.5. What is the solution to this problem if using the DMC sampler? Help, it’s driving me crazy! 
Thanks
its a guess but, if you have areas where its really dark in your main render, there may be not enough samples there, so multi matte being a constant color would show the lesser sampling in those darker areas. If that’s the case you can increase min samples.
as morbid angel said multimatte takes it sampling from the main render, changing min samples will probably help but might increase your rendertimes alot aswell, in that case it might be worth considering doing a seperate render for the multimatte.
Sorry to sound like a noob, but would I increase the samples in the “Vray: DMC Sampler” area under the Settings tab? And is this the only menu where I would change the settings? Also, any recommendations on what numbers to put in there? My default was Min samples: 8, Global Subdivs mult: 1.0
Thank you so much for the help!
Just an update on this: I was told by chaos support that the multimatte is not meant to be an anti-aliased render so no matter how I change my sample settings, it will not help. If anyone knows differently, please respond. I still haven’t been able to find a good solution to make masks.
hmm thats strange. I thought the whole purpose of multimatte was that it was anti-aliased and that it suported primary colors.
Do you have an example of your rendering to show the aa problem?
I’m not sure what chaos were on about, but it can have aa and it’s all to do with the min setting as morbid said.
What are your AA settings?
What I meant was, that V-Ray does not try to produce a clean AA on the multimatte; it simply follows the AA of the main image. That means that the multimatte is antialiased, but only to the extent required by the RGB channel at that part of the screen. After all, you want to mask an object as it is in the RGB channel, and not to modify the sampling of the image itself just for the mask.
Best regards,
Vlado
OK. So what I need is for someone to tell me exactly how to set my settings (which settings, what tab they’re under, etc.) so that I can clear up the jagged lines? I am not clear as to what numbers to change and how high to set them. Thanks.
One solution is to increase the min. rate for the DMC image sampler; the other is to use the adaptive subdivision sampler with the “Object outline” option enabled.
Best regards,
Vlado
you can try 3/10 or any value above 1. You can also try adaptive subdivision with 1/3 for example.
YEAH, that solved my problem, thanks Vlado.