Extreme aliasing in multimatte

[pixelapocalypse]

I'm trying to render out multimattes for a scenes that contains some buildings, 2 characters, a floor plane and a Pod in the foreground. When I mask out the building and the plane everything seems to work out fine.
http://i.imgur.com/Fcb8pk0.png

I also want to mask out the sky though because I want to put a new sky behind everything in comp. I have a straight plane that I assigned an object ID like the rest. When I render this out though I get pretty bad aliasing on the horizon.
http://i.imgur.com/2ml8R5l.png

I rendered this out in mental ray in a different way and the line are super crisp there. The problem with was that I didn't get even colors so I tried to do it in vray. Other shots I rendered out with multimattes don't have this problem....
http://i.imgur.com/CnPDNvI.png

What is causing this and how can I fix it?

[^Lele^]

low camera rays.
Use default settings, bucket renderer.

[pixelapocalypse]

I'm afraid I'm kind of a V-ray noob.
Where do I find camera rays.
Default settings for what? How do I set bucket render (isn't this standard?)

[^Lele^]

Default render settings.
Camera Ray is another name for AA ray
You're likely using a low "Max" value, or otherwise the noise threshold is forcing the renderer to stop way too early.
Either way, too few anti-aliasing (camera) rays are shot to clean the picture up.

Resetting V-Ray render settings to their default will get you in a good starting place.
I have no idea if in Maya the default isn't Progressive, like it is in max.
Either way, there are only two choices in the dropdown...
Aim for a 1-24, noise threshold of 0.01, and stuff WILL clean.

[yolov]

Yes, the progressive sampler is the default in Maya. What you need is to switch to Bucket (if you're using newer builds) or the Adaptive sampler (as it's called in older builds) and use 1 for min subdivs and 24 for max subdivs with a threshold of 0.01 like Lele suggested.

[emartin8907]

if you are still getting the problem, try increasing min sub d (under adaptive) to 2 or even 3 and see if that doesn't clean it up.

[pixelapocalypse]

Ah I was wondering where Bucket was but it is called Adaptive in my v-Ray build.
I reset to default settings. Used Bucket and 1 minimum. 24 max and 0.01.
Result is this. http://i.imgur.com/FRvb8j8.png

[yolov]

That's odd, because it looks like the progressive sampler.
Can you post the scene ?

[Neilg]

Are you rendering the main scene at the same time, or are you trying to do the multimattes as a seperate pass with an override material?
VRay does AA based on what it can see in the base render. if there's no distinction between edges in the base pass, you dont get any AA in the channels unless you raise the min. rate. If you want to render mattes with a black override you need the min rate going up to 4 or more.

[yolov]

Or you can see if enabling "consider for anti aliasing" in the multimatte options will do what you're after.

[pixelapocalypse]

Consider for anti aliasing doesn't change the result unfortunately.

[pixelapocalypse]

Are you rendering the main scene at the same time, or are you trying to do the multimattes as a seperate pass with an override material?
VRay does AA based on what it can see in the base render. if there's no distinction between edges in the base pass, you dont get any AA in the channels unless you raise the min. rate. If you want to render mattes with a black override you need the min rate going up to 4 or more.

I am rendering them as a separate pass with all other render elements deleted or turned off. I am also not using an override material as I don't know what is meant by that. I think I have seen this tab in a newer version of v-ray. Mine doesn't have it though. Using vray 3.10. I assigned a simple vrayLightmtl to all geometry to speed up the render.
I think I had the same problem earlier when I rendered zdepth passes for several shots because they were incorrectly setup by a teammate. I researched this a bit and concluded this should be pretty aliased as otherwise the zdepth information would be wrong but now in comp I get jaggy edges in some occoasions especially when using a zdpeth as rgbg to fake fog for example. As we also rendered motion blur this doesn't mix at all. So i'm thinking there is still something wrong with it somehow. This is a different problem and shot though.

The original scene i was talking about can be found here
https://dl.dropboxusercontent.com/u/...MatID_vray.rar

[Neilg]

I assigned a simple vrayLightmtl to all geometry to speed up the render.

That's an override amterial. you are overriding all materials in the scene with a new one. if you do this, you need to set the minimum rate in the AA to what your old max rate used to be.

If your base render is white on white, you will not get the same AA as you would otherwise. vray uses the contrast and color differences in the base render to perform adaptive AA that only goes as high as it needs - hence the use of a minimum rate and maximum rate. if you're forcing your base render to be white (or black, it doesnt matter) than there are no edges for it to detect, and it will use the minimum rate on everything.

[SonyBoy]

This is an interesting topic, and one that's currently relevant to some practices I've inherited at my place of work. I've generally rendered my matte passes along with my beauty renders in the past, so this issue has never come up. I can to some extent see the advantages of rendering a separate matte layer, especially if working with multichannel EXRs in order to keep file sizes manageable. But, if doing this to "save" render time but you have to crank up your settings super high, is there really going to be any noticeable time saving? Also, even if your quality settings are high, surely if rendering mattes with materials other than what's in the beauty passes, you run the risk of sampling not matching precisely, thus getting a matte that doesn't quite match the beauty?

[^Lele^]

The minimum AA level will mostly be overridden by the rendered, incurring in an overall minimal penalty, if any at all.
We have not been given a scene, or even an RGB pass to look at yet, so it's hard to be accurate, but should i be guessing, the RGB has low contrast there, for the noise threshold in question, and it skips after 1 ray is cast.
A min subdivision of 2 or 3 will cure the issue, without noticeable render time penalty.

As for sampling matte SCENES (not layers), which have been (hopefully procedurally) converted from a beauty one, as Neil explained, one samples the so called "Data" scene at fixed level, as no actual shading is occurring, and only the geometric properties are being evaluated (world position/geometric normal, mattes and so on), resulting in seconds-long renderings, regardless.
It's been done this way for a long time, because it's proven itself to work well: decoupling what will likely change (your lighting and shading), from what will likely not (the underlying geo/cameras and so on), is wise anyways.