Anti - Aliasing Halo

[csmithcg]

Ok guys, we've tried for a week to crack the code on this issue and have looked at everything from AA to color mapping to post solutions in AfterEffects. We have a band aid worked out that will help us meet our deadline but we're concerned our previous workflow is now invalid.

Heres the problem. We usually render a bg pass with just the sky bitmap in the max environment map, followed by a pass that includes all the hardscape and buildings that has the sky sequence rendered previously in the environment slot. Followed finally by a landscape pass that has all the buildings and such as a vray material wrapper matte surface, with the previous sequence in the env slot. Heres what we get.





the first is the shot comped where you can clearly see a halo around the hardscape pass. I've made the background green to help illustrate the point. The next two images are the layers by themselves. Any advise would be greatly appreciated.
Thanks,
Chad

[DaForce]

Looks like it could be a pre-multiplied alpha problem.Or im just not understanding how your doing it.

Who do you choose to do it in 3 layers as opposed to just one? Heavy scene?

[csmithcg]

very...

yes very heavy scene...and we've been doing it this way to allow more control over blending the two together. We are beginning to understand that this may not be the most modern approach but it has worked flawlessly up to this point....go figure..

In response to your pre-multiply suggestion. We thought that too!! Since then we've tried it in every possible configuration as well as several file formats and bit depths...as far as pre-multiplying goes...what is your suggestion??

TIA

-Chad

[Tdog]

You could check to see if you have "Dont Antialias Against Backgound"
checked, I think the default has it unchecked

Customize/Prefrences/Rendering/background

[Da_elf]

is green really the background you chose? its definately the pre-multiply problem. seems like using a green screen plugin to composite them might work better and forget the alpha channel. i prefer using straight rather than pre-multiply

[Seraph135]

This is not a pre-multiply issue...its the way the plants pass was rendered with the geometry of the hardscape matting it. In a non-premultiplied image the alpha channel is still identical to that of a premultiplied image...Its only the RGB channels that are affected by premulitplication. The problem in this scene is with the alpha channels.

Here is an example of what is happening.



When you matte things this way you get incorrect alphas because the background objects actually should exist behind the buildings, but your cutting them out.

The only fix for this is to not matte your background with your foreground.

Tim J

[Da_elf]

so let me get this straight. your taking the background, then cutting away parts of it to make room for the foreground then applying the forground onto the background? whats with the background matted by forground that you have there top left?

[Seraph135]

The top left image in my example is set up the same way that Chad set his up. His background objects (trees) are matted by his foreground objects (hardscape). Or in my case box matted by teapot. Look at his examples and you'll see that he matted the trees with the building.

Tim J

[Tdog]

Sup Tim

[Da_elf]

who is chad. what example (link) and that doesnt seem to make sence to me. why not just overlay the forground over the background without matting out the background.

[csmithcg]

and the answer is........

SIMPLE.....don't matte your bg with your fg. thanks a million....we will definitley be revisiting our pipeline for the next go round. I think why it failed on this attempt was the inclusion of so many 3d landscape elements both in the foreground and background. If I take what DR. Jones is saying correctly, a more appropriate configuration would be to divide the scene into 3 elements foreground,middle, and back. That sound right??

THANKS EVERYONE FOR YOUR COMMENTS!!!

this was our first foray into the VRay forum and it has been a very pleasant experience.

-Chad

[Da_elf]

well. hope to see some finished work in the images section from you then

[csmithcg]

be happy to post some pics once we get it all wrapped up, in the meantime check out our site...has tons of 3d work

www.kitchensinkstudios.com

thanks again,
-Chad

[csmithcg]

Tim,

Is there any particular reason this solution has worked for us in scanline. This has been a very successful workflow for us up to this point i.e. being able to split the scence into two to three passes. Trying to part it out into layers sequentially seems like a daunting task. Interested to hear your thoughts...

Thanks,
-Chad

[Seraph135]

I don't think its possible to make the problem completely go away, but in some cases those transparent edges may be a lot less visible and maybe they just weren't noticed. If the pass behind those matte edges was a similar color or even a darker color it would be much harder to see the issue. Motion blur also plays a big role. The larger the blur the more obvious the matte edge will be. Another reason your seeing it here and maybe not in other projects is the camera angle. Your merging these passes over a bright sky, but if the camera was a more aerial shot then you'd likely be merging over the hardscape pass only and you wouldn't have a problem.

Notice in your original post that edges only appear where your merging over the background. Part of your plants pass is not being merged over the sky (hardscape pass exists behind plants pass) and in those areas you don't have matte problems because there the hardscape pass isn't being cut out by the plant objects.

Tim J