Recommended settings to reduce fringe-ing antialias on trees?

[windrockwater]

I am using a variety of trees and plants that have opacity mapped leaves. I am getting a lot of anti-aliasing on the leaves which is creating problems when i want to composite backgrounds in. Can anyone recommend some render settings or changes to minimize this effect? For now i am using photoshop to clean it up but that seems like a 'hack'. It seems that there could be some settings to help but I am not sure where to start.

Thanks a bunch!

[Alan Iglesias]

How do the opacity maps look? Is there a nice even multi-pixel blend (number depending on the map resolution) in the transition between the white and black? If not, that may be causing your problem and you could do a little blurring in photoshop to smooth out the transition.

Also, I've seen folks try to use compressed images like jpg for intensity maps like this, and it is usually a bad idea. Make sure all intensity maps are pristine and uncompressed as compression artifacts will often cause all kinds of problems.

-Alan

[windrockwater]

So if I open a blank drawing and render a few of the same trees, there are no anti-alias issues. So i am not sure if because the trees are in the background that it creates antialias problems? Any tips other than adjusting each of the trees' opacity leaf bitmaps? Are there settings in Vray or Max to jack up to minimize this? Or other settings and methods?

[Dariusz Makowski (Dadal)]

U need to premultiply alpha if I'm not wrong...

[windrockwater]

I should have mentioned that I am using PSDmanager with default settings. One of the default settings is Premultiplied Alpha (box checked). Any other ideas? Is it sampling? Or maybe settings within the Xref materials for the trees?

[dlparisi]

You're best off rendering against a black background but even then that won't completely get rid of the fringing as Photoshop doesn't properly handle premultiplied images.

EDIT: You can also try "Remove Black Matting" under Layer-->Matting (you have to apply the layer mask first though).

[Dariusz Makowski (Dadal)]

Is that coming from refraction pass? Or just transparency, just put 4x group and apply same alpha to each image, each group will decrease the edge ever so slightly...

[trick]

ALSO: make sure that the opacity map for the leaves is 'treated' as a gamma=1.0 image through the whole process. If there is only one phase (import, export/save, PSD-pass) in the process where the texture is seen as a gamma=2.2 image the greyscale anti-aliased edges are brighter as they are supposed to be and therefore the alpha-mask is slightly larger too !!!

[dans]

For the following I'm assuming your scene has a background color/image or an environment map set (this wasn't 100% clear to me from your description). So I'm sorry if the following is already basic knowledge to you.

Problem:
That you are using psd-manger and Photoshop doesn't matter. The problem is that you are rendering against a blue sky background. So due to anti-aliasing (and also transparency) many leaves get a blue tint as expected. This works nice as long as you don't change the background dramatically or have large transparent areas (like with the leaves here). There is no simple way to get rid of the blue tint. (In theory it is possible to reconstruct the original leave colors by using the alpha channel and the original background image. - but it's not the best and easiest way to do it).

Solution:
You need to render against a pure black background. So set the Background Color/ Environment Mapping to black. That will later allow us to undo the anti-aliasing against the black background and compose any other background properly.

Preserving the reflections of the original background:
If you now lost the reflections of your background e.g. in windows/pool etc. then:

• The old school: Wrap a huge sphere around your scene, give it the original background color/mapping, Then change the Object Properties (you find it in the right click menu): Visible to Camera: off, Cast & Receive Shadows: off. Now the pool still "sees" the sphere and shows the old reflections again.
• Or the V-ray way: In the V-ray render options set Reflection/refraction environment override to your original background color/mapping.

psd-manager Photoshop export + Adding a new Background:
3 ways to do it:

• In the psd-manager UI you deselect the Premultiplied Alpha option, render and then you just put the Render layer on top of your background of choice.
• If you prefer to render with Premultiplied Alpha set to on. Set the psd-manager rendered image output to "Layer + Transparency" and in Photoshop use the Layer > Matting > Remove Black Matting command on the Render layer. Then put the new background as a layer below the Render layer.
• A fancy way to do the last step is to set psd-manager to output the rendering as "Layer + Alpha" and Premultiplied Alpha on. In Photoshop now put your new background layer on top (yes on top) of your render layer and set it to "Linear Dodge" and then apply the inverted Alpha Channel as layer mask to it.

Best Regards,
Daniel

[windrockwater]

Daniel,
Thanks alot. That is definitely the issue. I have been using a Vray Sky in the environment map slot. Then in photoshop i would place a backplate behind my render. As long as the backplate was mostly blue sky then the antialias was not that bad. But sometimes the antialias would be very noticeable against a dark background. I just did a test by rendering the same scene with the Vray Sky map removed (black background) and the problem disappeared! Hooray.

Now i have to compensate my overall lighting since I am losing the positive lighting effects of the Vray Sky in the slot....

[super gnu]

put the vray sky in the vray/environment override slots... put it in both the gi override and reflection override, then you can render against black but keep the sky's effects on the scene.

[windrockwater]

put the vray sky in the vray/environment override slots... put it in both the gi override and reflection override, then you can render against black but keep the sky's effects on the scene.

Awesome! Thanks. All

[windrockwater]

Thanks again, problem seems solved. Great forum. See below for the non-antialias trees latest render. No problem now. Happy happy joy joy.

[simmsimaging]

put the vray sky in the vray/environment override slots... put it in both the gi override and reflection override, then you can render against black but keep the sky's effects on the scene.

out of curiosity: could you just put the Vraysky in the tex slot of a Vraylight, then toggle it to invisible to get the same effect? I don't use the environment much at all, so more curious than anything, but I was under the impression the sampling was better/faster with the Vraylight rather than the environment - but maybe it's all the same...

tks /b

[super gnu]

i could be wrong, since i dont use the domelight too much (when doing flythoughs, a really detailed imap / fast rendering frames beats a quick imap and slow rendering frames imho), but i think if you had it set to invisible, youd still need to have it in the reflection override slot for it to show up in refs.