AA filter or no filter for animation?

[Companioncube]

i was wondering what people's opinion are about using AA filtering, or not, when rendering for animation.

What is better for compositing? i've heard before that it's better not to use filters if the renders will be used for compositing.

On the other hand i know that a lot of people like to use soft filters (area, quadratic, cubic, video, soften) as it helps with moiré and fine details, but also adds some time to the render.

[Morne]

Quadratic works great for me

[Crush]

I tend to use a softening filter for animations. It adds more render time, but in my experience the filtering is needed to stabilize fine texture detail and geometric detail. I'll usually start with Area filter and switch to Quadratic filter if necessary.

[^Lele^]

Filtering the rendered screen is useful in cases where there is thin geometric detail (thin texture detail ought to be filtered in texture space, not in screen space, at render time) which causes moire patterns, all the more if animation makes those moire patterns shimmer across frames, where blurring also provides for temporal stabilisation (on top of the spatial one for single frames).
However, filtering (let's consider the wholly positive ones, not the sharpening ones with negative components) blurs the image AND the alpha edges.
The bigger the filter kernel, the more the blur.
This may not be a problem at all if the render doesn't need integration with other Cg or with a live plate, but will create a LOT more work for compositors, where that's needed.
There is always going to be a slight, often half-pixel in size, gap between objects rendered in different passes (one being a matte to the other, and then the opposite), however the blurring properties of image filtering will exacerbate the issue, and broaden the gap.
If using nuke, unpremultiplication, and merging with a Matte operation will solve most of the RGB issues, with or without filtering to the image, but it will still leave the alpha to be patched up (dilation/erosion. curse words for compers...) after the merging is done.

It's a simple test to do, if you have the tools (nuke, chiefly, or AE, although I haven't used that since version 5...).
If you do not, then likely you don't need to worry about alpha filtering issues at all...

[joconnell]

What I've found is that using a filter that's too heavy like soften, video on the soft side or mitchell and catmull on the sharp side is that you've to be careful that it doesn't change the image and specifically edge details that you make it inflexible to work with afterwards. Really sharp edges are normally done by brightening the bright values and darkening the dark values where edges meet - it's like having a localized contrast colour correction done just at the areas where you've got any edges and sometimes it can bit a bit too harsh to remove afterwards. Likewise the softening filters are like a localized blur on your edges so they'll soften definition. Totally fine if that's what you want, but you can't unblur a render afterwards if you've used too much.

The main thing to make your decision on is the content of the animation itself. If you've got really small details like wire fences, hair or the such then you might end up with details that are smaller than a pixel wide. All of the aliasing errors happen in animation when you've got slow movements and these really fine details end up appearing and disappearing in and out of the lines of pixels in your render - ultimately it's a limitation of the pixels we have in a broadcast render to be able to capture such small detail.

You've a few options as to how you go about fixing this stuff. One is to really whack up your anti aliasing and then drop it's noise threshold in the aa sampler section. This'll force Vray to work to a very very high standard for the lot of your render so it's a pretty brute force approach. The second and probably preferable option on moving stuff is motion blur. Normally it's just a case of setting your motion blur duration to 0.5 and using a little bit more AA to clear it up. Post motion blur can be a handy one for this too - similar to the issue with using a really obvious anti aliasing filter, if you use the wrong amount of motion blur then it's burnt into your image. If it's too much and makes the image smeary, or if it's too little and you still get some shimmering in your aa then you've taken the hit for your full render. If you use no motionblur, put out a velocity pass and then use vector blur in 2d, then you can vary the amount to something that looks good, hopefully the right amount to solve pixel creep / shimmer and you're only taking the render hit for some 2d images and not your full render. Post motion blur is bad for anything that rotates / blurs in an arc though such as a wheel or rotor. A final option if you've got a really slow moving camera or objects (which means very little motion blur, thus not solving the issue) is to blur the entire lot a little bit or use some sort of smart blur that only hits your edges (kind of like what aa is doing). It's not ideal since it's defocussing the entire image but again you've got control over the amount and only take a 2d rendering hit, not a big 3d hit.

Typically I'm rendering stuff for VFX and integrating into live action plates. I'll render using either no filter or area. The 3d always comes out sharper than whatever footage it's being comped into so it always ends up getting softened and has grain added which'll degrade it further. It's kind of hard to choose what's better as if you've a case where you don't have thin lines or small details then there's no point in softening the image for the sake of it. You've got to do a few tests on these things, ideally with simpler materials (even turn off reflections / refrations, maps and so on) to see if you can spot any shimmering issues and then figure out what'll solve the issue best.

Have you got any shots you're working on now that need looking at?

[jasonhuang1115]

Thank you John. Really informative post and I appreciate it.

[Companioncube]

thank you Lele and john for the in depth answers.

I'll have to experiment a bit.