Anti-aliasing on overbright light

[RE:FORM_STUDIOS]

This is probably my fault again , but I have found that anti-aliasing between shade and light on a direct light becomes pretty bad when you bump the light multiplier high ( 8 and above)... i've tried different image samplers at high samples,but it seems to make no improvement. Theproblem occurs both with and without any GI.

My current workaround is to give the light area-shadows.

Any ideas?

[joconnell]

You could probably use an area shadow on it to make the transition a little smoother to start off - Have you got a render of a specific area?

[Da_elf]

its something thats been discussed before about the AA on overbright things. it has to do with using the clamping (or something like that)

[re:FORM]

clamping? I don't have clamping on the image.... is there another clamping setting I've missed?

Yep, as I mentioned, using area-shadows is a good workaround.

I'll get an image up soon.

[joconnell]

Oh god i missed that in your post - that's what i get for working the weekends . Here's a great big stupid post which hopefully helps.

Okay the clamping is a major issue - What happens if you don't use clamping?

Vray renders an image to full brightness info so you can have pixels far brighter than rgb 1.0, like in a hdri image. In a small res image like for tv or dvd,you could have a shadow area in the render with an rgb brightness of 0.5 (or 128 / 50% grey - whatever way you want to think of it) sitting beside a really bright area with an rgb brightness of 5.0 which is really blown out. So if the anti-aliaser has to work out a value between the two, it'll be around 2.25 and with a really crisp shadow where the dark and bright meet really suddenly you get quite nasty looking aliasing along the edge.

Clamping pretty much decides to make the brightest point in the final render a value of 1.0 or pure white (rgb 255) because it doesn't really matter in the final image whether something has an rgb value of 1.0 or 100.0 - you can't display anything brighter than 1.0 in a final render anyway. How does this help the anti aliasing? Well if we take our shadow and bright areas example from above, it means that the shadow still stays at rgb 0.5 (the clamping only cuts values above 1.0) but our really bright highlight that was rgb 5.0 will get cut to rgb 1.0 . It will look the exact same in the final image but where the shadow and bright areas meet will be a lot smoother. Again if our anti aliaser has to work out an in between value for where the two meet, it'll now only have to pick a value between 0.5 and 1.0 which will be 0.75 - a much closer value to 0.5 and thus a softer transition and a smoother edge.

Why bother turning off clamping then?

Well, if you have a workflow that can use the extra information of the ultra bright areas then you get good benefits too. Say for example you've rendered an image to unclamped colour and stored it in a format that can save all the overbright info (like exr, hdr or vrays image format) and you start doing heavy colour correction. Take an example where you've rendered something with a lot of very bright highlights and burnt out areas of the image. If you work in floating point and decide to colour correct the brightness of the image to half of its original value, you may find that some of the pixel values that were previously just pure white (anything over 1.0) may now fall under 1.0 after the colour correction and add detail to your image. If you used a clamped colour image instead, you will have thrown this information away, all of your highlights will have cut off at 1.0 and now after a brightness colour correction to make the image half as bright, you'll end up with boring flat areas of grey in the highlights seeing as the clamping has thrown out any values over 1.0. Again this really depends what you're doing (I can't think of a situation where I've ever made something half as bright for example) but it is worth knowing.

So now that you know what the problem is then how do you solve it? There are four ways you can sort the problem.

1. Use a soft antialiaser like area, quadratic or video. This makes your edges less sharp so it means that the shadow and light won't meet as harshly together so you wont have really bright pixels beside dark pixels as often. Catmull rom or mitchell netravelli filters do edge enhancement like the photoshop unsharp mask filter so it'll make the problem edges stand out more.

2. Use area shadows or shadow maps. The edges of these area naturally softer so like above, you get a softer change between shadow and light areas.

3. Use different colour mapping so you don't get quite as burnt out images. Again this will try to stop your image burning out as much so you don't get as many bright white areas. Sometimes though you need the bright white areas.

4. Turn on clamping and use higher quality aa. If you need burnt out areas and a sharp aa filter like catmull rom, use the adaptive subdivision aa with a low threshold.

Here's a litle image that will show you the difference between three different AA filters and what happens with clamping on and off for them. At the bottom is a catmull rom render with adaptive subdivision that has a nice edge aa but is slower than the others. The scene material is medium grey, the skylight is the default blue with a 1.0 multiplier and there's a direct light which is yellow and has a multiplier of 4. You can see that clamping helps in pretty much all cases and the adaptive subdision does a good job too.

[re:FORM]

Thanks for the detailed explanations Jo! That helps explain things alot.

As it happens I am using the catmull rom filter, which explains why the aliasing was so bad.

I am using the linear workflow, so I do have the luxury of outputing to openEXR and then work on the adjustments in the photoshop.

I guess I'll stick to using soft shadows.... they give a nicer result than sharp shadows for the scene I'm working on anyway

btw.... the correct link to your image is http://www.joconnell.com/stuff/filter_clamping.jpg

[edit] ah, you just fixed the link as I was typing [edit]

[joconnell]

Yeah I really like the look of catmull rom - it's nice and crisp and really brings out the detail in your renders but it can cause a lot of problems. If you render to something less harsh like area or mitchell, you can use an unsharp filter mask in photoshop and get pretty much the same look as catmull (it's pretty much what catmull is doing anyway) plus it also renders quicker.