Stairstepping with unclamped images

[tricky]

We are increasingly working with unclamped open exr renders out of vray. The problem we find is that bright highlights on objects exhibit 'stairstepping'. What is the way to eliminate this?

Rendered images are being comped in PS CS5.

[vlado]

One way around it is to use a different antialiasing filter, or glow/glare to bleed the result into adjacent pixels. The other way is to clamp at a value like 5 or 6, which still gets you some high dynamic range, but helps with AA as well.

Best regards,
Vlado

[tricky]

Thanks Vlado. Will try the clamp at 5 or 6.

[nlo]

I started working with unclamped colors, but I have great problems with "hot pixels" in reflections.
Mostly those a just single pixels with a high intensity (from 5 to 80) and low intensity neighbors (less than 1) around.
Glares/Glows would work for still images but since those hot pixels have a tendency to wildly jump around, the resulting animation looks very wrong...
The only solution I found, was to clamp at 1, but then whats the point in using floating point if all the dynamic range is cut off. Very high DMC settings like 100 did not help to fix the problem either.
Any suggestion would be appreciated.

Kind regards.

[super gnu]

i find the "blend" aa filter at a very low blend value (.02 or something) helps. you get a little glow around any bright highlites that can hide stairstepping. its not a real fix though. as Vlado said, clamping output to 5ish (or as high as you can get away with) is another way to help, and often the only real fix, however i often find turning on "sub-pixel mapping" can eliminate single hot pixels, as can using the adaptive subdivision AA system, but that comes with its own penalties

[nlo]

thanks for the info. i did turn on subpixelmapping and made the sun invisible. that reduces hot pixels. clamping at 5 or lower also reduces the problem, but those bad pixels are still "hot". and the problem gets more obvious when i apply a dof in post. for example the reflecting leafs of a tree have then little sparkles jumping from frame to frame
i usually don't use any aa filtering but i will try your suggestions.

to be honest i don't want to try to go back to adaptive subdivision. i always had problems with small details...

in imagination the problem is a result of diffuse reflection rays cast into the scene and some of the random rays directly hit the sun and aa is just not capable of making the right judgement to tune down the resulting energy for those specific pixels. but i am not sure if i understand the problem correctly.

[instinct]

In general subpixelmapping does what you described in your last paragraph. It is applying the AA filter to individual subpixel samples rather then the final result so to say. If you still end up with hot pixels i'd say you have too few samples resulting in that "one odd ray hitting a bright spot" getting a rather big weight within the pixel.

On a second note the negative neighbors of hot pixels are caused by sharpening filters. Sharpening filters have negative areas along the filter curve, and with values over 1, these multiply your value to go way below 0. Sharpening filters should be avoided when rendering unclamped. (In general i'd say it is a better idea to add sharpening in post if you really have to or want to)

As for stair-stepping, this is not an issue of not antialiasing properly. VRay is properly antialiasing. But within the standard range of 0-1, the pixel is still way over 1 (even tho blended correctly), hence percieved as a hard edge rather than antialiased. This would be present in real live too, if it wasn't for the low dynamic range of real sensors and all the side effects (both digitally for digital sensors and chemical for film) hiding things like that. Burning out, blooming and other "artefacts" or plain soft clamping caused anywhere along the way.

Kind regards,
Thorsten