I am aware I can “remove” jaggy edges in vray with clamping to 1 and enabling subpixel-mapping. But I obviously lose HDR information there.
Is there a way in photoshop to import a unclamped exr (with the jaggy edges) and then basically selectively clamping it in photoshop and mapping it in a way that I lose the jaggy edges and they become smooth.
Obviously it would have to be selective since I maybe want to paint in some more light here and there with the HDR information but at the end mapping it in a way that the image stays the same look but smooths out the jaggy edges.
I don’t think “jaggy edges” have anything to do with “clamping” and “subpixel mapping”. The “jaggy edges” come from the AA; you might want a filter like “area”. I might be wrong.
They come from extreme differences in lighting from one pixel to another. No amount of AA samples I tried got rid of the issue (even 1/100 with 0.001).
Yes, it’s the high values, not AA. You could unpremultiply, then clamp, then premultiply, then mix selectively. In PS you might be able to divide the beauty/rgb layer by the alpha as a layer on top, then clamp, then multiply by the alpha again.
Ah you may have misunderstood me. while alpha may be a problem at the moment I am talking about interior edges. Say a nice high gloss material which has a sharp highlight along somewhere and the rest is significantly darker.
Ah I see. One option is to render it clamped and then mix. Another is to put a glow/glare on top of it so the jagged edge won’t be seen, this would be the most realistic.
Edit: You could add in a 1-5px blur with lowered exposure, so it will soften up the edges of the overbright areas only.
Edit2: Worth testing could be rendering with image filtering off, then blurring in PS. Might not be the best for your image quality though.
one trick i used sometimes in the past, is to use the max “blend” aa filter, on an extremely low value.. say .03 or something (youd need to play)
since this combines a hard and soft filter together in the image. on normal values it looks all smooshy (good for underwater shots) but on *very* low values you only really see the hard filter - apart from when the aa is swamped by a bright highlight.. in this case the very faint soft filter becomes much more visible.. if you get the values right, it will give you a soft highlight on your bright bits without clamping.
If you are using V-Ray 3, you can use Max ray intensity instead Clamp output.
This option allows the user to suppress the contribution of very bright rays, but the Max ray intensity is applied to all secondary (GI/reflection/refraction) rays, as opposed to the final image samples.
This will keep your dynamic range of the HDRI.
Would it be possible to send us your scene, or the problematic part of it on support@chaosgroup.com?
Raising the minimum samples to 2 or even more like 4 should help. Also using a softer image sampling sister like soften (which is Gaussian) at a radius of about 4.2 will help a lot as well.
BTW, I just had this issue in a render of a bright chrome object, and using soften with a radius of 3.98 or 4.2 or so (3.9 should be a hint faster) does indeed smooth out these type of jagged highlights. It is hard to get a smooth edge when using a rectangular image sampler. Soften also just looks so much more photographic. I hate the area sampler, but it is a little faster so I end up using it sometimes.
I can’t wrap my head around why you would want to have 32bit range but not deal with jaggies by tone mapping in post. Glare/bloom would remove these problems on a normal camera.
