Black Fringes in Render Elements Composition

[JulianT_CD]

Hi everyone,

I am trying to recreate the beauty pass of a simple Max scene in Photoshop, using ProEXR 1.9.1 as importer. I rendered the scene against a black background.

Two issues:

1) With both ProEXR import options, I am getting black fringes around my objects. With import option 2, the effect is more pronounced and even the Beauty pass has a fringe which is not the case with import option 1! I can probably get rid of this by using "Defringe" or "Remove Black Matte" in Photoshop's Matting tools, but these do not work reliably in every case. It also slightly interferes with my workflow, since these tools only work on rasterized layers.
So how can I avoid these black fringes? That can't be normal, right? And which import option should I use with ProEXR?





I uploaded both Photoshop files (the one created with ProEXR import option 1 and the one created with import option 2) as well as the Max file:

http://www.filedropper.com/maxandphotoshopfiles

Lastly, in both Photoshop files there is a subfolder called "Unused" which contains lots of other render elements in case you need them to get rid of the fringes.


2) How exactely do I use the "Refraction" render element? To me it seems like I can recreate beauty passes with refractive/transparent objects without even using the Refraction element. Whenever I add the Refraction element to the comp, the objects seen in the refractions get too bright.
I guess what irritates me is that the Refraction element seems to contain all light contributions (diffuse, reflection, specular etc.) of the objects seen in the refractions. However, the Reflection, Specular, Lighting elements etc. also contain the light contributions of objects seen in refractions. Adding the Refraction element on top then of course adds the light contributions twice.
So how am I supposed to work with the Refraction element?

I really appreciate any help! Many thanks!

Julian

[JulianT_CD]

Ah OK 2) seems to be caused by having the refractive material's "Affect channels" option set to "Affect All". Setting it to "Color+Alpha" fixes the issue of double contribution. But isn't setting "Affect All Channels" the correct thing to do for compositing? At least I think I read that somewhere here...

[JulianT_CD]

Anyone? This is really a pain to deal with. I can't imagine I'm the only one getting these fringes...

[Svetlozar Draganov]

The fringes appears because V-Ray sampled the border pixels on black color and later you've changed it to a white one.
If you intent to make sure drastic changes of the background color at compositing stage it is necessary to unpremultiply any render element before applying any operations to it otherwise the original background color which is used in the rendering will be visible at the border pixels like in the image bellow:


Here is the result when all the elements are unpremultiplied before mixing them and replacing the back background with white:


We are not familiar with Phothoshop and its ProEXR plugin and we cannot advice how to apply those operations in Photoshop and what ProEXR import presets to use.

If Photoshop doesn't have the ability to unpremultiply images you could export unpremultiplied images directly from 3dsMax but only as TGA file format and only when Max Frame Buffer is active.


Regarding the Refraction(Reflection) render element, when All Channels option is activated V-Ray stores the Refraction(Reflection) information in Reflection(Refraction) render elements and into other render elements as well. So if All Channels option is activated you should not use Refraction(Reflection) render element in the composition since the information from it is already distributed over the other elements. Please be careful with that setup since if some of the material in the scene use All Channels and other ones use Color Only or Color+Alpha part of the information will be distributed over all elements while another part will be stored only in Refraction(Reflection) element and then when compose them some of the information might be missing or might be applied twice.

[Art48]

Strange....we render against black all the time, and we comp it in front of bright skies with zero problems. The premultiplication-problem only becomes apparent when doing drastic changes to the rendered objects, not the background against it is composited.

[JulianT_CD]

Thanks a lot for everyone's replies! Svetlozar, I skimmed a few other threads here about this issues before posting and I remember that the black fringes were deemed "not normal", at least not with this severity. A few have also said that this doesn't happen with mental ray for instance. Art48 has also suggested that this problem doesn't occur in his projects even when changing the background color drastically.

So despite your logical derivation, I'd kindly ask again if you are 100% sure that this can't be avoided and is not a problem inherent to V-Ray? Could maybe the AA filter (Mitchell, Catmull, Lancosz etc) cause this issue or worsen it?

[Svetlozar Draganov]

Strange....we render against black all the time, and we comp it in front of bright skies with zero problems. The premultiplication-problem only becomes apparent when doing drastic changes to the rendered objects, not the background against it is composited.

There are different compositing approaches and software, can you send me a quick example of the setup you are using including 3D scene, render passes and Nuke composition?

[Svetlozar Draganov]

Thanks a lot for everyone's replies! Svetlozar, I skimmed a few other threads here about this issues before posting and I remember that the black fringes were deemed "not normal", at least not with this severity. A few have also said that this doesn't happen with mental ray for instance. Art48 has also suggested that this problem doesn't occur in his projects even when changing the background color drastically.

So despite your logical derivation, I'd kindly ask again if you are 100% sure that this can't be avoided and is not a problem inherent to V-Ray? Could maybe the AA filter (Mitchell, Catmull, Lancosz etc) cause this issue or worsen it?

According to my experience as a compositor(I am not very good one though) the background color is always embeded into the border pixels and becomes visible at some point unless it is unpremultiplied or using deep-exr setup. I am not familiar with Mental Ray and I don't know if this issue exist there but I would like to see a Mental Ray scene and composition file if possible?

[Art48]

There are different compositing approaches and software, can you send me a quick example of the setup you are using including 3D scene, render passes and Nuke composition?

We comp with PS, it's really just drag and drop the EXR File and select "Transparency" instead of "Alpha channel" on the import options. Max is set up so that environment is black. It's a really simple setup if you really want, I can scrape something together.

[Svetlozar Draganov]

Yeah if possible please send me a scene and explain me the compositing operations.
Meanwhile you could try to comp the scene bellow and replace the background color to white, it contains just two render passes GI and Lighting rendered on black background.
http://ftp.chaosgroup.com/support/scenes/vray.max

Here how the result without looks like here (no unpremultiplication applied)
http://ftp.chaosgroup.com/support/sc...1_09-28-09.png

[Art48]

Oh, I think there was a misunderstanding, I didn't express myself clearly:
We don't to the fancy passes comping, we basically drag+drop the beauty in and only use elements here and there to slightly enhance stuff. Sorry.

[JulianT_CD]

I don't have time to reference the ProEXR documentation at the moment, but from the top of my head: I read that according to the documentation the correct way to comp render elements together in Photoshop is to import with "Layer Transparency" (i.e. don't import with the alpha on a separate layer!), then set the bottommost render element to "Normal" Blending Mode and place all other render elements additively as clipping masks on top of it. In my experience this doesn't completely get rid of the fringes, but makes the effect much less pronounced.

If you were to instead use a "Layer Mask" on each render element (or on the entire group folder of render elements) and copy the Alpha into it, Photoshop applies an sRGB curve to the Alpha before it is used as Layer Mask which in turn makes the fringes more pronounced (if I understood that correctly). This doesn't seem to happen with Clipping Masks.

[Svetlozar Draganov]

Oh, I think there was a misunderstanding, I didn't express myself clearly:
We don't to the fancy passes comping, we basically drag+drop the beauty in and only use elements here and there to slightly enhance stuff. Sorry.

This is a different story then and it is normal to not have that issue with it.
The artifacts arise when blending the components of the Beauty pass, if you just swap the background everything is fine.