I am just starting to learn to composite render elements using Vray and Photoshop CS5. In the attached testexr_comp file you can see a white fringe around the square and the cone. In the testexr_beauty pass, I do not see it. I tested this using 32 bit and 16 bit exr. I also see it when using PSD Manager and saving either 16 bit or 32 bit PSD files. Turning on or off pre-multiplied alpha seems to have no effect. I have also searched the web for a solution but have found nothing. What may I possibly be doing wrong?
In Max I have Gamma/LUT enabled. Set gamma to 2.2, input gamma 1.0, output gamma 2.2. Checked Affect Color Editors and Material Editors. In Vray Color Mapping I have linear multiply, Dark and Bright multiplier gamma of 1.0, Gamma of 2.2, checked Affect Background and Don’t affect colors. I also see the white fringe with antialiasing turned off; however, it does seem a bit weaker when antialiasing is turned off.
By any chance did you try rendering to any format beside exr? I recall having a problem like that a while back and it was something specific to exr. If nothing else seems to be working you might want to try .hdr just to see if that helps.
You can’t just Mulitply these together. Multiplying anti aliased passes together will always cause this sort of problem. A lot of times this can be corrected by dividing the “reflection” buffer by the “raw reflection” buffer to create a new (but slightly different) reflection filter buffer. Then you can make alterations to the raw reflections and mult that by your new reflection filter buffer.
This sort of math is really best handled by compositing software like Nuke and always done in floating point. In fact I don’t think its possible to do this in Photoshop. (someone correct me if I’m wrong).
Here is an example of a similar problem but with raw light passes and the diffuse buffer getting multiplied together. Your problem is the same thing but with reflections. This problem can be fixed using the dividing method described above.
