Render elements using matte objects

[paraganek]

We are trying to use render elements in an animation composite with matte objects (alpha cont. -1, shadows on), and are having problems because shadows are being rendered in the alpha channel (or the main channels) of all our elements including VRay Diffuse Filter, RAW GI, VRay Reflection, etc.

This is an example of the Diffuse Filter Element getting shadows on a matte object:



If we render without matte objects or turn off shadows for the lights, the elements work correctly, like so:




Here is the composite layout we're using:



Is this a known bug and is there a workaround or fix for it?

We're using 3ds Max 8 SP3 and VRay 1.5RC2. We haven't upgraded to RC3 because of the shadow bug.

Thanks!

[vlado]

One thing you can do is to ignore the alpha of the channels, since it makes little sense for them anyways - the channels are designed to be added or multiplied as RGB colors, and not alpha-composited (but you can use the alpha for the final image, of course).

Best regards,
Vlado

[paraganek]

Thank you Vlado, that works just fine.
The only problem is when I use VrayAlpha render element to assign the alpha to the final image in post (assembled with the other elements with no Alpha) I get black outlines because (I guess) of the premultiplication against black background in render. I am rendering all the elements into PNGs right now. Should I maybe be using TGAs and disable premultiplied alpha for the VrayAlpha render element or what is the right way to do it ?



Thanks

Daniel

[vlado]

The way I would do it, is to multiply the background with the inverse of alpha element as pure RGB colors without alpha, then simply add the composite to that (again as pure RGB without alpha). Going with the premultiplied thing would be just a headache in my opinion.

Best regards,
Vlado

[richard_rosenman]

Hi paraganek;

I never use render elements. I do each layer seperately and I have never run into problems such as this.

If I need to render a character, I usually make the ground a matte object (-1 in the VRay object settings), and it renders the character without the shadow in the alpha and with the correct GI colution affecting it's feet.

I then render the BG without the char in the scene.

And finally, I make the character "not visible to camera", turn the ground back to a non-matte object, add a white VRay material and render it out for the shadow. Since the char is not visible to camera, it will cast it's shadow on the ground but not render.

You then comp the three layers together, using multiply for the shadow RGB layer.

Sometimes you need a shadow on something more complex than a simple ground plane. If this is the case then you need to derive the shadow from the BG. You render the same pass as above, and then remove the character and re-render the floor plane. In the comp, you do a difference comp with the two shadow layers and you are left with just the shadow. (Since one layer has the bg and shadow, and the other has just the bg, a difference of the two will result in just the shadow).

There are simpler ways than this to derive a shadow pass but this is guaranteed to work in every single scenario. Difference passes are extremely powerful and I often use them for a lot more than just shadows - ie. deriving a radiance pass, ambient occ pass, etc.

Hope this helps. Oh, we always use TGA with premultiplied alphas and comp in either Combustion or AE.

-Richard

[paraganek]

Hi Richard,

thank you very much, a lot of interesting info for me.
Now knowing real professionals do not use Render elements that much, I am thinking about abandoning that approach as well.

Is there anyone else on the forum who uses vray render elements (especially vraydiffusefilter, vrayrawGI, vrayrawshadow, vrayspecular) combined with with many matte scenes and different passes for moving objects (QMC GI)for compositing pipeline in production on every day basis ?

[paraganek]

Hi Vlado,

I am unable to get rid of the black halos around my render layers. You
mention multiplying the inverse of the alpha onto the background layer.
Right now I'm doing that to put the shadows in. How in this composite
would I rid myself of the black rings?

When the shadow layer is turned off, the black only appears on the
bottom of the cones. Is this a separate issue?

All the layers are composited using color only, and "Set matte" is using
the value of the "cones_up_vrayalpha" which has no shadows so it's white
only where the cones are.



Thanks!

[panthon]

Hi!

Im afraid im not able to spot the problem from the distance. It looks to me that layer order and blending modes are all correct, so we would be dealing with a problem related to how the alpha is processed in combustion. It looks like there's a divergence between the content of the layer (the cone) and the alpha for that layer (the cone mask), thus creating this black gap. If you change your background color in combustion, does the border color change? If so, the problem could be in the alpha.

You could try to use the cone's layer own alpha instead of the set matte (and dump the vrayalpha element) and premultiply the footage in combustion options, with premultiply with black ON (providing that u rendered in max against black background).

It´s important to choose wisely the output format from max. I always use tga with premultiply option ticked so that it gives me no problems later in combustion.

Hope it helps!