Object effect masking

[Shimakaze]

I'm trying to create masked out (alpha channel) elements of a single object and everything it affects in a scene. What I mean by this is the final masked image should contain the object and all reflections of it in other (hidden) objects as well as it's shadow and any GI it casts on surrounding objects without those objects being visible.
In other words, what I need is basically the same as what is done in this tutorial but with the final result being with transparent background rather then comped back into the original picture with blend modes.

Can anyone think of a way to do this? If you use the matte system in VRay you can do something a bit similar (though not exactly), but the reflections doesn't affect the alpha.

[Shimakaze]

No one knows?

[vlado]

Apply a perfectly white self-illuminated material to that object, and turn off all lights and environment maps (the environment color should be black too). Render this normally with GI and all, and the resulting RGB image should be what you need. It might make things faster to turn on the direct illumination option in VRayLightMtl too.

Best regards,
Vlado

[Shimakaze]

How do you make the matching RGB channel though? I've tried a ton of different ways but the result is always different from the original. The only way I've found that makes a reasonable result is to use the full original render as the RGB channel, but it only works as long as it's composited back into the original picture. Not if it's added to something else.
Is there any way to add this to VRay? Either as an option in the Matte parameters or as a render element?

[vlado]

The RGB channel would be what it says in the tutorial - a render with the object with a black material subtracted from a normal render.

The proper way to do this is to render several passes, like it says in the tutorial. There is no quick way to get these channels in one pass.

Best regards,
Vlado

[Shimakaze]

I still can't can't get this to work.
Let me see if I have understood you right...

1. You start with a normal render 2. A render with a black material 3. a render without the object and 4. A render with all lights turned off and the object itself white.
5. You then subtract the black render from the normal render 6. And use the white render as alpha? But at this point the reflections are too heavy and the shadows aren't there at all.
Using the divide method from the tutorial (inverted) to create the alpha (7) keeps the shadows intact but doesn't keep the reflections very well. Comping it with the result from the "black subtracted from normal" operation as RBG creates very hard shadows ( as well as a border around the object. Using the original image as RGB works better (9), but still looses most of the reflections and only looks good when comped back into the original scene.(10)

I must be missing something here... what am I doing wrong?

[Shimakaze]

bump... any ideas? I could really use a functional method for this...