here is what i want
a: my final composite needs to look natural like this.
b: lets say i filmed this scene with a film or video or photo camera.
c: now lets say i reconstructed that scene in 3d then made those objects matte objects. and rendered a 3d character into the scene
with alpha channel
d: this is the crap i get. whyyyyyyyyyy!!!
http://www.elfpro3d.com/Vray/e.gif
one frame is what i get contrasted with what it should be.
The simplest way imo would be to have a separate pass for the shadow, and mask out the shadowed area in the comp.
Of course it could be impractically long, tedious or complex, depending on the shot type.
well it seems the simple way (the one i used) wont work, so i guess its got to be more complex then
don’t render the cast shadow into the alpha channel of the same pass as the 3d object. as lele says you need to render the shadow pass spearately, then it’s all sorted out in the comp.
Since the alpha has the shadowing burned in, a bit more work is required, you’re right there.
What you have isn’t a shadow pass.
I mocked up a quick comp to achieve the result you are after in Fusion.
Here’s a snapshot of the flow, and of the result:
The apparent flow complexity is due to two things: the wrong passes are provided (so i have to separate body and shadow alphas, and mask out and CC the shadow itself), and my own ignorance of fusion, which might have led to a less than optimal usage of tools options and properties, and to a longer flow than effectively necessary.
ok. so how would i combine the shadow rendered in 3d with the shadow i filmed? remember im only using a full 3d scene as an example. pretend the box and the ground were real footage. its the combination of the real shadow and the characters that confuses me. the matte shadow object should be casting a shadow. basically when rendering the only part i should see is the shadow created by direct light (and if the matte object is blocking direct light then it shouldnt render shadow in that area) and the GI shadow
edit
you posted that image while i was typing… umm. it looks as though your creating a polygonal mask in the compositing program to mask out the shadow of the character. wouldnt that be the reeeally hard way to do it. if ive matched my matte objects perfect with the live action stuff then i should be able to use them to mask out the shadow of the character
edit
i guess there just isnt an easy way. its going to have to be done in multiple passes
You can render out the reconstructed scene without your biped dude but have all of the shadows on for those matte objects (they look like they’re set to not cast shadows now). This should give you a nice matte of the existing shadows in the scene - just subtract it from your shadow matte above. I think this will work for you.
With this in mind, your reconstructed scene is going to have to match up pretty damn exact to avoid any overlap or gaps between the real shadows and the partial 3D shadows. Do you think you’d be able to generate a good luma key from the existing scene shadows?
Rotoscoping and tracking IS the easy way, most times.
Much easier than try to match with pixel perfect accuracy a 3d set and its shadows.
Well, so i was taught in my days as a comper in London.
Maybe they just wanted me to feel the heat 
