How To Render Ground Shadow Only

[leejk]

Hi,

I need to render out the ground shadow only in a scene made up primarily of XREF scenes. I am familiar with the matte option in VRay properties and how to make that invisible in the render while still catching the shadows. Normally I would make the other objects invisible to the camera, but in this case, it is not really practical because of the many scenes and will be using them in multiple renders that require them to be visible. Is there a trick to achieve this?

thx

[hermit.crab]

Doesn't simply setting the ground plane as a Shadow Catcher work?

[leejk]

Hi, yes that does work... my problem though is having only the shadow in output and not the objects casting the shadows. I have since figured out that I can set the layer properties of the XREF scenes to not be visible to camera; before I just was not setting the property correctly.

So now I think I have that part figured out but not able to get the shadow pass written out to disk correctly. My scene environment background color is black, and so when VRay writes out the VRayShadows render channel to disk, all I get is a solid black image. I want to use this shadow pass in a composition in After Effects, but a little unsure of what this should look like. Any pointers?

[hermit.crab]

Hi, yes that does work... my problem though is having only the shadow in output and not the objects casting the shadows. I have since figured out that I can set the layer properties of the XREF scenes to not be visible to camera; before I just was not setting the property correctly.

So now I think I have that part figured out but not able to get the shadow pass written out to disk correctly. My scene environment background color is black, and so when VRay writes out the VRayShadows render channel to disk, all I get is a solid black image. I want to use this shadow pass in a composition in After Effects, but a little unsure of what this should look like. Any pointers?

The VRayShadows are equal to the VRayRawShadows multiplied by the VRayDiffuseFilter. Since the VRayDiffuseFilter ignores shadow catchers, the result will be 0 (black). Use the VRayRawShadows element instead OR for best results - the Alpha channel.

[leejk]

Hi Aleksandar,
That's good to know. I made some progress... I just used render elements inside the EXR file vs. trying to write them out separately. Seems to give better expected results. However, all of the shadow related elements give me shadows of the entire scene; I just need ground shadow. Now I have tried using the multimatte element to mask out the objects from the ground shadow in composition, and that works, but I still have the light shadow problem, and it looks wrong when used in the composition.

First image: Normal render from VFB with vehicle and environment in same scene
Second image: Composited shadow onto a separate image of the scene that was rendered without the vehicle
Third image: The shadow as it appears in the alpha

[hermit.crab]

Do you get better results if the grass is hidden? Simply rendering the shadow as if it's on a plane.

[leejk]

Yes I do, but then the shadows are incorrect, as they appear sharp and not feathered as the grass creates.

[Joelaff]

Is your grass object part of the shadow catching matte object? In other words, does the grass have the same matte object settings as the ground plane? (Could be in VRayProperties, or could be a VRayMtlWrapper).

We render shadows as a separate render pass with all the other objects set to invisible to camera and the shadow catching object set to be a shadow catching alpha -1 matte object usually using VRayMtlWrapper. Then you use the alpha of this render as your shadow mask. If you need the color info (like from shadows from an HDRI or other colored lights) then you can render this pass as Deep using the raw (EXR) output from the VFB and you will get the RGB colored "alpha", which can be subtracted in comp. But we rarely do this, as most shadows are done using color correction through a shadow matte in comp, and don't take their colors from the render engine.

I do really wish we could get this RGB colored "alpha" without having to enable Deep. That seems silly.

AFAIK, there is no way to get proper/clean ground shadows just in a render element with your main render, as they will have antialiased edges where the objects are overlapping the shadows. You could try to compensate for this in comp with dilation, etc., but I would rather have it clean. I would absolutely LOVE to be proven wrong here and shown a way to get a nice clean shadow pass in the main render (ideally without the overhead of DEEP)

[leejk]

Is your grass object part of the shadow catching matte object? In other words, does the grass have the same matte object settings as the ground plane? (Could be in VRayProperties, or could be a VRayMtlWrapper).

Correct, I am using VRay properties to do this. The grass is a Forest Pack object, so it set to a matte object, and the underlying terrain mesh is also set as a matte object. I have tried it both ways, with objects visible to camera or not.

I rendered out a VRayRAWShadows pass and saved it as a TGA, but it here is where I am getting stumped on how to use it in AE. It seems backwards, white where black should be. What am I missing?

[Joelaff]

So with what you have, what if you disable shadow CASTING from the forest pack object so it does not cast shadows on itself. (I really wish VRay had the option to disable SELF-shadows only. I was going to look to see if LPE could do this...) YOu only want shadows cast by the vehicle, not by the grass (assuming your BG plate has its own grass, yes??)

You want the ground and the forest pack visible to camera. The other shadow casting objects are invisible to camera. Disable shadows casting for the forest pack. Leave shadow casting on for the ground if you have a dome light that goes below the horizon. (Shouldn't matter much either way.)

Then load that image into AE and use it as a track matte for an adjustment layer upon which you add curves or whatever your color correction of choice is (levels, etc). You are then using the luminance of that image to act as a mask driving a color correction to darken your BG plate.

Hope that helps.

EDIT: Oh, I see you are using VRayShadow there. That does work if you use the VRayProperties method. Not sure if works if you use the VRayMtlWrapper option to make the mattes.

[ruby_vandeursen]

Not sure if it helps, I use Matteshadow and level it out in PS. With a mask of the object you can simply drop shadow on the object.

[leejk]

Thanks everyone, but let me ask a basic question, "How do I save out a shadow pass from VRay and use it in AE"?

I understand how to setup the scene to generate the shadow only, but it seems like there's a million ways to export the shadow from VRay and composite them in AE, but so far none of them are working for me.

For example, the attached is what VRayRAWShadows produces in the VFB. So far in AE I either get a too light of a dark shadow or it's white. What do I with that now?

Or should I really be using just VRayShadows? However with that I also get too light of a shadow. For some reason the shadow is always lighter than it would be in the beauty pass.

[Joelaff]

Are you familiar with Track Mattes in AE? You can use what you have as a track matte to mask an Adjustment Layer. The Adjustment layer would then have a levels or curves (or multiple effects) to darken the BG image to match the other shadows in the scene.


Track matte info here:
https://www.youtube.com/watch?v=pRP86xCworA
https://www.youtube.com/watch?v=rvAX...bnQgbGF5ZXI%3D

In the image below you would put your rendered shadow in the middle layer (blue) and the adjustment layer (red) is on top. The bottom layer would be your BG image sequence (source footage you are comping into).

You then set your middle layer to be the Track Matte for the top adjustment layer. First click the Toggle Switches/Modes button (bottom green). Then choose the middle layer from the Track Matte popup (underlined green). If your shadow data is in the RGB (luminance) rather than in the alpha channel of your shadow render the click the little square to toggle to Luma matte (green arrow). If you need to invert your matte click the box to the right of that.

Note that when you set the Track Matte it will hide (the eyeball on the left) the layer you are using as the track matte. This is what you want. Don't turn it back on.

Then you have something like a levels or curves effect applied to the adjustment layer which darkens your image to match the shadows to the shadows in the BG plate (Bottom layer). Because you have a Track Matte setup the adjustment layer will only affect the unmasked portions of your BG sequence (where the shadows are). So you are darkening the pixels in the BG sequence that are where the shadow matte is.

Hope this helps.



Note that I would still disable shadow casting for your grass object because it appears to be receiving shadows from itself.

To make the shadow pass I usually do not use any Render Elements. Instead I use a Scene State like I described above where I have only the shadow catching objects visible to camera (grass and ground plane). The shadow CASTING objects are set to be invisible to camera (usually via layers, but can be by object). This gives you the shadow in the ALPHA and a black RGB. In this case you would set your Track Matte to use the Alpha, not the Luma(nance).

[leejk]

Hi. Yes, familiar with track mattes and think I have tried what you are explaining, the difference being in that my background image has no shadows in it. I am wanting to composite the background image sequence + vehicle shadow sequence + vehicle. When I try and adjust the shadow darkness either by the method you suggest or simply adjusting the shadow layer itself, it doesn't look good when compared to the beauty pass I posted earlier in this thread. That shadow layer has the shadows in the RGB layer (although the background colors is also black, so they're not visible in the RGB layer), it has them in a VRayShadows channel, and in the alpha channel. The shadow pass channels VRay creates are inverted, so I am wondering if that is the reason for this, even though I have tried inverting them?

Also, I am rendering my shadows the way you suggest, but is your env background also black? Mine is, and there is no shadow in the RGB channel but only the alpha.

[Joelaff]

It looks like with ForestPack you can get a nice shadow that is closer to you want like this:

Set the the FP object and the source objects that FP is cloning to NOT cast shadows. But do NOT set them to be Matte objects. Their VRayProperties should be normal.

The ground plane can be a matte object or not. Doesn't matter (might render faster if it is).

Set your shadow CASTING objects to invisible to camera.

Then add a VRayMatteShadow element. This will give you the shadow you really want. As ruby_vandeursen said above you may have to apply levels or a curve to it in post to get the correct densities. So you would put an effect like levels on that layer and reduce the gamma to taste, which would end up applied before its luminance is used a the track matte.

This method give you this, which you can see is white where the shadow should be, both on the ground plane and the grass. Note that when viewing this render element in the VFB if will have the Display correction applied to it and appear much brighter (2.2 gamma). You can toggle display correction on and off in the VFB layer stack to see it correctly.



Here is the image this is for:


Hope this helps. As you can see here with FP there are often times where you have to experiment to find a workaround and get the actual output you need. There may be a better way to do this still. The FP thing complicates stuff. Often you will just render the shadow onto a ground plane and then apply a displacement to the shadow in post based on the luminance of the BG image (or based on fractal noise). This gets you the rough edges on the shadow. This method works well because it distorts to the actual BG plate, where as the FP method has the shadows distort based on the FP grass, which is unlikely to match your live action grass. Note that I am assuming your live action BG plate has grass in it, and you just want to comp an object onto the grass-- not that you are adding CG grass.