Question about VrayToonMtl shadow colors

[AEBI_VINCENT]

Happy Friday

I'm familiar with VrayToonMtl and scene setup, but some things are unclear. The main questions I have are:

-which shadow mode in the directional light do I use? (Adv. Ray Traced, Area Shadows, Shadow Map, Ray Traced Shadows, Vray Shadow or XShadow)

I picked VrayShadow, because I figured since I'm using Vray...you know. But maybe that's wrong?
Have a look at this simple scene:


What I'd like to achieve is the same 100% direct and self-shadow color over all objects in the scene. I have already done that by setting the same blue-ish color in the diffuse-ramp inside the material, which results in the blue shadows. But as you can see in the first screenshot, the blue color mixes with the diffuse color of the individual objects.

So what I did now is setting the diffuse color of every single material to white, which then resulted in the desired look:


So far so good, but the changing of the diffuse ramp and diffuse color on every single mat could become tedious in a huge scene. Also, this only worked because the diffuse color of the VrayToonMtl is set to white. If I load a bitmap in there, the shadows would again mix with the bitmap. Isn't there a "global" setting for shadow color? I thought I could just change shadow color or light color in the directional light settings, but that does nothing. In the docs, I read about shadow control in the VrayToonMtl, but that too does nothing when changed. What is shadow control supposed to do?

Also: do you know if there is any way to achieve black outlines/edges via VrayToonMtl? Or is that only achievable by using the VrayToon-effect in the environment slot? Were you ever able to achieve a natural, hand drawn looking, "ink"-resembling outline or do you know of any downloadable presets?


thank you

[hermit.crab]

Isn't there a "global" setting for shadow color? I thought I could just change shadow color or light color in the directional light settings, but that does nothing. In the docs, I read about shadow control in the VrayToonMtl, but that too does nothing when changed. What is shadow control supposed to do?

The Shadow Blend / Shadow Opacity options are what you're looking for. However, note that they are not supported with Standard lights (internal bug-tracker id: VMAX-12564). I'll add a bump to the issue on your behalf. You can workaround it by using Directional V-Ray Lights.

Also: do you know if there is any way to achieve black outlines/edges via VrayToonMtl? Or is that only achievable by using the VrayToon-effect in the environment slot?

In 3ds Max, it's possible only with the VRayToon effect.

[AEBI_VINCENT]

Hey, thank you for your reply. The use of a directional Vray-Light works with the blended shadow, but results in a jagged edge of the blended shadow that doesn't align with the shading that's caused by the diffuse ramp (position 0.5).

The jagged edge is caused by the VrayLight, it can't be fixed by smoothing faces, resetting normals or turbosmoothing the mesh, although turbosmooth will change the jagged edge, but it still won't align with the diffuse ramp. The directional light value is set to 1.0. Can this be solved?

[hermit.crab]

Hey, thank you for your reply. The use of a directional Vray-Light works with the blended shadow, but results in a jagged edge of the blended shadow that doesn't align with the shading that's caused by the diffuse ramp (position 0.5).

The jagged edge is caused by the VrayLight, it can't be fixed by smoothing faces, resetting normals or turbosmoothing the mesh, although turbosmooth will change the jagged edge, but it still won't align with the diffuse ramp. The directional light value is set to 1.0. Can this be solved?

The jagged edges can be fixed if you subdivide the mesh as you've already found out.

Regarding the alignment - shadow control works with Raw Shadows so it won't automatically sync with the diffuse color - it just gets blended with it. For best results, I suggest using a separate VRayToonMtl for the ground and using Shadow Control only there to blend the ground shadows (and not the unaligned self-shadows).

[AEBI_VINCENT]

The jagged edges can be fixed if you subdivide the mesh as you've already found out.

I'm afraid it's not fixed. The teapot is now subdidived by 64 and has over 130k polygons. It's still jagged. The sphere now has 20k polygons and is still jagged. With these results, I can't think of any scenario where I would want to use blending shadows. Also, if using ForestPack for example, I would have to go into every single vegetation mesh and increase polycount, to get rid of the jaggies.

Regarding the alignment - shadow control works with Raw Shadows so it won't automatically sync with the diffuse color - it just gets blended with it. For best results, I suggest using a separate VRayToonMtl for the ground and using Shadow Control only there to blend the ground shadows (and not the unaligned self-shadows).

So getting rid of shadow blending on the objects, which defeats the purpose of activating blending shadows entirely. Because If I can only use it for the ground, I'd still have to set up a diffuse ramp for hundreds of other ToonMtls, instead of setting up the same shadow blend over all materials, which would be easier/faster.

This seems like a dead end, since there seems to be no option for a global shadow setting concerning all ToonMtls in the scene.

[hermit.crab]

I'm afraid it's not fixed. The teapot is now subdidived by 64 and has over 130k polygons. It's still jagged. The sphere now has 20k polygons and is still jagged. With these results, I can't think of any scenario where I would want to use blending shadows. Also, if using ForestPack for example, I would have to go into every single vegetation mesh and increase polycount, to get rid of the jaggies.

Sorry to say, the jaggedness is not that easy to solve dev-wise.

This seems like a dead end, since there seems to be no option for a global shadow setting concerning all ToonMtls in the scene.

Why don't you use a single VRayToonMtl, where the DiffuseRamp is used (black and white values) to map outthe Diffuse Color? You plug a VRayMultiSubTex into the Diffuse and control each color based on an f.e. Object ID. Check the attached scene. The shadows on the ground are controlled separately via the ground VRayToonMtl.

[AEBI_VINCENT]

Why don't you use a single VRayToonMtl, where the DiffuseRamp is used (black and white values) to map outthe Diffuse Color? You plug a VRayMultiSubTex into the Diffuse and control each color based on an f.e. Object ID. Check the attached scene. The shadows on the ground are controlled separately via the ground VRayToonMtl.

That won't work, because if you plug any color into the diffuse slot, that color will mix with the color defined in the diffuse ramp. The difffuse has to stay white if I want all shadows to be the same color, regardless of the color of the object.



I guess I just have to do it like it did in my first post. Diffuse must be white, shadow color is defined in every separate ToonMtl via diffuse ramp. Object color is then defined by diffuse ramp, not by diffuse slot.
I'm starting to wonder why the diffuse ramp color has to multiply with the diffuse color. There should be an option to turn that off.

[hermit.crab]

I see. You can still use one material for all, where all the Diffuse Ramp keys are controlled by VRayMultiSubTexture/s (except the first, which will be the shadow for all). Check the attached scene and result. In this case, I used Color Correction nodes to make the primary color lighter/darker for some of the keys.