Streamline backplate integration workflow

[LudvikKoutny]

Hi,

i would like footage/backplate integration workflow in Vray to be simplified, as it currently takes really lot of steps to get it right, and creates a lot of space for error when doing multiple shots on multiple backplates:

Currently, to correctly set up HDRI+Backplate integration, one has to:

1, Create dome light
2, Make it invisible
3, Assign HDRI to it
4, Instance same HDRI to environment slot in the viewport to rotate it correctly so it's rotated same was as on backplate
5, Create ground plane and set its Vray properties to be matte object, also for reflections, and enable catching shadows
6, Remove HDRI from the environment slot and replace it with backplate
7, Create camera and match it to backplate
8, Put backplate bitmap into camera mapping per pixel map, and put that into environment slot (which will of course break display of backplate in viewport)
9, Pick camera you are rendering from as a camera for camera map per pixel map
10, Switch progressive to bucket rendering for final because affect background checkbox does not work with progressive, but backplate should not be colormapped.

Now, the real pain comes with the amount of steps it takes to readjust the entire rig to render different backplate in the same set

1, Swap backplate bitmap for a new one
2, Instance HDRI from DomeLight to environment slot to see it in viewport
3, Rotate it to match new backplate
4, Put backplate into environment slot to see it in viewport
5, Create new camera and match it to new backplate
6, Replace backplate bitmap with camera per pixel map
7, In camera per pixel map, pick new camera for new backplate

This is really a lot of steps... Some time ago I had to do this kind of job in V-Ray and it was not very pleasant experience, unfortunately. I had to do shots on several backplates from several angles. I did a lot of mistakes I had to fix, which does not happen when I do the same task in other renderers, simply because the process is not so tedious. I think most of the annoyance comes from having to pick a camera from "camera map per pixel" map every time camera is changed. It would be just great to be able to complete entire workflow without need to use camera map per pixel. It would reduce a lot of space for error, as well as allow me to see backplate in the viewport the entire time and be in context of a shot I am working on.

So firstly, I would like to ask for a fix of matte object mode, so that screen mapped environment map is truly projected in 3D space onto the matte object. Right now it's somewhat projected for reflections, but for example GI is still looked up in screen space, creating quite wonky results, as seen below:

This is screen mapped backgound without GI, so far so good:


But things go wrong once I enable GI :


Secondly, would it be, pretty please, possible to get affect background checkbox working with progressive? Progressive mode has already matured, so especially now, when it's default, it would be great to have no need to ever switch back to buckets.

[vlado]

Secondly, would it be, pretty please, possible to get affect background checkbox working with progressive? Progressive mode has already matured, so especially now, when it's default, it would be great to have no need to ever switch back to buckets.

But why would you ever render with your backplate actually in the environment slot? For most use cases that I've seen (granted most were in a VFX scenario) the backplate is always added later on in the compositing stage. Besides, that option is a huge hack, I would very much want to get rid of it.

Best regards,
Vlado

[vlado]

The other question is why would you need to use "camera map per pixel" at all? The matte objects do all that by themselves if you use the new "matte background" slot in the Environment section of V-Ray.

Best regards,
Vlado

[LudvikKoutny]

The reasons I like to render it directly are three:

1, WYSIWYG. I don't see my (for example) car sitting in a black darkness. I see it in context with the backplate, I can tweak exposure and highlight compression to match the camera response curve of backplate as well as possible.

2, To this day, no one told me correct workflow of comping the result together having it matching reference beauty pass 1:1. It's quite tricky, as some parts, such as GI, have to be comped on additively, while others (shadows) have to be subtracted. And then there is this weird thing with colored alpha matte objects spit out, that I never found out how to use. No software I know of supports colored alpha. There was a lot of very different approaches scattered both on this forum and internet, but none of them delivered the 1:1 result

3, If I managed to comp it 1:1, then I would do the entire process of composition in post just to get the 1:1 same thing rendered could spit out directly. And I would start to post process my image further from there, but I would get exactly the same thing, just with bit more steps. What additional flexibility would comping it give me? I can't move the object around, because parts of the backplate are baked into GI in alpha:


As for the matte background slot in Vray, I have to admit I never noticed it before. I looked it up in Vray help, but it just says what each button there does, not what is it really for. I assumed backplate goes there, so I've put the backplate in there, but I am getting exactly the same error as shown on images posted above, which occurs when I don't use camera map per pixel:


I am probably doing something wrong... ?

EDIT: Now i remember. It's from that old thread of mine. There was one use case that Vray did not cover at the time - rendering on black background but still having CG objects reflecting some projected backplate. But I still don't understand how would it help with problem of GI sampling being looked up in screen space instead of surface projection.

[LudvikKoutny]

Maybe here's better way to illustrate the issue:

Conditions:

- We have a simple scene, with teapot and torus knot sitting on a plane.
- All 3 meshes have default gray VrayMTL assigned - no reflection
- There is one Dome light in scene, set to invisible and mapped with interior HDRI
- There is checkerboard map as environment background set to Environment Screen mode

This is first picture: GI is enabled and no meshes are set to be matte:


This next picture has GI disabled and plane set to matte object, as well as matte for refl/refr, and catching shadows:

Everything is nice and dandy until...

In this next picture, I have enabled GI:

And as you can see, everything now looks kind of transparent. I believe this is because of GI samples are returned in screen space, not actual 3D space.

Now if we take the scenario above, and just feed the same checkerboard map into environment slot through "Camera Map Per Pixel" map, and physically project it on the surface, we get more or less correct picture:

Now GI seems to be correctly sampled in 3D space.

It seems that matte mode in Vray object properties projects the texture only for reflection and refraction, not for GI.

And I said that the camera mapped result is only "more or less" correct, because there seems to be going something odd with GI. If you look at last picture, there are still visible edges of the projection plane. I can't understand how GI could darken the plane, as environment illumination is completely handled by dome light, so GI should only brighten the ground plane by light bounced off the teapot. It should definitely not occlude it from the environment even more than Dome light does. Alpha looks kid of weird as well:

[kosso_olli]

But why would you ever render with your backplate actually in the environment slot?

Often I have to light cars with photographers or creative directors. They don't want to see the car sitting on a black background, they have to see it in context with the backplate in order to judge lighting, shadows and reflections. Also, the GI problem Recon442 speaks of is existent here as well, often I turn GI of completely for car exterior shots because of that. Actually makes a lot of trouble.

[LudvikKoutny]

So... nope on this one?

[joconnell]

Here's a dirty hack that may help:

1. Set up your lighting.
2. Set up your backplate in the environment and also camera map it on to your ground plane. For the material on your ground plane just use a normal vray material, it'll mean your texture is likely dark to start with, we'll fix that.
3. Temporarily turn off your object that you'll be rendering in the scene.
4. Add in the raw totallighting element.
5. Do a render of your scene using the vray frame buffer.
6. Change to the Raw total lighting element and colour pick an area of your ground plane to get the average of the light that falls on it.
7. In your vray material that has the camera mapped bitmap, swap this map for a vray comptex, keeping the camera map as one of the maps.
8. In the second map, put in a vray colour map and for the value, type in the values you've gotten from your colour picking of the raw total lighting element - this may be more than 1 for some parts.
9. Set the mode of the comptex to divide.
10. Now turn on your main object and do a render.

What happens with the above is we try to remove whatever light was falling on the texture in a way that it'll look the same as a normal 100% self illuminated texture, the difference is that shadows fall on it correctly and it'll bounce up a realistic amount of GI

It looks like a lot with your ten steps but really all it is is using a normal material on the ground with the camera map diffuse, then sticking it in a comptex with the vray colour swatch picked from raw light and you're done. You'll have to adjust that colour swatch as you change your lighting so it always reflects the amount of light falling on the ground plane.

[LudvikKoutny]

Here's a dirty hack that may help:

1. Set up your lighting.
2. Set up your backplate in the environment and also camera map it on to your ground plane. For the material on your ground plane just use a normal vray material, it'll mean your texture is likely dark to start with, we'll fix that.
3. Temporarily turn off your object that you'll be rendering in the scene.
4. Add in the raw totallighting element.
5. Do a render of your scene using the vray frame buffer.
6. Change to the Raw total lighting element and colour pick an area of your ground plane to get the average of the light that falls on it.
7. In your vray material that has the camera mapped bitmap, swap this map for a vray comptex, keeping the camera map as one of the maps.
8. In the second map, put in a vray colour map and for the value, type in the values you've gotten from your colour picking of the raw total lighting element - this may be more than 1 for some parts.
9. Set the mode of the comptex to divide.
10. Now turn on your main object and do a render.

What happens with the above is we try to remove whatever light was falling on the texture in a way that it'll look the same as a normal 100% self illuminated texture, the difference is that shadows fall on it correctly and it'll bounce up a realistic amount of GI

It looks like a lot with your ten steps but really all it is is using a normal material on the ground with the camera map diffuse, then sticking it in a comptex with the vray colour swatch picked from raw light and you're done. You'll have to adjust that colour swatch as you change your lighting so it always reflects the amount of light falling on the ground plane.

Thanks for the effort, but to be honest, entire point of my wish in this wishlist request is to make already quite complex process of setting up backplate integration easier. To me personally, your workflow makes it even more complicated than it already is. It's also quite a hack. I wanted to remove additional steps of having to change camera in camera map per pixel map every time i switch views. With your workflow, not only that this step remains, but it also adds another step of changing color swatch every time i change lighting. Also, there are often more complex matte scenarios than just a flat ground plane

[LudvikKoutny]

Since beta 2 is now out, I would like to bump this one. It should probably be in the problems section rather than wishlist, as it comes down to fixing of two bugs:

1, Matte objects do not correctly project screen-mapped background for GI rays, resulting in necessity of using Camera Map Per Pixel map.

2, Using GI darkens matte objects and creates visible seams even in scenes, where there are no objects that could occlude matte geometry (but even if there were, GI should not create any sort of occlusion as long as dome light is present)

The picture below shows a scene that contains only one dome light and a single plane that is set as a matte object, matte for refl/refr and has enabled catching shadows:

And this is what happens once you enable GI:


Since my last Matte thread, it has already improved a bit, so it would be good to get these last two steps done to finally get it completely right.

EDIT: The second problem probably comes down to GI+DomeLight as GI reacts fine without DomeLight (but of course can not handle high contrast HDRIs with hotspots)

[joconnell]

As far as I know when you enable gi and have a domelight, part of the light from the domelight goes into the direct calculations, the other half goes into the gi calculations, is the the gi half of the domelight calculations not being used properly?

[LudvikKoutny]

As far as I know when you enable gi and have a domelight, part of the light from the domelight goes into the direct calculations, the other half goes into the gi calculations, is the the gi half of the domelight calculations not being used properly?

AFAIK DomeLight completely falls into direct light category. It by design can not, or at least should not cause any global illumination (any light bouncing/color bleeding between surfaces). I don't know how easier could I explain what's going on. I think it should be already clear enough. In problem #2, when shadow catching plane set to matte object is present in a scene with DomeLight and GI enabled, GI seems to darken the plane, even though under given conditions, GI should have no visual effect on the scene whatsoever.

[joconnell]

Try this then:

Make a sphere, give it a grey vray material, no ground plane, just a sphere floating in mid air. Make a domelight and set it to full spherical. Do a render with both lighting and global illumination turned on in the render elements, don't enable GI yet. Have a look at your passes and as expected you're getting the full amount of your lighting in the lighting element and none in global illumination.

Now turn on gi, just primary brute force is fine and do another render, looking at your lighting and global illumination elements.

[LudvikKoutny]

Try this then:

Make a sphere, give it a grey vray material, no ground plane, just a sphere floating in mid air. Make a domelight and set it to full spherical. Do a render with both lighting and global illumination turned on in the render elements, don't enable GI yet. Have a look at your passes and as expected you're getting the full amount of your lighting in the lighting element and none in global illumination.

Now turn on gi, just primary brute force is fine and do another render, looking at your lighting and global illumination elements.

Just tried it, and yes, something weird is going on. But what's important here is that beauty pass remains pretty much identical.

Considering how quiet it is in this thread, I am afraid this issue will remain ignored forever :/

[kosso_olli]

Hope it won't be ignored, because I have exactly the same problems:

In problem #2, when shadow catching plane set to matte object is present in a scene with DomeLight and GI enabled, GI seems to darken the plane, even though under given conditions, GI should have no visual effect on the scene whatsoever.