Matching renders to flat DPX scans

[sharktacos]

DPX footage (for example from a Red camera) typically looks "flat" with low contrast and low saturation. The idea, as I understand it, is that the full range is put into the 0-1 range of the 10-bit DPX file, kind of like a film negative, the post facility would then color grade this, making it look good, and provide a LUT for this.

My question is how to match CG to this? You don't want to match to the LUT, but rather apply the LUT in the VFB. However it makes little sense to match the render to the flattened DPX since this is not really what the footage looked like, but more how it is packed into the DPX file, like clothing squeezed into one of those vacuum bags. In other words, it is practical, but ugly and visually wrong. I believe this is not a matter of incorrectly viewing a log file in Nuke however, and instead characteristic of the digital Red camera, which looks very different from a film camera.

Curious to hear how folks deal with this. I'm almost thinking it would be good to get two LUTS, one where the DPX is unflattened (similar to how one processes raw camera footage) and a second LUT where the "look" is applied. We could then match the render to the unflattened footage.

Thoughts?

[Morbid Angel]

No you usually convert the dpx with log to lin in nuke (which linearizes the log space) then you can view it with sRgb just the same.

[sharktacos]

My understanding is that setting the colorspace in the read node does that. Are you saying that there needs to be an additional conversion in addition to this?

I think the issue is not that Nuke is displaying the log file incorrectly making is washed out (as will happen if there is not a log to lin conversion), but that the DPX footage actually looks that way even when viewed properly in Nuke (low saturation and low contrast) because the file has been "flattened" on purpose as a way of storing all the image info into the DPX.

[sharktacos]

Here's a pic that shows how the DPX looks in Nuke viewed in the correct colorspace (top of image) and how it looks after the colorist LUT is applied.

https://static1.squarespace.com/stat...6/?format=750w

We don't want to match the render to either of these, because neither look like a camera. The top one has been flattened, and the bottom one has all sorts of post processing applied to it. What we need is an image that looks the way it would come out of the camera. So my question is how others have deal with this conundrum.

[sharktacos]

To clarify: I had said "scan" which implies it is film footage. It's not. This is digital footage from a Red camera.

I'm wondering if this "flat" look may come from the digital camera used (a Red)? Here's a link to a video that compares digital (Red) and (Kodak) film footage. Note that the digital looks "flat" (low saturation and contrast) compared to the film. Hmmmm...

[kagemaru]

We usually have this color pipeline (99 out of 100 we deal with arri quicktime ProRes 4444 files but dpx, or dng, it's the same).

- Footage is recorded in Log-C (arri Log Color Space)
- Color space conversion from Log-C to Linear
- Match hdri to converted footage and light with this reference
- Color space conversion from Linear to Log-C
- Final Grade

The "linear" color space I refer to is a custom one I created. It's similar to Rec709 (which is what directors, dop, clients, etc usually saw on set). but a bit more flat and desaturated.
I created my Luts with Arri Lut Generator but i reckon every camera manufacturer should have some kind of utility like this.

[jasonhuang1115]

Believe the approached mentioned here in terms of converting DPX is the way to go. What I care more is how lighters would light to match the plate....

In terms of getting an army of lighters to light against plate for integration, if I understand correctly, lighters would get graded plate (be it technical grade, neutral grade, or whatever grade) and be supplied with a corresponding LUT for their CG renders when they light and integrate against the graded plate.

But he issue is as I plug the plate into Maya's image plane, Background environment override, or the Background Image slot of VFB, the VFB LUT (when checked) is affected everything in the VFB. So, it's either A) lighters load the flat plate, light the scene, and view everything through the provided LUT, or B) we need a feature where the VFB LUT will not affect the plate so the LUT is only applied to CG while lighters use graded plate.

Curious how this is handled in your guys workflow?

[kagemaru]

Believe the approached mentioned here in terms of converting DPX is the way to go. What I care more is how lighters would light to match the plate....

In terms of getting an army of lighters to light against plate for integration, if I understand correctly, lighters would get graded plate (be it technical grade, neutral grade, or whatever grade) and be supplied with a corresponding LUT for their CG renders when they light and integrate against the graded plate.

But he issue is as I plug the plate into Maya's image plane, Background environment override, or the Background Image slot of VFB, the VFB LUT (when checked) is affected everything in the VFB. So, it's either A) lighters load the flat plate, light the scene, and view everything through the provided LUT, or B) we need a feature where the VFB LUT will not affect the plate so the LUT is only applied to CG while lighters use graded plate.

Curious how this is handled in your guys workflow?

I'm not sure I followed, but if I did, why not having the plate exported in a way that when in the vfb with the lut applied you get the correct output?
That's what I put in place here.

[jasonhuang1115]

I'm not sure I followed, but if I did, why not having the plate exported in a way that when in the vfb with the lut applied you get the correct output?
That's what I put in place here.

Yes, that's actually the A) approach I mentioned. Plate is processed externally from log to lin or what not (in Nuke or other app) and exported with a LUT. So if lighters view the plate without the LUT in VFB, it's fairly "flat". Then lighters can load the LUT along with the exported "flat" plate to light CG stuff, so everything (flat plate and lit CG) is viewed through the LUT and consistent.

[sharktacos]

Yes, that's actually the A) approach I mentioned. Plate is processed externally from log to lin or what not (in Nuke or other app) and exported with a LUT. So if lighters view the plate without the LUT in VFB, it's fairly "flat". Then lighters can load the LUT along with the exported "flat" plate to light CG stuff, so everything (flat plate and lit CG) is viewed through the LUT and consistent.

If you are using the word "flat" the same way I am, meaning it looks like the top image I posted, then this would not work. That flat image (low contrast, low saturation) is the result of the LOG format, and is used to store data. This is a post process. It would not be possible to take a photo that looked like this. It is therefore equally not possible to make a render that looks like this, to the extent that a renderer does what cameras do. So what you need is to have a backplate that looks like an actual photo, render to match that, and then apply whatever "look" LUT the show uses to both.

[kagemaru]

If you are using the word "flat" the same way I am, meaning it looks like the top image I posted, then this would not work. That flat image (low contrast, low saturation) is the result of the LOG format, and is used to store data. This is a post process. It would not be possible to take a photo that looked like this. It is therefore equally not possible to make a render that looks like this, to the extent that a renderer does what cameras do. So what you need is to have a backplate that looks like an actual photo, render to match that, and then apply whatever "look" LUT the show uses to both.

It's actually the opposite of what you're saying, if I understood correctly.
The log color space is what the camera sensor is actually seeing, or let's say it's a bit more faithful to what the camera is seeing. It's comparable to the film negative before the digital recording age.
Rec709 or sRgb for what matters is the conversion we do to see a more pleasant image compared to what we are used to, and people usually light and render in this color spaces.
As i wrote before after the 3d/vfx work you can go back to log in order to do color grading with the maximum amount of range possible (talking about shadow/mid/highlight ratio). But that's not a must. I guess it depends on budget/work/people involved, etc.

[sharktacos]

It's actually the opposite of what you're saying, if I understood correctly.
The log color space is what the camera sensor is actually seeing, or let's say it's a bit more faithful to what the camera is seeing. It's comparable to the film negative before the digital recording age.
Rec709 or sRgb for what matters is the conversion we do to see a more pleasant image compared to what we are used to, and people usually light and render in this color spaces.
As i wrote before after the 3d/vfx work you can go back to log in order to do color grading with the maximum amount of range possible (talking about shadow/mid/highlight ratio). But that's not a must. I guess it depends on budget/work/people involved, etc.

It's a matter of semantics. LOG is a way of storing data efficiently, not a way to show it. So, as you say, you can go back to LOG in order to do color grading with the maximum amount of range possible.

Visually, basically LOG does two things:
(1) It lifts the blacks considerably, which tends to desaturate the colors

(2) The brightest highlights are heavily compressed


That is, when viewed so that it looks flat, as described above, this is not meant to visually represent how a photo looks. Photos are not flat. It's basically a parallel to linear rendering. Linear EXR is a great way to store image data. That does not mean we should view the linear renders dark (i.e. without a sRGB gamma applied in a LUT). Visually flat is wrong, just as dark is wrong.

[jasonhuang1115]

Maybe check Jeremy Selan’s cinematic color course note or VES handbook regarding plate timing (Ch.6 around p531).

Believe there should be a log2lin process and a normalized/neutralized grade applied by DI or done inside of vfx facility before handing the pre-graded plates (could be low rez exr or other non-float format) to lighters to light against. If the final look involves significant creative grading and you want lighters to be able to preview or light against the final graded plate to some extent, certain percent of the final creative grade could be applied to the neutralized plates. Still, all these need to be reversible, hence a LUT to be supplied.

The "flat" image I referred to is the one that has gone through log2lin and technical grade (where you normalize or neutralize plates). I refer to that as “flat” mostly as opposed to the one that has the final “look” where a creative grade is applied if that makes sense.

Marcie.cin viewed with colorspace specified as "raw" in Nuke. I wouldn't light against this either.
Marcie.cin viewed with colorspace specified as Cineon in Nuke. (Nuke automatically linearize it without inserting a log2lin node). I think in our context, your lighters can light against this plate. If you do a normalized/neutralized/technically graded or certain percent of creative grade to this plate, a LUT needs to be supplied for lighters.

This is my understanding and will be happy to be corrected.

[sharktacos]

Thanks Jason, we are definitely on the same page with all of that. Thanks for the link, I'll check that out.

As an aside, as I have been testing things between Nuke and Resolve I found that Resolve does not have some really basic camera LUTS. For example I wanted to view LOG footage in Rec709, but there was not a Log2Rec709 LUT (it has like 95 LUTS, just not that one). So I wrote one in Nuke. Easy enough to do, but I was kind of baffled that Resolve would be missing something so basic. Curious.

[sharktacos]

Wow, Jason, this is a great read! Thought I would share some parts that were really helpful in expressing the stuff I have been trying to say (not nearly as well as the author) in this thread.

on VFX working with digital film footage and the use of LUTS...

"Plate photography is brought into the pipeline in whatever color space is delivered (camera log, commonly), and then typically converted to scene-linear using an invertible transform. Often a pure 1-D camera to linear transform is used, though sometimes the linearization is augmented with a matrix transformation. For visualization, a 3D-LUT is used which emulates the eventual output processing, which may either be a print film emulation or a similar “S-shaped” transformation. During the visual effects process, the display transform is never baked into the imagery except for “throw-away” deliveries such as editorial output or rough preview screenings. The delivery to DI is expected to be the highest-fidelity representation of the original photography."

and on DI and the use of LUTS...

"There are two main approaches to handing color in digital intermediate. The first “video-centric” approach is where display-referred imagery is loaded into the DI; no viewing transform is necessary. The display-referred frame buffer is directly manipulated in the color corrector (akin to a motion-picture version of Photoshop). In the second “film-centric” approach, scene-referred imagery is loaded into the machine and a viewing transform (3D-LUT) is required to create the final the color appearance. The color correction manipulates the underlying scene-referred representation, but all color judgements are
made previewing through the display transform."

"In the “filmic” approach, working with a scene-referred imagery, a high-fidelity representation of the full dynamic range from the original camera is loaded into the color corrector. Camera raw imagery is directly graded on rare occasion, but more commonly a camera-specific integer “log” encoding is utilized... the appearance of the imagery in DI is modified by color correcting the image in the scene-linear (or log encoded) color space, and making visual decisions based on the appearance post view transform."