Compression in saved exr's from the VFB

[Neilg]

Title says it all really.
Been working on a large animation job for the past month & right from the very start we'd planned to render in BF and use the denoiser. We're 10 days out from delivery and starting the rendering on an amazon ec2 setup today - we've got 15 sequences averaging about 900 frames each at 4k to render.
Yesterday I saw the file size of one of the frames... with the denoiser channels and 10 multimattes the exr's are around 1.4gb each. my maths brings the total frame size of the whole job out at around 19tb. using vrimg only shaves that down by around 10%

In comparison saving the exr's from the max frame buffer with 10 extra channels added on top of the multimattes - z depth, raw light, diffuse, reflection/refraction etc, produces file sizes of 120mb. all in - a total frame size of 1.6tb.

We're on miami beach so the fastest internet known to man here will still take 4 solid days of dedicated downloading to grab those frames, whereas the VFB ones with denoise passes will take somewhere around 50 days to download.

Maybe you didn't need to hear the whole story but at the end of the day my feature request is exr or vrimg compression that is the same as you can get out of the max frame buffer, there are 5 options for it there and none in the vfb. I would've really liked to use the denoiser on this, a day before rendering we're now resigned to delivering a much noisier animation than we have been showing the client in previews. unless we double the rendering power & spend $40k on ec2 instead of the $20k in the budget.


One other thing i'd like to request - when renting online only render licenses of vray, please don't make us go through re sellers. We thought last week that we'd get them early and be able to do some tests, but as of right now - 8 days after sending the first email to get render licenses, we have only just got them.
We know exactly how long every frame takes and our timeline to hit delivery is razor thin, and we're supposed to be rendering by EOD. we've now got less than one day to set up the drive mirror and the entire amazon render farm before we absolutely have to start rendering in order to hit the deadline - and i've never used it before so now my practice run is being forced to do it for real.

[Morne]

In the meantime, what if you save the EXR's with split channel? That should save in the same way that the max buffer saves, except the channels are split, isnt it?

[Neilg]

I'm not sure how that would reduce the file size. The vfb doesn't compress the files, max has the choice between no compression, zip, rle, etc.

The issue is we can only use the denoiser from the VFB - and the VFB with no denoiser already produces files that are 10 times the size of the ones that can be saved out of the max frame buffer.

[Morne]

the split channels option gives you the same saving options as saving from the max buffer

[Neilg]

I think denoiser channels only get saved from the raw image file option, we had some issues with that early on trying to figure out how to get the output files to include them.

[Morne]

It saves the denoiser element depending on the settings you selected. Not sure if it also saves the other additional temp elements the deneoiser creates.

Simple enough to test it on a single frame (mine is busy so can't test now). You'll have to check the docs for the standalone denoiser if it can use the elements like that

[Morne]

Also, are you saving:
32bit full float (32)
or
32bit half float (16)?

Half float files are smaller than full float. Not everybody needs full float

[Neilg]

16bit is about 60% the size, so still comes out to nearly 12tb

It's already too late to troubleshoot, we had to start rendering. My feature request is that when saving to exr's using the vray raw image option we can compress the files.

[Morne]

For the sake of interest, do you currently have in the VRIMG RAW output setting, the EXR enabled for "EXR/VRST/VRSM 32-bit output". Also is Deep EXR ticked?

[Neilg]

deep exr is not ticked. 32bit output is 1.4gb, unchecking that gives a file size of 850mb. just denoiser, nothing else.

A 32bit file with 8 color elements and 10 multimattes saved from max frame buffer (with compression) is 120mb.

[dans]

The VrayOptionRE render element provides additional options to specify compression of the V-Ray raw image file among other things.https://docs.chaosgroup.com/display/...C+VRayOptionRE

Daniel

[Neilg]

The VrayOptionRE render element provides additional options to specify compression of the V-Ray raw image file among other things.https://docs.chaosgroup.com/display/...C+VRayOptionRE

Daniel

Oh man, what a weird place to put it. Was that there in older versions of vray?
Useful to know for the future, thanks!

vlado if you are reading this, renaming that to VrayVFBCompressionsettings wouldn't be a terrible idea.

[vlado]

I'm not able to reproduce such a drastic difference in file sizes when there are the exact same render elements in the files. There is some difference because V-Ray will always write the world position element as 32-bits, but other than that my results show comparable file sizes. If you have a specific example, it will be welcome.

Beyond that, using DWAA or DWAB compression for the exr files (through the VRayOptionRE render element) may cut down the sizes two or tree times without apparent loss in quality, but some experimentation is needed.

Best regards,
Vlado

[Neilg]

I'll do some more tests when the workload slows down - maybe the denoiser elements take up a lot more space than I realized too. The file sizes of non denoised exr's are bigger than i thought too, they're actually coming out at 250mb when there's a lot going on in each frame. We were jumping back and forth with what we were rendering to get the file sizes as we were testing visual parts at the same time.
I have 3tb of frames that will be rendered in 6 days :/