Image file saving multithreaded

[3LP]

Hi there,

I was just wondering if the process of saving images out could be updated by maybe multithreading the saving process.

We have a bunch of elements that are saved on each render and it looks like the saving process is only single threaded for the moment.
This could take quite a wile, specially with 5k exr render or more...
So I though that because we are going from ram to disk, and the saving process is only converting raw data to image file type data, this could be maybe multithreaded, no?

What are your thoughts Vlado?

Thanks

Stan

[vlado]

Are you writing a multi-channel OpenEXR file? Or many separate files?

Best regards,
Vlado

[3LP]

many separated files.

1) Our main comp program is AFX and this one doesn't support well multi-channel OpenEXR file for my point of view.
2) I like to be able to open exr in pdplayer to see quickly what pass is what (specially for the MultiMatte Elements) when you're tracking a specific thing.
So you need to only "download" from the server a specific amount of data at a time, with multi exr, they are quite heavy and to see or use only 1 or 2 elements, it's bandwidth and ram used for nothing, am I wrong?

Thanks,

Stan

[3LP]

Hey there,

any news about this?

Could this be done and/or considered?

Thanks

Stan

[3LP]

Hi Vlado,

any news on this one?

We are still rendering big exr (5k at least) with 20 RE and it's all doing it single threaded...

Stan

[vlado]

We haven't gotten there yet, but I will bump up the priority on this one.

Best regards,
Vlado

[3LP]

Thanks Vlado,

Highly appreciated!

Stan

[Dariusz Makowski (Dadal)]

Correct me if I'm wrong but cant you just save openEXR as separate passes directly from max? I think I did that in past. Would that not work for you?

[3LP]

Hi there,

Once again, like often, I had to load 4 renders of 3Go vrimg file each and save out the passes to exr.
I was so happy to know that Vray 3 support multithreading.

But I was quite surprised, it does multithreading to read the file but it still doesn't do it for saving...

So saving out passes from a Vray VFB of a 8k x 6k image with 20 passes is still taking like 20 minutes, is stead of 2 if it was multithreaded...

any news if this has been actually advertised but not implemented?

Maybe I'm doing something wrong?

Thanks

Stan

[vlado]

I was so happy to know that Vray 3 support multithreading. But I was quite surprised, it does multithreading to read the file but it still doesn't do it for saving...

It *is* multithreaded and we did the best we could there.

So saving out passes from a Vray VFB of a 8k x 6k image with 20 passes is still taking like 20 minutes, is stead of 2 if it was multithreaded...

There are a couple of things here. First, you only have one disk after all. Not all the multithreading in the world will get the data on your disk any faster than it can physically handle. Two, the multithreaded saving requires additional memory. V-Ray makes some calculations and limits the number of simultaneous writes depending on how much free physical RAM is available. If there is not enough RAM, then we reduce the number of simultaneous writes. Also, the particular file format that you save to has some influence on this.

any news if this has been actually advertised but not implemented?

It is implemented in the best way that we could do it. From our experience it does help to improve writing speeds two to three times. That's probably the best you can hope for.

Best regards,
Vlado

[vlado]

Here are some timings from our internal tests for a 3000 x 3000 image with 18 render elements:

8 threads, exr - 77.5 s
8 threads, png - 19.0 s
8 threads, jpg - 11.4 s
8 threads, vrimg - 51.4

1 thread, exr - 121.6 s
1 thread, png - 86.7 s
1 thread, jpg - 54.8 s
1 thread, vrimg - 177.3 s

As you can see, multithreaded saving helps, but it's not proportional to the number of threads.

Best regards,
Vlado

[alexyork]

That's still an immense improvement. Good work

[3LP]

Ok, well, here are my bench test, just in case you're interested :

8000 x 6000 px - 42 Passes - 2.93Go VRImg file size - Export to : Exr 32 Float | scanline | zip

Max2012 - Vray 2.04.03 :


Load time : 1 m 27 CPU : 12%
Save : 8 min 37 CPU : 5-18%


Max 2014 - Vray 3.05.01 :


Load time : 18 sec CPU - 99-100%
Save : 8 m 53 CPU : 5-18%


Ram Used : 29Go out of 64Go

So for me, there is definitly a improuvement for the loading time, but as far as the saving, this seems to be the same or even a little bit slower...
I understand that the hard drive can be a buttleneck, but I'm writing to a internal SSD that should be close to 350-450Mo/s in writing speed.
Both output folder are doing 2.82 Go file size which means that in theory the SSD could write that in maximum 10 seconds, I'm not sure that this would be the bottleneck.

I've got plenty of ram free (~ 35 Go) at least that's what I think.

I thought I would let you know...

Thanks
Stan

[Dariusz Makowski (Dadal)]

Writing and compressing takes longer than just writting. If you want to save fast disable all compressions. Save to separate EXR not vrimg.

At least thats what I think its happening... then again I'm probably totally wrong

[yavor.rubenov]

The advertised multithreaded saving is done only after rendering when the "Separate render channels" option is on.
It seems that you are doing something different from what we thought. Could you explain how you load and convert those vrimgs ?