Image file saving multithreaded

[3LP]

Sure,

So just to have a general idea :
We have a farm of 50 nodes with 32 Go of ram.
We are working on a ongoing project for years and the assets are bigger and bigger every months.
We are now at a stage where when we render, the render takes between 29 and 31 Go of ram.

So for the actual render part :
We need to render high ress images of 8000x6000 px often, with loads of multimatte to tweak in post.
The Vray VFB takes 29 Go of ram with that 8000x6000 px with all the passes.
Like you can imagine, 30Go of ram for render, and 29 Go of ram for the VFB, it doesn't fit in the nodes where we send the render to.
I know it doesn't use the ram of the VFB for the spawners, but only for the Master Max rendering, but we send the render through BB and it renders on nodes that has only 32Go and not 64Go.

So my trick was to save out to VRimg, and load that VRimg in a fresh max scene (through the vray history frame buffer).
That means that I break up the ram needed in 2 different stages (render | saving), and each of this stage fits in the ram that we have (32Go).
After that, I use the multi save button to save out every passe to exr...

When we have so big images and project that use both so much ram, I rather export to vrimg just in case, even if max crash, at least we recover part of the render and we can then render again only what missed if we are on tight deadlines (if not I just resend to whole thing).

In any case, that's the only way I found to work efficiently and be able to handle so big jobs, if there is another way to speed this workflow up, I will be happy to hear it

So if the multithread has been implemented only at rendertime, yes then I understand why I don't see any improuvements

Thanks

Stan

[vlado]

So my trick was to save out to VRimg, and load that VRimg in a fresh max scene (through the vray history frame buffer).
That means that I break up the ram needed in 2 different stages (render | saving), and each of this stage fits in the ram that we have (32Go).
After that, I use the multi save button to save out every passe to exr...

Eh, I can see where you should have been a bit more specific Yes, we can try to improve that one too.

Best regards,
Vlado

[3LP]

That's right Vlado, sorry if I'm confusing

We do render some of our big renders like I explained just before, but most of them are normal renders send to the farm through BB but rendering with the Max VFB, not Vray's.

So someone said before that the multi-thread saving would be used if the "Separate render channels" is on, but this means that we need to use the VRay VFB to render?
Does this means that the multi-threading part can not be used when rendering with the normal Max VFB?

I'm saying this cause this could be a issue for us for :
1) RMPanager (I know this doesn't really concern you) doesn't create automatic paths with the Vray VFB but with the Max VFB. Probably this could be updated on RPManager side.
2) RPManager (Again doesn't really concern you) disable the Vray VFB when sending a pass to the network.
3) The crop render in the Vray VFB makes it hard to preview and then just send 20 renders/passes (different cameras) with RPManager without having to go through each pass to disable the crop on each pass...
4) I saw in a "Understanding DMC Sampler" thread earlier that having the "Hilbert Curve" is the best way to handle ram and rendertime (from what I read, maybe you could update us on this point). This means that if the track mouse is on in the Vray VFB, it's not the best in terms of render management. (maybe this has to be updated and disable at saving time like proposal for the crop render. I would go further on and disable both when closing the Vray VFB. When you work and send a render, you will often not close max between those 2 stages).
5) The names of the render elements are not the same as the one from max, but this is a minor adjustment to do on our part if the rest is working

Thanks

Stan

[olly]

The crop render in the Vray VFB makes it hard to preview and then just send 20 renders/passes (different cameras) with RPManager without having to go through each pass to disable the crop on each pass...

Which version of RPmanager are you using? There has been the option to disable the crop on submit for a while now. You can find it under the vray cache manager. It is global though and we are hoping Grant will address this to be on a pass by pass basis.

[3LP]

Thanks Olly, didn't knew about this section!



Yeah, this would fix the issue I would have think of if we rendering with the Vray VFB, but, how about enabling the VRay VFB when rendering to the network?
It seems to allways use the Max VFB when rendering to the network, any option to disable that and enabling the VRay VFB?

Every-time I need to render to VRimg, I load the pass, close RP and send through the normal max BB render.
A pain, I would be keen to ditch the normal max render for that particular case as well.
I might need to take contact straight with Grant for that matter

Thanks anyway!
Stan

[olly]

Just make sure you have enabled the vray vfb and untick the Max VfB check box under common params in rpmanager...

[3LP]

Ah ha, never tried to disable the max VFB,
Will try that

Thanks mate

Stan

[3LP]

Hey Vlado,
Sorry to dig this one out, but I just wanted to know if any improvement have been already made on this one? I'm still on Vray 2.4, but I wanted to keep updated on that matter, we are still waiting ages every day when big renders are saving from Max VFB.

If this has been ported already for Vray 3 this will one more huge feature for us to upgrade

Thanks

Stan

That's right Vlado, sorry if I'm confusing

We do render some of our big renders like I explained just before, but most of them are normal renders send to the farm through BB but rendering with the Max VFB, not Vray's.

So someone said before that the multi-thread saving would be used if the "Separate render channels" is on, but this means that we need to use the VRay VFB to render?
Does this means that the multi-threading part can not be used when rendering with the normal Max VFB?

I'm saying this cause this could be a issue for us for :
1) RMPanager (I know this doesn't really concern you) doesn't create automatic paths with the Vray VFB but with the Max VFB. Probably this could be updated on RPManager side.
2) RPManager (Again doesn't really concern you) disable the Vray VFB when sending a pass to the network.
3) The crop render in the Vray VFB makes it hard to preview and then just send 20 renders/passes (different cameras) with RPManager without having to go through each pass to disable the crop on each pass...
4) I saw in a "Understanding DMC Sampler" thread earlier that having the "Hilbert Curve" is the best way to handle ram and rendertime (from what I read, maybe you could update us on this point). This means that if the track mouse is on in the Vray VFB, it's not the best in terms of render management. (maybe this has to be updated and disable at saving time like proposal for the crop render. I would go further on and disable both when closing the Vray VFB. When you work and send a render, you will often not close max between those 2 stages).
5) The names of the render elements are not the same as the one from max, but this is a minor adjustment to do on our part if the rest is working

Thanks

Stan

[vlado]

Sorry to dig this one out, but I just wanted to know if any improvement have been already made on this one? I'm still on Vray 2.4, but I wanted to keep updated on that matter, we are still waiting ages every day when big renders are saving from Max VFB. If this has been ported already for Vray 3 this will one more huge feature for us to upgrade

We had to pull that feature back; 3ds Max does not like multithreaded writing and it lead to crashes or corrupted files. More research is needed here, it appears that some old image libraries are simply not meant to work in a multithreaded way.

Best regards,
Vlado

[3LP]

Oh that's bad news

Well since you guys made it work with Vray's VFB, I guess we can just go with the Vray's VFB...
But, is there a way to use the VFB and still have all the passes saving like the Max on? But without going through the save separate channels option? That makes it so hard to configure with scripts like RPManager...

Thanks,
Stan