Automatic Device Management

[Joelaff]

I was wondering...

We have a heterogeneous render farm, especially when it comes to graphics cards. There is a lot of variety and various amount of memory.

How hard would it be to have VRay GPU disable a GPU if it failed to allocate enough memory on that device? The particular device (GPU) would be disabled for the current run only (not for future renders). However, the render would continue on any other GPUs and the CPU as well.

We are finding sometimes we have to go through and disable the GPUs on some nodes for complex scenes. Obviously this is slower, but the CPUs alone are still quite productive on an animation.

Right now VRay GPU spits out an error that it can't allocate the memory. Perhaps it could instead just disable that card and continue trying the other devices (GPUs/CPU). This would fall into the It Just Works category! (it could be built with the Do what I Mean Compiler )

Thanks.

[a7az0th]

This is something we wanted to do in the past, byt it proved much harder than initially thought. However things have changed since then, so it might be worth revisiting the idea. I will make a note to estimate the complexity Thanks for bringing this up

[Joelaff]

Thanks for the reply, Alexander. Would be great if there was a way to make it work.

I know we have eschewed GPU due to issues like this. It is typically just faster and easier to eat the render time on the farm than to deal with the issues. But if there was a way to automatically ignore cards without enough memory (for that scene or even just that frame-- in my current scene early frames are no problem, but as more bubbles appear the memory use skyrockets, and later frames cannot use the more limited hardware). that would be amazing!

GPU has made such progress, and now most things are working well, just need to figure out the hardware teething troubles.

[a7az0th]

With the upcoming Out-of-Core we are working on this issue will be gone automatically. Each and every GPU will run on its own speed and capabilities. The ones with lower memory pool will just start evicting sooner and will therefore render slower, but they should not fail.
We're fully commited to Out-of-Core, so stay tuned as we share updates on our progress.

[Joelaff]

That sounds incredible! This is kind of the gold standard we are hoping for (I'm sure you too). See my other thread, about out of core when you get a moment. https://forums.chaosgroup.com/forum/...re-limitations
Curious about that.

Thanks.