GPU farm, Deadline affinity

[LL_Will_J]

I am looking at putting together a GPU farm. I would like to stack 4 GTX1070's per machine. Ideally I would be using the 4 GPUs working together for some tasks (large print resolutions) and individually on separate frames for image sequences (rendering 4 frames at the same time).

I am currently batch rendering with Deadline. I know Deadline can Handle GPU affinity (which should allow me to manage this exact situation). But does Vray GPU support this way of working?

I was wondering if anyone has experience of this setup? How difficult was it to get it up and running? Managing the switch at submission time? What are the pros and cons with regards to render time? Any hardware tips you guys could point out?

Many thanks

Regards

Will

[LL_Will_J]

Hi Guys,

Just wondering if you have an answer about this? We are gonna go GPU farm soon anyway (particularly hearing about Vray Next). We are often rendering a high volume of light scenes (1 or 2 mins a frame and relatively low memory usage) - so this could be a really nice workflow.

https://deadline.thinkboxsoftware.co...ity-and-beyond

This is what deadline says about it. So it seems like that side of things should work. Its just a case of having control of this in Vray.


Also - (I hate to do this!) here is an example of how it works in redshift:

Does Redshift support multiple GPUs on one machine?

Yes! Redshift can be configured to use all compatible GPUs on your machine (the default) or any subset of those GPUs. You can even mix and match GPUs of different generations and memory configurations (e.g. 1 GTX TITAN + 1 GTX 1070). Redshift supports a maximum of 8 GPUs per session.

Using a render manager (like Deadline) or using your 3d app's command-line rendering, you can render multiple frames at once on systems with multiple GPUs. This can help ensure that the GPU resources are used as efficiently as possible

Cheers

Will

[vlado]

You can use the environment variable VRAY_OPENCL_PLATFORMS_x64 to specify which devices to use for GPU rendering. See here for more details:
https://docs.chaosgroup.com/display/...ables-V-RayGPU

There are some examples provided.

Best regards,
Vlado

[LL_Will_J]

Thanks Vlado. That is cool.

Using the example of a machine with 4 GPUs:

Using the environment variable - I could submit 4 jobs and have each one render on a different GPU simultaneously . Which is very cool.

But what about submitting 1 job and having each GPU pick up a different frame? So a 40 frame sequence would render 10 frames on each of the 4 GPUs on the same machine?

(ie: rendering multiple frames from a sequence at the same time)

Regards

Will

[vlado]

But what about submitting 1 job and having each GPU pick up a different frame? So a 40 frame sequence would render 10 frames on each of the 4 GPUs on the same machine?

That would be a question for the Deadline guys.

Best regards,
Vlado

[psanitra]

It`s working with deadline just fine. In Deadline submitter set your GPUs per task to 1, it`s in the Render tab. Then set number of concurrent tasks to what ever number GPUs you have on that machine... , it will even pick up the cuda c++/CPU if you want to.

[LL_Will_J]

Thanks Guys,

This is great to know - really looking forward to trying this out.

Cheers

Will

[tobiasroesli]

Hello,

I want to update this post with some Vray/Maya Facts about Deadline.

I recently created a little Renderfarm with a few GPU Nodes.

I also read this post before I set up the Farm.
After installing the Farm I tested a few Days with GPU Affinity and also some support from Deadline itself.

We figured out that GPU Affinity is not yet supported with Deadline 10.0.28.2 and Vray for Maya.

The thing is that if you render with the Concurrent technic the GPUs add the Ram with the amount of GPU's you have.
So for Example, if you have 4 Cards and you say you setup 4 Concurrent Tasks on every Card will be 4 Tasks. So you render 16 Tasks at the same time. Most of the Time the Job Crashes because of the full Vram.

There's another technic with Slave Instances (multi-worker) https://www.awsthinkbox.com/concurre...rker-rendering

I really like this Technic but somehow Deadline can't setup the GPU Affinity right yet, it always takes all of the GPUs.

Thinkbox told me that they opened an internal Feature Request on that GPU Affinity/Maya/Vray Problem.
So we have to wait.


Cheers
Tobias

[francomanko]

anychance you can post your machine specs/setup/cooling etc for the 4xGPU?

[psanitra]

tobiasroesli Hi, i`m pretty sure multi slave works. I`m using multislave daily on my workstation with each GPU assigned to different slave. attaching screen that i`m rendering right now just on second GPU.
I guess you just don`t have deadline configured properly. Try to remove slaves completely from the list - disable them(so they show offline) and then remove them in monitor and delete their slave config files from disk. Then create new slaves and set the GPU affinity again. Close monitor and restart the system, it should pick up all stuff correctly. good luck

[tobiasroesli]

psanitra Hi there, thats good to hear. I also did it like that. I can just speak for Maya and Vray. The Thing is that it is harder to detect if you just use 2 GPU's. I guess I would have never figured out with two GPU's. The thing is no matter wich GPU you assign in the GPU Affinity Tab it will always take all. I made a Chime Session with the Technican from Thinkbox and after an hour he said that he check with the Developers and they said its not supported yet it always takes all the GPU's. Maybe its working with Houdini and or Max but Maya it isn't...

I always compare the Vray Statistics for the Vram usage with GPU-Z. You see it pretty good if its multiplying with the amount of GPU's.

Cheers
Tobias