GPU render node build

[cam_910]

Hi, Looking for some advice if anyone has the time on a the specs of a new GPU / RT render node.

I am wanting to build a machine with 4 x 1080ti cards, and I have a few question if anyone wants to chime in much appreciated

-What impact does the CPU have on a GPU rendering machine - ie can I basically pick the cheapest intel i7 I can find?
-Motherboard specs? I see in some of the commercially pre built systems from various vendors they seem to use workstation motherboards and xeons, (at greater expense) is there any reason for this over say a gaming atx baord with 4 card slots?
-What impact does RAM have...?
-Power supply, I am assuming a 1500W would be required.
-Of the cards available, anyone recommend a particular brand? Does a factory overclocked card offer a significant performance increase?

Here's my initial spec based on my assumptions above...which may or may not be reasonable!

https://pcpartpicker.com/list/rYFdTH

Intel - Core i7-5820K 3.3GHz 6-Core Processor
Corsair - H60 54.0 CFM Liquid CPU Cooler Asus - SABERTOOTH X99 ATX LGA2011-3 Motherboard
Crucial - 16GB (2 x 8GB) DDR4-2133 Memory
Samsung - 120GB 2.5" Solid State Drive
Gigabyte - GeForce GTX 1080 Ti 11GB Gaming OC 11G Video Card (2-Way SLI) x 2 (will become 4 eventually)
NZXT - Phantom 820 (Black) ATX Full Tower Case
Corsair - AX1500i 1500W 80+ Titanium Certified Fully-Modular ATX Power Supply



Total: $2884.80 USD

Thoughts?


Thanks!

[Zdravko.Keremidchiev]

What impact does the CPU have on a GPU rendering machine - ie can I basically pick the cheapest intel i7 I can find?

The CPU is used to initiate the GPU rendering and in case you are using LC it will be calculated on the CPU as well; Also not all V-Ray features available in the CPU renderer are supported in GPU. You should check the list of supported/unsupported features: http://docs.chaosgroup.com/display/V...rted+Features; Furthermore we are working on hybrid rendering with GPU and CPU at the moment.

What impact does RAM have...?

GPU memory does not stack, if you have two GPUs in example, one with 2GB and one with 4GB you can use only 2GB; You won't be able to render scenes that require more memory than the one available on the card with the less VRAM.

Gigabyte - GeForce GTX 1080 Ti 11GB Gaming OC 11G Video Card (2-Way SLI) x 2 (will become 4 eventually)

We don't advise you to use SLI you can leave the multi-gpu support to the RT engine

[cam_910]

Thanks for the reply Zdravko.

Sorry I was referring to RAM as in normal CPU RAM (not Vram on the GPU I understand this does not stack, and aware of the non SLI setup sorry that's just PC parts picker copy paste error)

So I guess if GPU/CPU hybrid rendering is on the way it makes sense to put a half decent CPU in there in case the scene maxes out teh VRAM it can 'unload' onto the CPU is that the idea? Whats the ETA on that being implemented?

Is a separate network card required or is standard motherboard Ethernet/networking sufficient...is there potentially any bottlenecks in terms of speed with that much data being pushed to the 4 cards?

Thanks!

[savage309]

Sorry I was referring to RAM as in normal CPU RAM (not Vram on the GPU I understand this does not stack, and aware of the non SLI setup sorry that's just PC parts picker copy paste error)

This is correct. V-Ray (in the beta nightlies) supports NVLINK between Quadro GP100, which allow stacking of the GPU memory though. https://labs.chaosgroup.com/index.ph...e-nvidia-gpus/

So I guess if GPU/CPU hybrid rendering is on the way it makes sense to put a half decent CPU in there in case the scene maxes out teh VRAM it can 'unload' onto the CPU is that the idea? Whats the ETA on that being implemented?

Yes, for the most part. The hybrid CPU render can be used as a fallback mechanism when the scene can't fit the GPUs (insurance like). Also, you can use existing CPU farms (if you have any) and be sure that the result will match the V-Ray CUDA GPU ones. Or render on a machine that do not have CUDA devices at all (like MacBook) or to use CPU+GPU for rendering (CPU+GPU should be generally faster compared to GPU only, but sometimes it does not scale perfectly when you have a relatively slow device like the CPU and a fast one like the GPU rendering together).
Hybrid rendering is coming with V-Ray 3.6, which is planed to be released rather soon.

Is a separate network card required or is standard motherboard Ethernet/networking sufficient...is there potentially any bottlenecks in terms of speed with that much data being pushed to the 4 cards?

If you are not going to use distributed rendering you don't need Ethernet at all. Otherwise, 4 GPUs can produce a lot of data, especially with settings for fast feedback (low rays per pixel and low ray bundle size). In those cases a low bandwidth network can be a bottleneck. If you have production-like settings (lets say 32 rays per pixel, 256 ray bundle size) the network will be utilized much less (check here https://docs.chaosgroup.com/display/...aheavyDRsetup?).

Best,
Blago.

[biochemical_animations]

Hi, Looking for some advice if anyone has the time on a the specs of a new GPU / RT render node.

I am wanting to build a machine with 4 x 1080ti cards, and I have a few question if anyone wants to chime in much appreciated

-What impact does the CPU have on a GPU rendering machine - ie can I basically pick the cheapest intel i7 I can find?
-Motherboard specs? I see in some of the commercially pre built systems from various vendors they seem to use workstation motherboards and xeons, (at greater expense) is there any reason for this over say a gaming atx baord with 4 card slots?
-What impact does RAM have...?
-Power supply, I am assuming a 1500W would be required.
-Of the cards available, anyone recommend a particular brand? Does a factory overclocked card offer a significant performance increase?

Here's my initial spec based on my assumptions above...which may or may not be reasonable!

https://pcpartpicker.com/list/rYFdTH

Intel - Core i7-5820K 3.3GHz 6-Core Processor
Corsair - H60 54.0 CFM Liquid CPU Cooler Asus - SABERTOOTH X99 ATX LGA2011-3 Motherboard
Crucial - 16GB (2 x 8GB) DDR4-2133 Memory
Samsung - 120GB 2.5" Solid State Drive
Gigabyte - GeForce GTX 1080 Ti 11GB Gaming OC 11G Video Card (2-Way SLI) x 2 (will become 4 eventually)
NZXT - Phantom 820 (Black) ATX Full Tower Case
Corsair - AX1500i 1500W 80+ Titanium Certified Fully-Modular ATX Power Supply



Total: $2884.80 USD

Thoughts?


Thanks!

Consider checking my thread here
https://forums.chaosgroup.com/forum/...3-ghetto-build

Total cost was $4,000 and was the fastest GPU build on the Vray Benchmark until recently, now it's 3rd, but it is still the best bang for buck. 1st and 2nd place spent more on half their build than I did for my entire build. This thing is a beast, rendering crazy refraction scenes in less than 5 minutes at HD size. 4K takes about 30-40 minutes...it's a beast.

Active shade rendering is almost real time. I can make lighting and texture adjustments and see the result immediately. No more waiting a minute or two, or region rendering.

CPU's useful for light cache/IR calculation only, minimal usage for GPU render. I actually switched to brute force + brute force recently which is faster.
Power supply, see my thread.
I'm running 1600 MHz RAM, and it's fine.

[richard_rosenman]

Hey gang;

I'm about to get rid of my renderfarm and go with GPU rendering as well. Therefore, this post was very helpful. I too have my eyes set on the GTX 1080 Ti's. Ideally, I'd like to build a system with 4. I understand some of the limitations such as ram and some not fully supported RT features. However, for my needs I think I could get by and I'm sure VRay RT GPU will continue to mature.

I currently have a small renderfarm of 16 core i5's (I believe) which I use to render all my animation. Is there any way to determine how to produce a similar performing system with GPUs? For instance, how many 1080 Ti's would I need to match that kind of performance?

-Rich

[biochemical_animations]

Hey gang;

I'm about to get rid of my renderfarm and go with GPU rendering as well. Therefore, this post was very helpful. I too have my eyes set on the GTX 1080 Ti's. Ideally, I'd like to build a system with 4. I understand some of the limitations such as ram and some not fully supported RT features. However, for my needs I think I could get by and I'm sure VRay RT GPU will continue to mature.

I currently have a small renderfarm of 16 core i5's (I believe) which I use to render all my animation. Is there any way to determine how to produce a similar performing system with GPUs? For instance, how many 1080 Ti's would I need to match that kind of performance?

-Rich

Instead of "getting rid of your render farm" why not add gpu's to your renderfarm and use hybrid rendering....that way your existing cpu's can be utilized as well. You can plug in the gpus with an extender if your mobo's are limited with space.

[richard_rosenman]

Instead of "getting rid of your render farm" why not add gpu's to your renderfarm and use hybrid rendering....that way your existing cpu's can be utilized as well. You can plug in the gpus with an extender if your mobo's are limited with space.

Well, the reasons for getting rid of the renderfarm are many:

- each of the two racks of 8x blades each weigh approx 300 pounds
- they require specialized cooling
- they require specialized electrical
- the produce an insane amount of heat
- they are noisy
- they are rack mounted so yes, an extender for video cards would have to be added requiring space outside of the chassis
- they break down often and require constant maintenance
- they are not portable

I could probably go on and on...

But the main reason is I have to move to a new studio and want to be able to be portable. Portable allows me to move into any office. If I want to take these, I need to find a space with a machine room, and set up cooling, electrical, etc, again. More costs.
I could take one render rack with me - so 8x Core i5's and just use them in whatever office I end up at - but they are extremely noisy and although one might meet standard power outlet requirements, it will heat up the office in a minute to unbearable levels.

Hence my interest in GPU.

But the project I'm doing now, it takes 10 minutes to render a 1280x720 frame with VRay RT GPU, with an older Nvidia 970. That makes rendering minutes of animation impossible on that one computer. Hence I was interested in knowing if buying a system with 4x Nvidia 1080 ti's would at least come close to allowing me to render animation on that one computer in a decent amount of time. I feel I am still a bit too early for GPU rendering (on the cusp) but the move is forcing me to go ahead in that direction.

At the moment, and due to that 10 minute rendertest, I am not very hopeful.

Any thoughts?

-Rich

[richard_rosenman]

Actually, just double checked and they are Core i7's - so fairly decent.

-Rich

[biochemical_animations]

I just updated my ghetto build thread with pics and info on my super ghetto gpu render farm.

[biochemical_animations]

Well, the reasons for getting rid of the renderfarm are many:

- each of the two racks of 8x blades each weigh approx 300 pounds
- they require specialized cooling
- they require specialized electrical
- the produce an insane amount of heat
- they are noisy
- they are rack mounted so yes, an extender for video cards would have to be added requiring space outside of the chassis
- they break down often and require constant maintenance
- they are not portable

I could probably go on and on...

But the main reason is I have to move to a new studio and want to be able to be portable. Portable allows me to move into any office. If I want to take these, I need to find a space with a machine room, and set up cooling, electrical, etc, again. More costs.
I could take one render rack with me - so 8x Core i5's and just use them in whatever office I end up at - but they are extremely noisy and although one might meet standard power outlet requirements, it will heat up the office in a minute to unbearable levels.

Hence my interest in GPU.

But the project I'm doing now, it takes 10 minutes to render a 1280x720 frame with VRay RT GPU, with an older Nvidia 970. That makes rendering minutes of animation impossible on that one computer. Hence I was interested in knowing if buying a system with 4x Nvidia 1080 ti's would at least come close to allowing me to render animation on that one computer in a decent amount of time. I feel I am still a bit too early for GPU rendering (on the cusp) but the move is forcing me to go ahead in that direction.

At the moment, and due to that 10 minute rendertest, I am not very hopeful.

Any thoughts?

-Rich

building a WS with multiple gpus will help, amazingly. But the benefit decreases the more gpus you add to a certain point. For example, my tests have shown, rendering a 2 hour frame with one gpu (evga 980 Ti classified) , could be cut in half by adding a second card. Adding a third card, may not halve it again to 30 minutes but reduce it to maybe 40-45 minutes, adding a forth card may only reduce it to 30 minutes, adding two more cards to six may only get me 20 minute renders. I'm still happy with that, but when I watch the gpu usage gadgets on my desktop, I see the gpus work 100% during the BF+BF crunching, but during the render passes, denoising stuff, pixels per whatever reduction stuff, only one or two gpus are bouncing 100 % for a number of minutes... which tells me maybe the drivers or vray rt software needs to be developed to take into account multi gpus better. I remember Vlado mentioning rt was going to totally rewritten or something like that. I think the gpu render software currently written is great and I'm super grateful for it, but I do think there is room for improvement.

the only way to know is to jump in with what you want to spend...test things out and expand as necessary. But getting the cards your mentioned will be night and day over the 970 you use. Btw, that is where I started last year...one 970.

one more thing...what I mentioned above applies more towards production rendering...when ur working in active shade setting up lights and material, having all thos gpus update almost in real time is going to flat out amaze you...that is the benefit of the WS with multi gpus. But I've found the multi gpu WS struggles with backburner renders. My render farm ( referenced in my ghetto build thread) handles BB jobs really well because each node only has 2 gpus. But, if you production render outside BB on you WS you should be fine. In fact I don't run my WS with a BB render...I wrote a script that renders outside bb in reverse. So my render farm uses bb forwards, and my WS renders outside bb in reverse. Using skip existing images keeps renders from overlapping and stops everything accordingly. It's wonky but it works until all this software gets worked out properly.

long story short, I would highly recommend the multi gpu setup you mentioned.

[richard_rosenman]

Hi biochemical_ animations;

Thanks for the info - I wasn't aware that adding extra cards doesn't double or triple the performance. I actually believed the opposite. So that's not great news.

But more importantly, it doesn't sounds like this can replace a 16 core i7 renderfarm, even partially. That's the big problem. It looks like you'd need maybe 4 of those quad-gpu machines to be able to plow through the same amount of animation as the 16 blade renderfarm does... and that's just a quickly guessed number on my end...

So that's unfortunate.

-Rich

[biochemical_animations]

Hi biochemical_ animations;

Thanks for the info - I wasn't aware that adding extra cards doesn't double or triple the performance. I actually believed the opposite. So that's not great news.

But more importantly, it doesn't sounds like this can replace a 16 core i7 renderfarm, even partially. That's the big problem. It looks like you'd need maybe 4 of those quad-gpu machines to be able to plow through the same amount of animation as the 16 blade renderfarm does... and that's just a quickly guessed number on my end...

So that's unfortunate.

-Rich

it does double or triple performace to certain point. But then the performace gain isn't as great with the last few cards as it is with the first few.

for example, adding a second card, you should see double performace. Adding a third card will probable half render time again, the 4th card you may not see another halve in performace, but a bit less, 5th card even less. And so on.

my WS has six cards, you'd think it performs like 3 nodes (2 cards each), but it's more like two nodes.

Having a multi gpu WS is great and you'll see a gain, but it won't replace a farm. I upgraded my farm first, and loved it, then found "look" workflow slowdowns that pushed me to build the WS.

it's a lot to process I know, but it can be done very cost efficiently, thus the purpose of my ghetto build thread.


Hopefully this helps a bit.

[JezUK]

it does double or triple performace to certain point. But then the performace gain isn't as great with the last few cards as it is with the first few.

for example, adding a second card, you should see double performace. Adding a third card will probable half render time again, the 4th card you may not see another halve in performace, but a bit less, 5th card even less. And so on.

Hmmmm, I'm not so sure about this - I pretty sure (from memory) that my performances increased linearly when I went from 2 to 4 gpus.....

Could it be an issue with riser cables or the pci-e lanes ?

I've gotten all the parts for my open rig (but still haven't gotten around to building it) - and the plan is to have 7 gpus - so I'm hoping my memory is right..... (actually, I can try a test later this week as I am currently back to two cards temporarily).

[richard_rosenman]

Hi JezUK - I see you have a similar system to the one I was considering with 4x 1080 Ti. I would be interested to see your test with regards to performance with 1 vs 2 vs 4 GPUs on a scene. Perhaps I can also see how long that scene takes to render on a CORE i7 and get an estimate of CPU to GPU performance...

-Rich