4 GPUS on 1 system or 2 GPUS on 2 systems?

[biochemical_animations]

Which is better for cuda rendering?

Rendering two frames on a motherboard with 4 gpus, or two frames between 2 systems with 2 gpus each.

Are there bottlenecks on the motherboard with 4 gpus that cause all the available cuda cores not to be fully utilized?

Are all available cuda cores better utilized when splitting up the frames over multiple motherboards?

EDIT: The title should say "4 GPUS on 1 system or 4 GPUS on 2 systems?"

[Ivan1982]

First one is better, there is lot's of single socket MB who have 4-5 pciex, which is enough! It is much better solution to have everything in one computer because space and cost are reduced greatly compared to 2 computers and it's faster as long as you are using it just for rendering and they are not idling along.

If you plan to use it as WS than you can end up idling that power for most of the time since you will be working other stuff and would be better to use it in WS(1gpu) + NODE(3gpu) configuration.

My personal WS have 1x 1060 only for viewport and 2x 1080 for activeshade and rendering. Nodes however have 4-5 x 1080 that can work 24/7. There is no limitation with utilizing cuda, you can combine different manufacturers, no need for SLI cables everything works.

Take care

[biochemical_animations]

Thanks Ivan. Isn't the pciex slots on single socket MB reduced to 4x speed when multiple GPUs are installed, or is 4x speed enough? Can you give me an example of a MB that you would use with 4-5 slots? I'm planning to expand my render farm by 2 nodes and was wondering about this...thanks.

[Ivan1982]

No, it depends on the cpu you have, if you got cpu with 40pci lanes it will be capable of using 5gpus @ 8x pciex(at least asus x99 deluxe can have it like that), if you are having 2 xeon(40 lane cpu) on asus z10pe-d16-ws you can have all slots at 16x i believe or 16x16x8x configuration(per cpu), not quite sure as i never tested 6 gpus.
It doesn't make any difference performance wise if they are running at 8x pciex either.
From experience you will be better of sticking to single socket motherboards as having more than 5gpu, 6 or even 7gpus per node puts you in another price bracket, it's better to invest that money in another single socket node.
It will add considerable costs if you decide to go with multisocket and more than 5 gpus, additional CPU, more expensive board, ecc dram, better PSU... And you won't be getting anything except waiting it 10 times more for booting as it last forever with server boards.

There is whole array of reasonably priced x99 MB that have 4-5 or even more full length pciex slots, cheap memory, everything is much cheaper for desktops compared to servers.
Either way, you will pre calculating LC for nodes, apart from pcie lanes on CPU(that needs to be 40 for pciex slots) i don't see reason why wouldn't you buy cheapest CPU today for your build as it makes no difference.


https://www.newegg.com/Product/Produ...82E16819117647 overkill 40pcie lanes cpu for gpu node(as it wouldn't do anything except load scene...)

You can also get away with single xeon 2650v3 from ebay(they go for pocket change) they do have 40pcie lanes, they are much slower than 6850k but you won't see any difference for GPU rendering(as it is not used anyways).

https://www.asus.com/us/Motherboards...pecifications/ You can see under expansion slots how resources are menaged when you plug in cards

Any other mb will be more or less same so you can get away cheaper on that side also

https://www.newegg.com/Product/Produ...82E16813157596

most of them support xeon cpus, hope you got general idea how you can build your node cheaply and efficiently.

Take care

[biochemical_animations]

Thanks Ivan. I forgot the CPU was the limiting factor for the lanes.

[biochemical_animations]

The nodes prefiltering LC is done per frame right? Since I recently upgraded to vray 3.4 from 1.5 and things are faster nowadays, I've moved my workflow from creating an IR file, to BF+LC on single file mode for animations, eventhough my camera and objects are moving in the scene...I basically left it at default since it seems to render fine. I do have this right, correct? I did notice the prefilter LC is done by the cpu.

[iancamarillo]

Thanks for the info! I just bought a 1080 and I left my 970 in the machine to see if it could participate. It does on small scenes but eventually doesn't render and has a memory error on larger scenes. I plan on replacing the 970 with a 1080ti (when and if it comes out). But I shouldn't have any configuration problems with a 1080 having 8gb ram and the 1080ti with possibly different amount of ram?

[biochemical_animations]

OK, heres a question, if you have a board that runs 4slots at x8, thats 32 lanes with a 40 lane cpu, 8 lanes are not being used right? Is it more cost effective to go with a 28 lane cpu board and 3 slots that run at x8 for 24 lanes so only 4 lanes are wasted?

[biochemical_animations]

What sized power supply do you run on your node with 4 gpus?

[JezUK]

What sized power supply do you run on your node with 4 gpus?

Dang ! I wanted to know that too

[AlexP]

We've got 4 x Titan X and using this PSU
EVGA Supernova 1600 W G2 80+ Modular Gold Power Supply Unit

[JezUK]

We've got 4 x Titan X and using this PSU
EVGA Supernova 1600 W G2 80+ Modular Gold Power Supply Unit

Thanks Alex.

Do you know what sort of Power In / Out you're getting, especially when under load ?

[AlexP]

No idea no, I'm not sure how to monitor that?

[biochemical_animations]

I just posted a thread on my new WS build in hardware that may help answer some of your questions...hope it helps.

[biochemical_animations]

No idea no, I'm not sure how to monitor that?

Kill-a-watt meter $20ish from amazon