Mobo for multiple GPUs

[AlexP]

Can anyone recommend a good mobo for, say 4 x Titan X?

Is anyone running a similar set up? if so what sort of hardware is making up the rest of system?

Any pointers appreciated

[3LP]

Hey
we are running 4x Titan X here

We went for the ASUS X99-E WS as we are trying to cack it to get 7 of those babies onto it, but haven't managed yet, so we run it with 4 cards currently and it works great.
http://www.ldlc-pro.be/fiche/PB00181977.html

I'd say any motherboard that support quad SLI for dual slots will work.

Not sure if you'd like to have any other infos, if so, just let me know.

Cheers
Stan

[AlexP]

Cool thanks.

Is it really possible to go more than 4 on one board with a bit customising? That would be the ideal!

How about things like Ram and CPU, are they things worth worrying about. I ask because, our normal supplier gave me a quote for a GPU workstation which was a bit OTT with 96gb ram and dual xeon CPU, but I was told this is a minimum requirement for the 4 x Titan X. Which sounds like nonsense to me??

[3LP]

Hey,

Yes I think it's possible to get more than 4 cards with a bit of customisation, we are just in the process of playing with PCIe risers and we are really busy at work so can't get much effort into it ATM. Although I think you'll not be able to get more than 7 onto a single CPU platform.

CPU and ram does play a little bit.
For CPU, there are two things :
1) you should get one with 40 pci lanes and not 28 like the cheaper intel cpu.
2) it's more about how it will manage the GPU to keep them working at full speed and you don't want you cpu to be the bottleneck. On our GPU node we have a 5930k and it uses about 20-25% when rendering, so just guessing it will use about 40-50% if we manage to get it working with more cards.

Now, if we don't want to hack thing around, the proper way (or at least how it's advertised out there) is that you should not go for more than 4 cards per CPU, but till 4 you're cool.
So if you'd like to build a setup with more than 4 cards, you should probably look into getting dual xeons to be able to get up to 8 cards in a single rig. power for $ though, I would not go down that road, and that's why we tried to hack it but even if you need to build two nodes, it will end up way cheaper than if you build it with xeons, but like always, it takes more space.

For the Ram on the motherboard, it also depends on what scene you'll render out. For some reasons, and I don't know why, CG seems to be looking into it actually, some scene take a lot of system Ram to get it working on the GPU.
The scene I'm working on for the last 3 days takes 9.8 Gb tops in GPU ram, but eat up my 32 Gb of ram on the system, and I suspect it to swap to the ssd.
I knew about that issue before purchasing the gpu node so I throw 32Gb in straight away but it seems can still be not enough. Bare with me, it does render, it's just lagging when the render starts and end because of the swap, and often max freeze if I want to send 2 times the render during the same max session, I guess because it swap so much.

I hope this will be fixed and that any gpu node would work with 4 or 8Gb of ram. Money is better spend in the real deal, the GPUs

So if you want to go with a 4 Titan X rig I would definitely not go xeons and not go for 96gb of ram, this seems way overkill, a simple i7 with 32Gb should do the trick.

If you have a typical scene you'd like me to send a render and see how it react on our node, just let me know.

Cheers
Stan

[AlexP]

Interesting... good info thanks

We have borrowed 3 x Titan X to test at the moment.

How are you planning to approach cooling if you get all 7 in?

[AlexP]

Ready for some RT
Cant wait to get all these hooked up, another 4 on their way

[3LP]

Nice, seems like we all did the same kind of shots



So you'll get 12 in total?
On 3 different nodes?

Stan

[AlexP]

Yep to start with, then look in to doing something similar to your second shot where we have a 4+ on one board.
What is that cage/frame? Where did you get it from?

[3LP]

Because the initial website is still under construction, here is a page with the details as reference only as you should not buy it from there :
http://www.samenkopen.net/action_product/280710/905123

Here is the direct email of the guy who sells them : info@bitmakers.nl

I search a lot through the net for weeks to find a good frame/build and purchased already two of those, they are great, I can only recommend them and the guy that sells them is very professional.

Let me know how you go

Cheers
Stan

[AlexP]

Nice, thanks. Love to have a few of those fully loaded.

[3LP]

I can imagine!
What's your goal? How many cards are you planning to get per node?
I'm still in the process of getting 7 on the same node, but I have some issues with the risers.

Stan

[AlexP]

Goal is MORE POWER!!

Really though, to start with we have our workstations loaded with the 4 x GPUs so we can work effectively, then slowly replace our CPU renderfarm with a GPU renderfarm of sorts I suppose?
Maybe over time add a couple more cards to our workstations also.
Ideally Id like to shove a Quadro in there for the viewport, but not quite there yet.

Ive seen some server mobos that have plenty of PCIe slots and it should just be a case of working out the risers and mounting situation. Very interested to hear whether you manage the 7 on one board eventually?!

[auke]

Does anybody know the actual limits (and what causes them) of the number of GPU cards in a system? As I have been looking around but I can't find working examples of systems with more than 4 GPU's, not even with PCIe risers. This suggests to me that there is some limit, other than physical space, to the amount of GPU's installed.

[super gnu]

there are motherboards which accept more than 4 gpu's, and there are expanders such as nvidia VCA and Cubix expander that allow you to beat that limit. but

a) more than 4 cards doesnt really work with sli / games, so nobody uses more than 4 for that.

b) the number of pcie lanes is limited so once you add more than 4 cards the number of lanes can be reduced per card on most single cpu systems.

c) ive been researching 8 + gpus in a single system, using expanders/risers, and there are also bios limits to the number of unique addresses the bios can provide to pcie devices. its hard to find out what the limit is for a given motherboard, as its not in the documentation. again, pro/dual cpu mobos will have higher limits

d) its hard to find psu's that will power more than 4 hot gpus, so that is another reason its not common.

if you want to see systems with more than 4 gpus, dont look at gamers, look at bitcoin miners and HPC.

[auke]

Hmm, im still a bit concerned, both the VCA as Cubix Xpander only use 4 gpu's per cpu. (VCA has 2 cpu's internally, Xpander stations for desktops require a dual CPU workstation to connect to and can only hold 4 GPU's. The server based Xpander rely on another server for control but these generally have 2 or 4 CPU's anyway.) I am aware of the rest of your points, First I was under the impression the PCI/e lane number was the only limiting factor (apart from physically being to fit so much gpu's in a mobo. Though looking around on bitcoin miner forums am I finding systems with more than 4 gpu's, but they all have either multiple CPU's or windows is having trouble recognizing the devices (which some people seem to have fixed but I can't find how anywhere.). Do you, or anybody else, have a link to a single CPU system running more than 4 GPU's in windows without issues?