Ghetto build

V-Ray V-Ray Common Hardware
1

So, I’ve had a bit of fun the last few weeks on my new ws build that I thought I’d share.

After doing research on other sites, and getting advice from members, this is the current result.

It’s not the most sophisticated build, but it is a nice and fast gpu powered build.

The MB is a supermicro x9drx with 10 pcie slots at 8 x gen 3, and an 11th at 4x gen2 i think, they all actually run at gen 2, but are plenty fast.

There are 7 x 980 ti’s classifieds, which I got for between $290-$380 each as they are evga b-stock.

Dual e5-2670 which were only $233 for the pair run the cpu computing. I was able to get 8x8Gb sticks of ddr3-1600 for $200 from natex (used server ram ecc).

Power was a bit of a challenge. The key, I believe from what I read and tested, is to isolate the graphics cards power not plugged into the MB. I have two cards in the MB running off a main evga g2 1000w psu powering the MB and CPUs. I went with powered pcie riser ribbons 8x-16x that are also powered by the main g2 psu. An extra evga g2 1000w psu powers 3 cards, an 750w psu powers 2 more cards.

The extra psu’s are always on via the tester tool, basically just a jumper, then turn on the main psu to powerup the system. I got psus that have an eco mode smart fan, so the fans only turn on when they need to. I measured the extra psu’s when on and the system off via a kill-a-watt meter pull only 15 watts…so I leave them on all tne time.

Everything runs great, and the gpus run between 60-74 degrees C.

I’m sharing this because I really believe gpu rendering is the next wave of the future, at least for animation, and I’d like to see it continue to catch attention and advance in development. Granted, I’m still pretty new to learning vray, but if those that knew hadn’t shared with me, I would be where I am now…so thanks.

image.jpg
image.jpg818×611 155 KB

image.jpg
image.jpg818×611 158 KB

image.jpg
image.jpg818×611 147 KB

image.jpg
image.jpg818×611 194 KB

2

sweet, looks like you know what your doing there, nice work

3

Yeah, thanks to your help

4

Very nicely packed :slight_smile:

Best,
Blago.

5

Nice work, reminds me of my first farm I built to sit between the wall studs in the basement of my old house. It worked pretty well too as the studs were mounted against the concrete basement wall and the wall stayed cool all year which helped in the cooling of the slaves especially in the winter. Just need to remember to keep all components an inch or two away from the concrete or put a vapor barrier between in case any moisture seeps through :slight_smile:

6

I have a newbie question as I go about learning about the benefits of GPU rendering.

And I’m new to this type of building.

My motherboard has 7 x pci-e slots (16x) and I’ve currently taken up all the slot space with my 4 x 1080 ti’s (each 1080ti takes up the space of 2 slots, even though a card only uses one pci slot, it blocks another one).

So, is it a question of just moving the GPUs away from the MB (by using Risers) that I’d be able to theoretically use all 7 slots with 7 x 1080ti’s ? (my PSU is 1500w and I’d believe that’d be powerful enough for 7 cards).

Dang, I need to know where I can buy a stand for 7 gpus… (I don’t have the facilities to make my own)…

7

1500w psu would unfortunately be nowhere near enough for 7 x 1080ti’s under load each pulls between 250 and 300w.. so thats 2.1kw just for the gpus, not including the rest of the system. obviously those are max figures, but you should always leave a healthy amount of headroom in your power supply calculations (as psu’s age they handle max load less well) . id not feel relaxed with less than 2.6kw for 7x gpus and a system to run them.

8

I use these

one thing about power, use MB psu for gpus plugged into MB…use extra psu’s for gpus plugged into risers…use MB psu to power riser.

use kill-a-watt meter to measure wattage.

check gpu power usage beforehand…for example if gpu uses 250 watt, plan that for psu, either extra psu or MB psu.

mb psu will use cpu, memory, hdd, so check https://outervision.com/power-supply-calculatcalculators for psu calc.

you need to get the power correct or you could fry your cards and system. Having more power on hand is recommended. I have three 1000w psu’s now, one for MB, cpu, hdd, risers, and a gpu, the other gpus run off the other psu’s.

9

btw, yes, if you move the gpus you could use all of them with risers…if your MB and os can support it. I had to enable 4G decoding in bios after like 5 cards…windows 7 was able to handle all 7.

10

Hmmm, I’m not so sure about that (though I have seen a lot of videos since I posted last night and yes, people are using 2 PSU’s so maybe it’d be the right thing to do, I do have another spare lying around :slight_smile: )…

But my Corsair Link tells me my 4 x 1080ti’s under full load pull something like 850w out of the wall and a little less into the system (circa 830w). From memory, when I just had 1 x 1080ti it was around 350w under full load. So I believe adding another 3 would take me around the 1350w mark.

I think it’d be under 1500w by a suffficient margin but I could watch it carefully during first tests and assess it from there.

Either way I’m committing going for this as my tests on my first ever GPU project last week showed that using RT Production my renders were taking just 6 minutes versus 25 minutes when tested under Adv CPU.

So I totally concur with biochemical_animation that “gpu rendering is the next wave of the future” :slight_smile:

11

http://www.tomshardware.com/reviews/…ti,4972-6.html

power consumption is a tricky thing.. id always plan for the absolute peak power your card *can* draw. (and then add a healthy chunk for luck) . not what you see when using it. otherwise you might have unexpected crashes.

id also definitely not want to run a psu at its maximum rating.

you may of course find that on *average* your system might well support 7 cards, although running flat out, your psu could only supply 217 watts to each card, and thats ignoring the rest of the system. .

grab an extra psu for a few of the cards, it cant hurt.

12

I think you’re probably right - I think PSU’s tend to be at their most efficient at circa 50% max, so it’d make sense to have two PSU’s.

Would one 1500w for the 7 x 1080ti’s and one 750w for the MB and Proc do the trick ?

I’m currently looking up Frames for rigs (I don’t have the facility to make my own) - here is the kind of rigs I’m seeing (they all seem ‘mining’ related);

http://www.ebay.co.uk/itm/Open-Air-Mining-Rig-Frame-Case-MinerPark-with-6-Fans-For-8-GPU-ETH-BTC-Ethereum/142428804322?\_trksid=p2047675.c100005.m1851&\_trkparms=aid%3D222007%26algo%3DSIM.MBE%26ao%3D2%26asc%3D43781%26meid%3D80ffc0a0e062460abb0b4edeccf2881f%26pid%3D100005%26rk%3D4%26rkt%3D6%26sd%3D322559966792

13

id do 5 cards on your 1500w psu and get another of 1000 to power the mb and two cards… gives you a bit of headroom, that way the absolute peak draw of the cards will only be the max of your 1500w psu, and on average it would only be putting out around about 1000 - 1200w..

14

that open rig appears to have either pcie1x or pcie 4x connectors on the risers.. youd want at least pcie 8x

15

a few risers here to look at…

16

Thanks Super Gnu - I wasn’t really paying attention to that detail at this stage, but that’s a very important point for me to remember, so thanks for that. Isn’t it actually x16 I should be looking for ?

I like the form and materials of it one though - of course it wouldn’t be too hard at all to build one myself. But I would rather have it going upwards though, like a tower case, so that I could put it under a desk / table.

Hmmmm… maybe I will build myself one…

17

yeah 16x is ideal, but iirc there is almost no detectable difference when running RT if your cards are in 8x or 16x slots, and you can happily plug an 8x (or 1x , 4x) card into a 16x slot. just like you can plug a 16x card into an 8x slot.. it just hangs out of the back.. pcie is good like that.

18

Actually, just realised I can’t go with any 1080ti’s on the mobo as even one would block a slot, so it’ll have to be 6 x 1080ti and a slngle slot gpu on the mobo…

19

why not 7x gtx1080 all on risers? you have 4 on shorter risers and 3 on longer ones..

you might have an issue with the ribbon cable of the riser obscuring the air intake on the side of the lower gpus, i dunno without seeing it all laid out.

20

consider a kill-a-watt meter…$20 from amazon to test wattage pull from the outlet…test on full gpu production test render to get a measured value, then add cards and psu’s as you see fit
https://www.amazon.com/dp/B00009MDBU/ref=asc\_df\_B00009MDBU5061719/?tag=hyprod-20&creative=394997&creativeASIN=B00009MDBU&linkCode=df0&hvadid=167125429392&hvpos=1o1&hvnetw=g&hvrand=18339390314219376247&hvpone=&hvptwo=&hvqmt=&hvdev=t&hvdvcmdl=&hvlocint=&hvlocphy=9031734&hvtargid=pla-306572288073

btw, I totally love have the immediate feedback using active shade when creating materials and setup lighting, camer effects. The production test renders at like 1600x1200 take a 2-5 minutes and 4k 30-45 minutes…a must have for animations