advices, thoughts and ideas on new custom rig 7950x + 2x4090

[filip]

Hello,

I am considering upgrading from my old 2990wx to the fresh am5 pcie5 platform.
Not all parts are available yet.

Meanwhile, plans only.

New RTX cards are even bigger, the best distance between the pcie slots have top models of the ASUS mobos only. (16x or 8x and 8x)
Still, if I put two new GPUs there, I will have only 1cm spacing for some airflow.
I am looking for some cases where I can set two cards on risers (currently have this setup inside the TT P8 case with two 3090) but with these new bricks it looks even more tricky.

Muhammed posted here some pictures of his vray mining rig - that's quite an inspiration.

MOBO: asus x670e hero - main reason is their SW, I would like to try some OC, but don't have much experience so would like to try their AI overclocking
CPU: 7950X
GPU: asus rog strix OC 4090RTX - but will see if benchmarks are promising, if not I will use my older asus strix OC 3090RTX
RAM: total 128, some low timing as 30 and 6000Mhz - will see the options when there are some AMD EXPO 2x32 sticks available (G.skill DDR5-6000 30-40-40-96 2x32GB - not available yet)
SSD: 4TB pcie 5 - again when available and if the benchmarks will be promising (MSI SPATIUM PCIe 5.0 SSD?)
PSU: corsair AX 1600i not sure if to use the adapters that should come with the new gpu or the new cable that corsair offers extra (600W PCIe 5.0 12VHPWR). Is the 1600W PSU capacity enough? Or is there some ATX 3.0 PSU with enough capacity available already?
Cooling: ARCTIC Liquid Freezer II 420
Case: custom alu profiles and plexiglass. It is going to hang on the wall, with all connectors facing down. Does this idea have any potential? attached some early plans:

[Muhammed_Hamed]

Hello,
New RTX cards are even bigger, the best distance between the pcie slots have top models of the ASUS mobos only. (16x or 8x and 8x)
Still, if I put two new GPUs there, I will have only 1cm spacing for some airflow.

Hi filip

Checkout the X670 ProArt motherboard, you will use the first and third PCIe slots to have good spacing between 2 4090s without riser cables. This spacing is gonna work well with some good case fans and airflow
You will need to adjust the bandwidth for the 3rd PCIe slot from Bios, you will set it to X16 mode and you will not be able to use a second M.2 NVMe

I am looking for some cases where I can set two cards on risers (currently have this setup inside the TT P8 case with two 3090) but with these new bricks it looks even more tricky.

Not many options sadly, you are better with a mining rack like what I used. You can do 3x 4090s with an X670 ProArt Motherboard
Or 2 cards without riser cables, you still need a massive case. the ROG Helios case is a good contender with a really good GPU airflow(triple front fans and one back fan) this is what I use for my other machine, see here and here

This board has a similar layout to the X670 ProArt, you can see there is enough spacing between the GPUs. You will need to remove the PSU cover completely

If there is time I can make some research on similar cases, I will post here as well

Muhammed posted here some pictures of his vray mining rig - that's quite an inspiration.

Rendering rig actually, I never used it for mining. It is way too expensive to be used for mining.
For mining you would use cheap riser cables and cheap Motherboard and CPU

I would like to try some OC, but don't have much experience so would like to try their AI overclocking

No need, this CPU comes already with all cores at 5 GHz at full load, which is short of a miracle to exist
You will be thermally limited as well, don't waste time on overclocking... Any automatic OC will not work well, you will need manual tuning to make this worth(takes a lot of time and effort)

PSU: corsair AX 1600i not sure if to use the adapters that should come with the new gpu or the new cable that corsair offers extra (600W PCIe 5.0 12VHPWR). Is the 1600W PSU capacity enough? Or is there some ATX 3.0 PSU with enough capacity available already?

This is a sensitive topic, you need an ATX 3.0 power supply for sure. Adapters are not gonna work well
So you need native PCIe gen 5 cables for both cards. 1600 watts should be more than enough for 2 GPUs, or maybe 2 power supplies
I don't know when exactly these power supplies will be available

Case: custom alu profiles and plexiglass. It is going to hang on the wall, with all connectors facing down. Does this idea have any potential? attached some early plans:

Looks really good, I would love to hear more about this, I'm a bit skeptical about the use of glass here. The GPU will eject hot air through the back IO and towards the top, if you use glass you will need to carefully plan your airflow. Or don't use glass like what I did, with a cleanup every month or so


Best,
Muhammed

[filip]

Hello Muhammed and thanks for the reply!

-proart sounds like a good advice for the case if I want to go with standart XL box. I am already counting on with removing the PSU shrout or even moving the PSU somewhere else (see my current garage junk built in TT p8 in attached image). But now I am leaning more towards trying my own pc shell. Lets see...
-if I am on GPU risers and AM5 should be 5years lasting platform so for future upgrades still the hero is probably better choice (not big gap in price as well comparing to proart) - your opinion? Regrdless of OC, I want to stick with asus as some people have good reviews about their bios/software maturity.
-regarding to PSU I see in your photo you have asus rog thor PSU right?- is not it the one that should already come with the new cables and be pcie5.0 ready (not available here yet)? There are slowly emergin new ATX 3.0 PSUs but all of them are 1000w-1200w max. So far nvidia and corsair stated that current PSUs should be ok with the new 600w cable (not adapter). Anyway I am still waiting for the gen5 ssd so no rush to buy this.
-SSD - Do you have any info, if the gen5 ssds will be available soon? First I have heard, they will come alongside with the am5 platform, now heard some rumors that in november... Performance wise - do you think will there be some loading heavy scenes/compiling differences in GPU vray workflow comparing to Gen4?
-RAM - I already found some AMD EXPO 2x32 sticks from corsair, but best is 40-40-40-77 5600mhz. Do you think will be there big preformance difference in comparison with upcoming g.skill DDR5-6000 30-40-40-96 2x32GB? AMD says 6000Mhz is sweet spot - but for 2 stick only, not for 4 at 128gb together...
-with the box/shell I am now on the "meshify" version, construction covered with the anti pollen fabric (to windows for alergic people) hold on velcro stripes. Now looking for some external panel/ button /switch with audio jack and 2 usb at least. Aliexpress only so far;(

cheers,
Filip

[filip]

....progress so far, that was actually quite easy to setup and very cheap, the construction is very light. But still nothing to put inside;(

[alen_agic]

If I may ask, why 2x 4090 ? Don't see a point now when 4090 doesn't have nvlink. I would rather wait until early next year and pick up 1 RTX A6000 Ada 48gb, it is full fat AD102 with more CUDA, Tensor, RT cores than 4090 rated for 300 W TGP, and you can put 2 of these in a normal pc case
I already have 7950x, if you want 128gb of ram you would be lucky to run it at 4800mhz or 5200 and be stable at all, on default it will run 128gb at 3600. My ram is 64gb (2x32gb) gskill 5600 cl30 and not stable, I'm running it now at 5200 to be stable. So don't waste money for high-speed ram, if you want the capacity.
Don't try to overclock 7950x at all, not worth it, in fact, go and undervolt, Thermal limit Level 1 on Asus board will give you a few % uplift and drop temps 5c.
X670E have overkill VRMs, you don't need anything better than x670e ProArt or x670e E-Gaming if you want a board for 2gpus. These boards have more and better VRMs than X399 boards that power Threadripper CPUs.

Btw, nice case design
Good luck

[filip]

Thanks Alen for your reply!
-I already have two 3090 with the nvlink, that I finally disconected, all my scenes, in my opinion heavy with forestpack and railcone usualy use around 14gb gpu ram (I dont use displacement). One frame takes from 7 to 15mins, so my intention is to drop the render time only.
-already ordered 6000/cl30/amd expo g.skil - two 64gb kits - lets see, all parts should arrive during the next week.
-decided for the proart and maybe will use the third slot for one of my 3090, I just realized that the third slot is max pcie 4.0 x2 (not x4) - at least it is stated in the manual. Is it good idea to use this slot for the card with three monitors connected? or better for the render card only?
-for the gpu I ordered two of the cheapest 4090 finally - gainward not oc version and will try to power limit it (Improving Nvidia RTX 4090 Efficiency Through Power Limiting | Tom's Hardware (tomshardware.com)). All current melting cable photos make me scary
-and thanks for the advice - undervolting sounds as more clever way with this cpu as well.

cheers,
Filip

[alen_agic]

Thanks Alen for your reply!
-I already have two 3090 with the nvlink, that I finally disconected, all my scenes, in my opinion heavy with forestpack and railcone usualy use around 14gb gpu ram (I dont use displacement). One frame takes from 7 to 15mins, so my intention is to drop the render time only.
-already ordered 6000/cl30/amd expo g.skil - two 64gb kits - lets see, all parts should arrive during the next week.
-decided for the proart and maybe will use the third slot for one of my 3090, I just realized that the third slot is max pcie 4.0 x2 (not x4) - at least it is stated in the manual. Is it good idea to use this slot for the card with three monitors connected? or better for the render card only?
-for the gpu I ordered two of the cheapest 4090 finally - Gainward not oc version and will try to power limit it (Improving Nvidia RTX 4090 Efficiency Through Power Limiting | Tom's Hardware (tomshardware.com)). All current melting cable photos make me scary
-and thanks for the advice - undervolting sounds as more clever way with this cpu as well.

cheers,
Filip

Hi Filip,

If your scenes don't go over 24gb of vram you are good, 4090 is a beast, I saw benchmarks for Vray on Puget System that 1x4090 is fast as 2x3090ti and faster than 2x3090. It is crazy how far we got. But for current prices of 4090 I personally would still wait for RTX A6000 Ada that will come out in December most likely if the price would be around 5000-6000 cuz my projects can't fit on 24gb of VRAM.
About ram, I'm sure that you will not be able to run those ram stick near 6000mhz sadly, just few hours I saw guy on reddit with 7950x and 128gb running at 5200mhz which is great for 128gb, but he didn't do any mem test/stress to check how stable it is. On default without EXPO ram was running at 4800mhz and his kits are rated for 5600mhz. Don't know a lot about this but I think first problem is the memory controller on cpu and second problem is ram sticks, currently 32gb sticks are DS (double side), maybe later when there is 32gb SS sticks they should be able to run faster.
ProArt is great great board I wish that I also picked this one. But I'm not sure about runinng 3 gpu on this platform at all, I think last pci slot and bottom one or two m2 slots are lanes from the chipset and not from cpu. Honestly don't know, but it's unlikely that you can run 3xgpu. But now when I'm thinking about it, cuz ProArt have Thunderbolt 4 you could run external gpu with it. Something like this https://www.razer.com/gaming-egpus/razer-core-x
Regarding cables, If you are not planing to buy new ATX 3.0 PCI 5.0 PSU get some quality 12vhpwr cables , there is some from Cablemod and Corsair already, if you want new PSU, go with new MSI ATX 3.0 psu or wait for Seasonic Vertex ATX 3.0
Gainward non OC models are limited to 450W, I don't think that you will peak at that W while rendering anyway, so you are good.

All the best, show us once you complete your build.

[filip]

Hi Alen,
finally got the most important parts, but still cannot get the GPUs (having old 2x3090 for now) - I really dont want to pay the extra money to the bloody scalpers and still waiting for the new gen5 SSD.
I can run 2x32 memories at 6000Mhz, with the 4x32 after 6min first post I have 4800Mhz with windows startup, but then the next boot get stuck forewer, only way back is CMOS reset, tried many times with different settings, so now I am on the two sticks only.
Tired, after two nights installing and testing but at this moment happy with the fast and stable system in comparison to an old 2990wx and even the temperatures are in my opinion good. Full load on cuda with two GPUs and CPU the max Wattage PSU draw is around 1000W only (corsair usb PSU monitoring)!
Found really cool way how to turn an old smartphone to monitoring LCD trigonesoft is worth for donation!

[Muhammed_Hamed]

Thanks for sharing filip, the build looks really nice!
Let us know when you receive your 4090s

Did you design/build the rack yourself? Great job as well

I can run 2x32 memories at 6000Mhz, with the 4x32 after 6min first post I have 4800Mhz with windows startup

Wait for Bios updates, this is normal to happen on new platforms. Even on Threadripper maxing out all RAM slots used to cause instability and issues
If you have more time you might wanna tune the timings manually

About the ProArt X670 board, you need to consider that this board doesn't support NVlink(if you plan to use 2x 3090s)
On another note, the third PCIe slot runs at x2 Gen 4 and shares bandwidth with other devices. So a better option is the X670E Gaming board, which has the third PCIe slot running at x4 Gen 4

Is it good idea to use this slot for the card with three monitors connected? or better for the render card only?

I would run the monitors on this card personally, and leave the other 2 lanes with higher bandwidth to GPU rendering

And yes it is possible to run 3 GPUs on both motherboards, ProArt and Gaming E

Best,
Muhammed

[Muhammed_Hamed]

I would rather wait until early next year and pick up 1 RTX A6000 Ada 48gb, it is full fat AD102 with more CUDA, Tensor, RT cores than 4090 rated for 300 W TGP, and you can put 2 of these in a normal pc case

The RTX Ada 6000 will not be faster than a 4090, running 2 GPUs they will get very hot. You will need enough spacing between them, and good airflow

Best,
Muhammed

[filip]

Hi Muhammed,
it is my design, but first inspiration came from your rig photo It is not precise, not 100% right angles and finally more holes that I need, coz during the adding the parts I realized there is need for more clearance for the cables. But works ok so far and when the fabric get dusty I can wash it...
Was able to run all the memory sticks finally, the issue was on my side - still amateur. All sticks at 4800 now, sometimes getting crash but it is rare and the same happens on lower settings as well.
Testing with the one card only now, connected to PCIEX16_3. In the bios there is setting to set the gen1234 only. Nothing to specify x0 x2 or x4 - all seems to be automatic. Now, even the GPU is placed in PCIEX16_3 x2 all seems to run OK - I cannot see any difference so far.
The new cards should arrive tomorrow. Gainward phantom has horrible PCB and is one of the cheapest, still hope that without OC it will run OK and wont melt, corsairs cables still not available so will use the adapters.
Just for the curiosity will test the setup 2x4090+1x3090 (PCIEX16_3x2) for the display. PSU is 1600w only, hope it wont break any part in the build...

cheers, F

[filip]

last parts added,
-RAM at 4400 for 100% stability (can go a bit up probably), AI OC set from performance to optimized, all is very stable
-GPU finally OC GS version arrived (ordered non OC...), not enough PCIE cables from PSU so each GPU use one PCIE double split cable and rest is normal
-running all 3 GPUs the system wattage never go above 1100w (approx 1hr rendering)
-all temperatures 60-65 during the long load
-vray benchmarks little above average
hope no more problems and crashes and hope this is end of the story - thanks for the help!

[Muhammed_Hamed]

Looks pretty good!

Nicely done filip, congratulations!
I'm building mine later this month, based on the same motherboard and CPU. Lets hope bios updates will improve memory stability

Best,
Muhammed

[filip]

Hi Muhammed,

now at 4400Mhz and it is 100% stable, 4600 had rare crash, 4800 more often and 5000 right after windows boot. Looking forward for your results - let me know when you have all setup finished, hopefully I will learn some better settings as you are the 3D gpu guru here. Thanks! F

[davius3d]

Hey filip! Uber nice setup! I'd love to see your system benchmarked on some scene evaluation jobs in 3dsMax.Could you run your system through some 3dsMax scenes we have prepared?

https://www.youtube.com/watch?v=mrTzHKefLJI

Cheers!