Help Choosing New Gpu Rendering System?

[Donfarese]

I am looking at building a new GPU rendering workstation.

Right now I have a 3990x on a Asus ROG Zenith II Extreme Alpha with 3x RTX 3090. I like having the 64 cores for rendering with CPU renderers. But now we are mostly using GPU so I am looking at building systems better for that. So here are some choices I have and wanted to get your opinions.




1) Sell the 3990x and the Asus ROG Zenith II Extreme Alpha and replace it with:

CPU: Threadripper Pro 3995wx
Motherboard: Pro WS WRX80E-SAGE SE WIFI

This will enable me to keep the 64 core and now add up to 7 GPU's with it's 7x x16 slots.

Or


2) keep the 3990x and motherboard that can hold only 3x RTX 3090, and add a 3950x with 3x GPU's also:

CPU: AMD Ryzen 9 5950X 3.4 GHz 16-Core Processor
CPU Cooler: ARCTIC Liquid Freezer II 280 72.8 CFM Liquid CPU Cooler
Motherboard: Asus ROG Crosshair VIII Dark Hero ATX AM4 Motherboard
Memory: G.Skill Trident Z Neo 128 GB (4 x 32 GB) DDR4-3200 CL16 Memory

With this system I can have up to another 3x RTX 3090 but much faster single threaded performance. Some people have said that the Single threaded performance would really help with GPU rendering speed of compiling each frame compared to the 3990x, is that true that I will notice a big difference? Also working in 3dsmax they say is much faster with 3950x than 3990x cause of single threaded, is that true also? I was looking at getting the MSI Meg x570 Godlike motherboard because it's the only board with 4x Pci slots so I can fit 4x GPU's. But there is no extra power connectors for PCi-E slots on the board, so not sure if that would work or last with the GPU's pulling that much power through the PCI-E slots, you thoughts? I would definitely also overclock the 3950x since it can handle it no problem.

[Muhammed_Hamed]

Hello Donfarese

2) keep the 3990x and motherboard that can hold only 3x RTX 3090, and add a 3950x with 3x GPU's also:

CPU: AMD Ryzen 9 5950X 3.4 GHz 16-Core Processor
CPU Cooler: ARCTIC Liquid Freezer II 280 72.8 CFM Liquid CPU Cooler
Motherboard: Asus ROG Crosshair VIII Dark Hero ATX AM4 Motherboard
Memory: G.Skill Trident Z Neo 128 GB (4 x 32 GB) DDR4-3200 CL16 Memory

Keep in mind that this motherboard can only run 2 GPUs in x8/x8 and the third slot is from the X570 Chipset, running at x4 and this bandwidth is shared with other devices(Asus is not clear on that)
A better option is Asus Pro WS X570 ACE, which will run 3 GPUs at x8/x8/x8 (last one is from Chipset, but more reliable)
And MSI X570 Godlike which will run 3 GPUs at x8/x4/x4 (All from CPU) or 4 GPUs at x4/x4/x4/x4 (All from CPU)

So here are some choices I have and wanted to get your opinions

Option one is much better, I would go with that. You run every GPU at x16 PCIe Gen 4
Also you will not need to bother with Distributed rendering, and one single machine takes less space and consumes less power

A few things to keep in mind,

-You don't necessarily need to buy a 3995wx, if all what you do is GPU rendering. Could downgrade to PRO 3975WX with 32 cores and 64 threads, it will do a good job and costs 60% less
But don't get the 12 core or 16 core models, Vray GPU still uses a lot of CPU resources with bucket mode..

-You will not be able to fit 7 GPUs on the motherboard directly, not even 4 of them.. Maximum is 3 as most 3090s are 3 slots in width or more.
You will need to use Riser Cables and mount the GPUs to an external rack or one of those mining cases. Find some good Shielded PCIe Gen4 x16 riser cables (35 CM or more in length) like this
Or your other option is liquid cooling, and in this case you can fit the GPUs directly on the motherboard. (If you want more info on this let me know, I have done 7 and 8 GPU servers in the past)

- Running 7 GPUs are not gonna work out of box, will need to edit some bios settings.. enable "above 4G decoding" and might need to set the PCIe bandwidth manually, let me know as well if you run into issues

-You will not be able to use NVlink with this setup, could only use it with one pair of GPUs (driver limitation that Nvidia patched a while back) but there is still a way around this(let me know if you will be running NVlink)

Some people have said that the Single threaded performance would really help with GPU rendering speed of compiling each frame compared to the 3990x, is that true that I will notice a big difference? Also working in 3dsmax they say is much faster with 3950x than 3990x cause of single threaded, is that true also?

Yes, it is true. I guess you mean the 5950x in this case, it can reach over 5 GHz out of box on one or 2 cores. This CPU is an impressive peace of tech
This single threaded performance affects everything from m.2 drive performance to your GPU scores in benchmarks
Threadrippers will boost to around 4.3 or 4.4 GHz, it is not too bad either(I have both at home, I can tell the difference in single threaded tasks). In your specific case I would still go with Threadripper Pro and WS WRX80E-SAGE SE WIFI board
Keep in mind that x16 PCIe Gen 4 is future proof, while PCIe Gen 4 x4 is barely enough bandwidth for GPU rendering now specially with high-end cards like the 3090

But there is no extra power connectors for PCi-E slots on the board, so not sure if that would work or last with the GPU's pulling that much power through the PCI-E slots, you thoughts?

It is not really an issue as far as I know, plenty of people are using this board with 3 or 4 GPUs
You will need 1600 Watts PSU for 4x 3090s or dual power supplies 1000/1200 Watts

I would definitely also overclock the 3950x since it can handle it no problem

Mostly not needed talking about the 5950X, as you don't wanna lose on single-threaded performance. You can use PBO with curve optimizer, to improve on both single and multi-threaded performance.. but an all-core OC is hard to achieve and hurts your single-threaded performance

If you have more questions let me know

Best,

Muhammed

[Donfarese]

Thank you so much for all the help! The MSI x570 Godlike would be the Ideal for the 4x 3090, but I can't get past the x8/x4/x4/x4. A x16 compared to x8 doesn't really make a difference you will ever notice but I don't know about x4? The Asus Pro WS X570 ACE looks like a great board for x8/x8/x8, but it only has one m.2 x4, the other is x2 which is slow. I need to run two M.2 drives. Also are the two m.2 drives going to pull from the x8/x8/x8 making them even slower? Need to run 2x Samsung 850 pro 2TB, not in raid just so you know.

[Muhammed_Hamed]

would be the Ideal for the 4x 3090, but I can't get past the x8/x4/x4/x4. A x16 compared to x8 doesn't really make a difference you will ever notice but I don't know about x4?

This PICe bandwidth is relevant on moving the scene to your VRAM at the start of your render, but once everything is loaded it will not affect rendering speed(unless you are using Out Of Core) (I have tested and verified this with Vray GPU and other renderers)
Personal advice, don't run more than 3 GPUs on this platform.. and in this specific case I would go with X570 ACE Pro to run at x8, x8, x8 and one single M.2 drive
More than 3 GPUs or if you want 2 M.2 NVMe drives, you better be looking at Threadrippers

x4 vs x8 does make a difference yes, on moving the scene to VRAM at the start of the render.. and minimum for NVlink is x8 (tested this extensively as well)
Also this PCIe bandwidth is relevant with Chaos Vantage if you will be using that.
x4 from CPU is not too bad, but x4 from chipset is not gonna cut it (will be shared with bunch of other devices)

Also are the two m.2 drives going to pull from the x8/x8/x8 making them even slower?

the first x8, x8 are coming from the CPU and don't share any bandwidth .. they will be fine, the last slot is what shares the bandwidth with other devices (coming from Chipset) but at least for this board it is x8 (not x4 like all the other X570 boards)
I recommend a single M.2 NVMe

Best,

Muhammed

[Donfarese]

I guess I better just stick to threadripper then. I do need the 2x m.2, so.... And with that I can do 4x gpu or more, just really wanted better single threaded so I can work faster before rendering. I was thinking of the 3995wx, but it seems like the 5990x and 5995wx is right around the corner. I can just stick with my 3990x and get a different motherboard to support 4x gpu and then the 3990x is capable to overclock, I think the Silverstone IceGem 360 will enable me to of. But I have never even seen a review of it yet. Hopefully then I can get better single threaded out of the 3990x.

[Donfarese]

So here is my new idea, take my existing 3990x and Asus ROG Zenith II Extreme Alpha and replace just the motherboard to the GIGABYTE TRX40 AORUS Xtreme. This will enable me to put one more RTX 3090 into the system. Also remove the Noctua NH-U14S cooler, and replace it with the Silverstone IceGem 360 AIO Liquid Cooler, which should hopefully enable me to overclock the 3990x. I loved PBO mode on the ROG Zenith although I couldn't keep it on because of temps, hopefully the Gigabyte Aorus has just as good overclocking. Never used another board except for Asus, but they really let me down with sTRX4 boards with only being 3x GPU capable. So the System would be as follows:

CPU: AMD Threadripper 3990X
CPU Cooler: Silverstone IceGem 360, 360mm AIO
Motherboard: Gigabyte TRX40 AORUS XTREME
Memory: G.Skill Trident Z Neo 128 GB (4 x 32 GB) DDR4-3200 CL16 Memory
Storage: Samsung 980 Pro 2 TB
Storage: Samsung 980 Pro 2 TB
Video Card: EVGA GeForce RTX 3090
Video Card: EVGA GeForce RTX 3090
Video Card: EVGA GeForce RTX 3090
Video Card: EVGA GeForce RTX 3090
Power Supply: EVGA P2 1600 W 80+ Platinum Certified

The only issue I have is the 1600w is not enough so I need a second PSU and I have never hooked up dual PSU's so don't know how difficult it is.

I will then wait since the Threadripper 5000 series is coming out soon and build a second system then. Those should have better single threaded performance.

[francomanko]

is there much point in having such a powerful cpu in a GPU based render rig?

[Donfarese]

is there much point in having such a powerful cpu in a GPU based render rig?

No, but we already have this cpu, and we do use the system for CPU rendering also. But if it was strictly a GPU rendering rig and we needed to buy a new cpu, probably just go with a 3955wx.

[Muhammed_Hamed]

The only issue I have is the 1600w is not enough so I need a second PSU and I have never hooked up dual PSU's so don't know how difficult it is.

Should be very simple, look for a 24 pin splitter like this one
This is for your second Power supply to turn on/off whenever your system is on/off

You can also use a paper clip to short the 4th and 5th pin slots on the 24 pin cable on the second power supply, this tells it to turn on
like this

and replace just the motherboard to the GIGABYTE TRX40 AORUS Xtreme. This will enable me to put one more RTX 3090 into the system

You can buy 2 shielded riser cables and mount 2 GPUs externally, and 2 GPUs on the board directly
Zenith board can also equip quad GPUs, although the issue is the space between 2nd and third PCIe slots (which is shame)

Another cheap solution you can try is this

For each PCIe slot you can use 4 GPUs, all at x16 but with slightly higher latency. Many people rely on these exclusively for their render nodes
I mean you already have the GPUs, could buy it and test the performance..

Also remove the Noctua NH-U14S cooler, and replace it with the Silverstone IceGem 360 AIO Liquid Cooler, which should hopefully enable me to overclock the 3990x

This is gonna help with temps massively, at least this hot air is directed outside of the case..
On another note I think the single threaded performance of the the 3990wx is fine and workable out of box.. it is not like a 5950X but lets say we are comparing a Porsche to an F1 car .. the Porsche is still very fast and reliable, but the F1 is fastest
In my testing I can improve the multi-threaded performance massively with OC on my 3990x, but not single threaded performance.. You can still try

I loved PBO mode on the ROG Zenith although I couldn't keep it on because of temps, hopefully the Gigabyte Aorus has just as good overclocking

The same PBO controls are present on Aorus Extreme, but I honestly prefer Asus's
I'm a big Asus fan myself, and I think their bios and software is ahead of the rest

the Zenith Extreme is such a great board, you don't need to move to the Aorus extreme.. I hope my other suggestions will help in some way

I will then wait since the Threadripper 5000 series is coming out soon and build a second system then

There is big chance we will not see these on shelves until next Summer sadly, it also happened early with the 3990x there were massive delays
And the situation is worse now with the chip supply shortage, might need to find a long-term solution from now on

Best,

Muhammed

[Muhammed_Hamed]

And MSI X570 Godlike which will run 3 GPUs at x8/x4/x4 (All from CPU) or 4 GPUs at x4/x4/x4/x4 (All from CPU)

There is a mistake here,
4 GPUs will run at x8,x4,x4,x4 .. last one is from chipset and shares bandwidth with other devices

[Donfarese]

The only reason I'm thinking of switching the motherboard to Aorus is because you can fit 4x GPU right on the board, which will then fit in the case perfectly. Also the Front Panel Header gigabyte actually puts in a good place, on the Zenith II Extreme Alpha and other boards they put the Front Panel Header at the bottom right of the motherboard right where the GPU would sit, thus getting in the way of plugging in the bottom PCI-E slot with a GPU. I had to crush mine to just fit the 3rd GPU.

[Muhammed_Hamed]

I had to crush mine to just fit the 3rd GPU.

Yup, this always happens.. the board makers know about it and they accept it
The standard place for those panel connectors is like what the Zenith have.. most boards have it this way ..

The only reason I'm thinking of switching the motherboard to Aorus is because you can fit 4x GPU right on the board

You need to double check that, most 3090s nowadays are more than 2 slots in width. I don't know of any dual slot cards, but they might still exist
I personally recommend the beafy big cards to my clients.. because they have proper cooling for VRAM .. it usually stays within 70-80 degrees
Most other cards will have VRAM reach +100 degrees and in some examples up to 110 where it throttles (all Zotac, PNY and Gigabyte cards I tested)

I have stacked up 4 GPUs in the past, on the board directly.. used it with 980tis and 1080tis
The good old times where GPUs used to be dual slot, and blower style cards would be more than enough cooling for components

That being said I wouldn't stack 4x 3090s on the board directly, I would use Riser cables to make sure each card breathes and a rack for the GPUs.. so that the cards last for a while

This is just my take on it, Tear S of GPUs I tested so far are Rog Strix 3090 and MSI SuprimX 3090.. these are the best on the market right now
Then comes Asus TUF/MSI Gaming X Trio/EVGA FTW3 .. they all have proper cooling for VRAM

Almost everything else on the market will have VRAM at over 100 degrees .. out of box in a single card setup
Imagine stacking 4 of these together.. Check Memory Junction Temp. sensor in HWinfo
And feel free to ignore, some don't care about this throttling much..

Best,

Muhammed

[Donfarese]

You need to double check that, most 3090s nowadays are more than 2 slots in width. I don't know of any dual slot cards, but they might still exist
I personally recommend the beafy big cards to my clients.. because they have proper cooling for VRAM .. it usually stays within 70-80 degrees
Most other cards will have VRAM reach +100 degrees and in some examples up to 110 where it throttles (all Zotac, PNY and Gigabyte cards I tested)

I have stacked up 4 GPUs in the past, on the board directly.. used it with 980tis and 1080tis
The good old times where GPUs used to be dual slot, and blower style cards would be more than enough cooling for components

That being said I wouldn't stack 4x 3090s on the board directly, I would use Riser cables to make sure each card breathes and a rack for the GPUs.. so that the cards last for a while

This is just my take on it, Tear S of GPUs I tested so far are Rog Strix 3090 and MSI SuprimX 3090.. these are the best on the market right now
Then comes Asus TUF/MSI Gaming X Trio/EVGA FTW3 .. they all have proper cooling for VRAM

Almost everything else on the market will have VRAM at over 100 degrees .. out of box in a single card setup
Imagine stacking 4 of these together.. Check Memory Junction Temp. sensor in HWinfo
And feel free to ignore, some don't care about this throttling much..

Best,

Muhammed[/SIZE]

I'm big on cooling and use only dual slot Evga RTX 3090 FTW3 Ultra Hybrids. They stack fine. I also only use Horizontal motherboard cases, This way heat doesn't raise through the cards but up and away. Blowers are great also but they will turn your office into a sauna in the matter of minutes.