[tturgut]

Hi guys, can you guys help me with my build:

CPU: 2x AMD Epyc 7B12 (64 core, 2.25Ghz, 240 TDP)
Mobo: SUPERMICRO MBD-H12DSi-N6-O
Case: Fractal Design Meshify 2 XL
nvMe: SAMSUNG 980 PRO SSD 2TB PCIe NVMe Gen 4,MZ-V8P2T0B


1) ??? Coolers: (air preferred) 2x Noctua NH-U14S TR4-SP3 ???
Is the orientation correct?
TDP is not very clear will it handle it?
what else would be good?

2) ??? PSU: (will run a RTX 4090, or 2x RTX 3080's if it fits)
ATX 3.0?
1600w?
silver? titanium?
which company?

(computer will run 24/7, with both processors and videocard (s))

3) VRM fan : for air cooling do i need a fan?

thanks
tturgut@aol.com

[flotus1]

1) ??? Coolers: (air preferred) 2x Noctua NH-U14S TR4-SP3 ???
Is the orientation correct?
TDP is not very clear will it handle it?
what else would be good?

Noctua NH-U14S TR4-SP3 is pretty much the best air cooler here. No need to look for alternatives, they are perfectly capable of handling any CPU for this socket.

2) ??? PSU: (will run a RTX 4090, or 2x RTX 3080's if it fits)
ATX 3.0?
1600w?
silver? titanium?
which company?

ATX standards can be ignored. As long as you don't buy something that came out more than 10 years ago.
80Plus ratings only describe the efficiency of the PSU. Personally, I don't go lower than "Platinum" these days. With your PC running 24/7 with these components, an efficient PSU is probably a good investment.
As an example, Seasonic Prime PX-1300 1300W should work. Or any other quality PSU with similar or higher wattage.
Maybe look in the 1500W range if you really want to run dual RTX 3090.

3) VRM fan : for air cooling do i need a fan?

I think that's already discussed in detail in this thread.
With the components you intend to run on air cooling, there is no way around filling ALL possible fan mounts in this case and running these fans at fairly high rpm. That should create enough airflow to keep the CPU VMRs in check.

[tturgut]

Thanks!

The orientation of the noctuas look wrong.
Am I mistaken?

[flotus1]

Wrong in which sense?
On the H12DSI, their airflow is from bottom to top. That's fine in a workstation case that has exhaust fans at the top.
Server cases would be a different story.

[tturgut]

But the bottom will be saturated with the videocard. may not have enough space for good airflow.

So its not a problem to have a bottom to top (instead of front to back) air flow?

[flotus1]

That's more a general problem with air cooling. Exactly the reason why you need lots of case fans: creating enough air flow to keep the temperature inside the case low.

With a front-to-back orientation, the rear CPU gets the pre-heated air from the front CPU. Not necessarily a problem, all servers work this way. You just need higher rpm on the CPU coolers to offset that.
But...
front-to-back: air exhausts mostly through a single 140mm fan in the back
bottom-to-top: air exhausts mostly through the top, where you can have 2-3 140mm fans

With something in the order of 1000W of heat dumped into the case, you need a lot of airflow. Hence my recommendation to fill all fan mounts in this case:
- 2x140mm front intake
- 2x140mm bottom intake
- 1x140mm back exhaust
- 2-3x140mm top exhaust (the one towards the front of the case might not help too much)

[tturgut]

:-)
thanks.
1) Do you see any problems with my case?
Fractal Design Meshify 2 XL

2) 7V12 (2.40 Ghz, 280 TDP) ?
7B12 (2.25 Ghz, 240 TDP) ?

3) I think the 2 back 140 mm fans at the bottom will be blocked bu the GPU.
I was wondering if I should try to find a closed loop RTX 4090?

like this one: https://www.newegg.com/msi-geforce-r...82E16814137759

4) PSU:
https://www.newegg.com/corsair-hx150...quicklink=true

5) The other option is to put 2x3080's on air:
https://www.amazon.com/dp/B09QKL8XG5...v_ov_lig_dp_it

Not sure if it will fit/ cool enough to run 24/7 (I will use this for chess analysis)

thanks again!

[tturgut]

https://www.supermicro.com/en/produc...ard/H12DSi-NT6

this should be ok for 2x280 TDP CPU's correct? ( your messages above were saying 240 for H11DSI.

[flotus1]

Nah, the Meshify 2 XL is a good case.
The power supply you found will do the job.

Don't overthink stuff like "air flow paths" or "blocking airflow". That's something you can't avoid in a DIY computer, and it's not a big deal as long as you have enough powerful case fans.

The H12DSI officially supports CPUs with 280W TDP.
Whether the OEM-specific CPUs you have in mind will run on it, you should double-check before buying. It will probably work, but I just don't know for certain.

Whether you want air or water cooling comes down to how silent you need this thing. If you don't mind having 8x140mm fans running at 1200-1500 rpm, air cooling will work just fine. Water cooling will allow you to get quieter cooling. It's up to your preference.

Single vs. dual GPU...
I don't know much about the software you intend to run.
I'd imagine you will end up configuring the CPUs in NPS=1 mode, leaving you with 2 NUMA nodes total. In which case it might be beneficial to have one GPU for each NUMA node? If multi-GPU is even a thing? No idea, you would have to ask some experts for the chess program you intend to use.
Blower-style GPUs would also help, especially if you need more than 1. They are loud, but they don't dump the heat into the case. Instead, they exhaust it right out the back. There are RTX 3080s with this type of cooler, but no 4090s that I am aware of.

[the_phew]

Originally Posted by flotus1

Motherboard
Supermicros H11DSi boards officially support CPUs with up to 240W TDP. You might be able to run higher TDP CPUs, but only with special care towards cooling the CPU VRMs. I really would not recommend it.
Even if you remain within the specs, the VRMs will be difficult to cool in a workstation case.

We run a CFD cluster using those motherboards w/ EPYC 7542s in workstation cases (albeit in a cold server room), with water-cooled CPUs. My experiences:
-Water cooling was overkill; even under 100% load 24/7/365, CPU temps haven't even cracked 40C. At least the water cooling allows the CPUs to boost up near the max single-core boost on all 32 cores
-VRM temps haven't been an issue for us, despite the VRMs getting virtually no airflow. I think the highest I've seen is 70 C
-Memory temps have been our biggest cooling challenge; the DIMMs in the top slots creep above 80C

I only run LBM-VLES CFD on this cluster, so that may tax the memory more relative to the CPU/VRMs vs. Navier-Stokes solvers.

If I had to do it again, I'd go 4U rackmount just for better memory cooling, with better VRM cooling being icing on the cake.

[flotus1]

The problem with memory overheating also correlates with memory density, i.e. how much capacity each module has. 16GB DDR4 DIMMs are fairly easy to cool.
32GB and 64GB RDIMMs can overheat easily with the high amount of ICs crammed onto the PCB.
Water cooling for these CPUs is a double-edged sword. You don't need a lot of case airflow to keep the CPUs cool. Airflow which the memory modules would really benefit from. Plus the Noctua CPU coolers for SP3 tend to get a bit of cooling on the memory modules as well.
I agree that water cooling for the CPUs is not necessary. It just allows for quieter cooling.

Have you tried adding a bunch of case fans and and running them at max rpm? Should not be a noise problem in a server room. Which case are you using?

[tturgut]

thanks guys!!

I ordered 2x 7v12's from ebay for the Supermicro motherboard.
I will try aircooloing with Noctu N14 SP3.
PSU Corsair 1500w Hxi

GPU question:
Now, I need either 1 RTX 4090 or 2x RTX 3080's. (almost same total cuda cores)

AiO RTX 4090 is on black market and ridiculously expensive.
Turbo RTX 3080's may work, but will be too loud.

Would 2x 3080's with 3 fans each be OK? how thick are they, will they fit?
(blower style would fit, but too loud i guess)

what kind of strong GPU system can i run 24/7 without running into overheating problem? (not watercooled)

[the_phew]

Originally Posted by flotus1

Have you tried adding a bunch of case fans and and running them at max rpm? Should not be a noise problem in a server room. Which case are you using?

It's a proprietary case (with awful airflow for memory cooling, IMHO) from Titan Computers:


The reservoir blocks the top intake fan, so the top DIMM banks get nothing (especially the aft, top bank). I could zip-tie some case fans somewhere to impingement-cool the DIMMs, but I haven't seen any evidence of memory throttling or whatever. These are 64 GB 3200 MHz DIMMs.

[tturgut]

Fractal Design Meshify 2 XL

https://www.gamersnexus.net/hwreview...ge-case-review

i like the case.. (though I am not an expert in this)

[tturgut]

i am building this as a chess computer . I play correspondence chess as a hobby. :-)

it needs to run both alpha beta programs (CPU) and alpha zero programs (GPU)
Wont play any video games on it. :-(

It will need to run 24/7. so good cooling is very important.

I built a 2x E5 2699v3 system (overclocked at 3.25 Ghz) a couple years ago, and used Noctua's for the CPU's with very good result.

I wonder if i can put 2x 3080's and get away with it without causing the temps to go high...

[flotus1]

Originally Posted by the_phew

It's a proprietary case (with awful airflow for memory cooling, IMHO) from Titan Computers:
The reservoir blocks the top intake fan, so the top DIMM banks get nothing (especially the aft, top bank). I could zip-tie some case fans somewhere to impingement-cool the DIMMs, but I haven't seen any evidence of memory throttling or whatever. These are 64 GB 3200 MHz DIMMs.

Oh wow, that's rough. Seems like they managed to engineer a solution where no amount of fan speed would really help with memory cooling.

Originally Posted by tturgut

thanks guys!!
GPU question:
Now, I need either 1 RTX 4090 or 2x RTX 3080's. (almost same total cuda cores)

AiO RTX 4090 is on black market and ridiculously expensive.
Turbo RTX 3080's may work, but will be too loud.

Would 2x 3080's with 3 fans each be OK? how thick are they, will they fit?
(blower style would fit, but too loud i guess)

what kind of strong GPU system can i run 24/7 without running into overheating problem? (not watercooled)

Just to manage your expectations a bit: the system you are building won't be silent. 1000W can be air-cooled, but not without some fan noise.
Now everyone has a different threshold for what they consider a "silent enough" PC. Mine is pretty low, maybe yours is higher and this will be perfectly acceptable for you.

Looking at the block diagram in the motherboard manual, you probably want to use the top-most PCIe slot for the first GPU, and the second from the bottom for a second GPU. This way, each one is connected to a different CPU.
That leaves 4 slots for the first GPU. So a 3080 with the most common triple-slot axial fan cooler still has some room to breathe, which is good.
You just need to make sure there is enough clearance towards the bottom of the case to fit the second GPU.
If there isn't enough space to leave one slot empty between each GPU, blower-style coolers are the way to go. The radial fans skew towards higher static pressure, which is crucial to overcome the flow restriction. And the benefit of exhausting the heat right away instead of dumping it into the case still remains.

Long story short: there is no way get quiet air cooling for a multi-GPU setup that also has two CPUs with 280W TDP. Whether it is quiet enough for you depends on personal preference.

Edit: I moved this discussion to a separate thread

[tturgut]

can you guys also help me:
I bought this motherboard and also 2 EPYC 7002 processors.
i am planning using this computer for chess analysis.
I have a Fractal meshify XL2 case.
https://www.fractal-design.com/produ...empered-glass/
It will run a RTX 4090 for chess analysis also. (not for gaming)

But I would like to use this not as a server if possible.

1) Where do the front I/O's go exactly (not in the manual)
https://www.supermicro.com/manuals/m...0/MNL-2363.pdf

2) Are there any USB 2.0, USB3.0 or USB 3.1 connections directly on the motherboard (not on the rear I/O)
I found 1 only: JUSB1

3) Sound (HD Audio)? is there a connection?
4) i can deactivate the VGA I guess

thanks!

[flotus1]

The H12DSi is a server motherboard. So the answer to most questions about front I/O (and rear I/O) is unfortunately: no
It's got one USB 3.0 header for the front (JUSB1), and that's pretty much it aside from power/reset/etc.
It has no onboard audio, hence no front panel audio connectors either.
If you need any of those things, you have to add it via PCIe adapter cards.

Not quite sure what you mean with "deactivate the VGA"
In the bios, you can choose which graphics is used as a priority: offboard (a GPU in the PCIe slot) or onboard (the one connected to the VGA port on the rear). And I'm pretty sure there is a jumper somewhere that changes the default setting here.
What do you want to achieve?

By the way, I moved your post here.
You don't have to rely on finding similar threads to ask your questions. That's usually considered bad etiquette.
If you have a new question, just open a new thread. Or in this case, since the question still revolves around the same computer, add the question to your own thread.

[tturgut]

TPM module?
is this necessary for Windows 11 pro?

https://www.supermicro.com/en/produc...oard/H12DSi-N6

there are 3 versions of it.. do you have idea which one?

thanks

[flotus1]

Sorry, no idea about that.