[cbooks77]

Hi everyone! I am interested in building a dual EPYC 7H12 64 core ea. machine on a H11dsi-nt (rev 2.0) board. My main concern here is thermals. Even with liquid cooling which I intend to use I have doubts that it will be sufficient to cool such monster power draw (~280 TDP ea. cpu).

Does anyone here have experience with such a build and would building a 96 core machine be a better option on that board so maybe a dual 7642 Rome machine or even a dual 32 core 7532 machine. Any recommendations are deeply appreciated!

I will be running Ansys Fluent with 3 HPC packs and I'd like to be able to utilize all 128 cores for larger jobs (upwards of 50 million cells). I will mostly be running smaller jobs around 5-10 million cells but I am running multiphase with population balance coupling so around 25-30 PDEs are being solved and its very expensive. In this case we are dealing with lots of large matrix operations so would clock speed even matter?

My intention was to purchase one of these for cheap https://www.ebay.com/itm/25560332929...kAAOSwjfxiug8W
and then pull the 7252's out and swap them for the 48 or 64 core cpus and then retrofit a larger liquid cooling system into it. The reason I'm doing it this way is because I am inexperienced in PC builds and getting a fully working system then slowly upgrading made the most sense for me. This way I can't screw things up too badly as I am just pulling and replacing parts essentially

Also I found these 7H12s for cheap on ebay but they are from China which concerns me anyone see a problem - seller seems reputable. https://www.ebay.com/itm/38485741975...AAAOSwhzhhz97O

Thank you for reading

[flotus1]

I will write a more detailed answer later, but for now just one thing: I would not trust the ebay seller you linked for the 7H12 CPUs. It looks way too sketchy. Tons of positive feedback from a single account over the last few weeks, all on private listings.

[cbooks77]

Thanks looking forward to your response!

I am suspicious too. There are a ton of 2nd gen EPYC selling for real cheap on ebay. The thing is I have 30 days for full return and refund so I could easily inspect the chips and have them tested by an IT specialist to make sure they weren't delidded or engineering samples etc.

[flotus1]

Motherboard
Supermicros H11DSi boards officially support CPUs with up to 240W TDP. You might be able to run higher TDP CPUs, but only with special care towards cooling the CPU VRMs. I really would not recommend it.
Even if you remain within the specs, the VRMs will be difficult to cool in a workstation case. These boards are intended for server applications with plenty of airflow, you can't do that in a workstation while keeping it somewhat quiet.
On the H11DSi in my personal workstation, I used water cooling for the CPU VRMs aswell, and there are plenty of reports from people online with thermal throttling on these boards thanks to VRMs overheating.
Alternatives are the H12DSi, or Gigabyte MZ72-HB0. I used the latter recently, it's a solid option for workstations.

CPU choice / core count
It may be tempting to go for 64 cores per CPU, but it has been shown time and time again that there is barely any scaling happening on these CPUs with more than 32 threads. You can of course still use the 64-core options if you find them at a similar price as the 32-core variants. Just don't expect to get much additional performance.
Same story for CPU clock speed: since the memory subsystem is the limiting factor here, you don't get much return from higher clock speeds.
What does help is large L3 caches. So if you choose 32-core or 48-core options, those with 256MB of L3 cache are definitely better.
Unless you have definitive proof that your particular models scale beyond 4 cores per memory channel, spending more on high core counts/clock speed is not worth it. You can test scaling behavior on the hardware you have now.

If you want to make full use of your HPC licenses, adding a second workstation with another 64 cores is the better option. You can probably get away with using 10G Ethernet for the interconnect. If that's not enough to get good scaling, you can still invest in two used Infiniband adapters.
See also General recommendations for CFD hardware [WIP]

DIY+prebuilt+watercooling+lack of experience
That's a tough one.
Jumping straight into the deep end with watercooling expensive enterprise gear is daring. I am not trying to discourage you, it definitely can be done. And there are forums where you can get all the help you need for planning and execution. Servthehome for example has an enthusiastic bunch of people with expertise in this area. Just be prepared to spend quite some time learning and planning.
If you want to go down that route, I don't think a prebuilt workstation is the best starting point though. You will quickly run into limitations caused by the component selection of said computer. The case in particular is important for water cooling. If it doesn't have many options for radiator and pump/reservoir mounting, you will have a harder time than necessary. And if you want to stay with air cooling, you need one with very good ventilation for cooling these high-power components. The case shown in their listing does not look promising in that regard. A good case just makes things much easier.
Also, I can't see which motherboard comes with it. And the PSU also seems to be unknown, which can come back to bite you when installing more powerful CPUs.

A note on water cooling these CPUs in general
They can be air-cooled fairly quietly if you plan ahead with the right case (e.g. Phanteks Enthoo Pro 2), enough case fans and Noctua NH-U14S TR4-SP3 CPU coolers. The high TDP and power draw is not a huge issue, since the heat is spread out over a large area, unlike current-gen desktop CPUs.
If you watercool the CPUs, you have to do something about the CPU VRMs. Either by attaching additional fans to the pre-installed heatsink, or by going all-in with a waterblock for the VRMs themselves.

[cbooks77]

Thank you for your detailed and thoughtful response. I had read up on some other posts and saw you recommended the Phanteks Enthoo pro. My intention is to take the internals of the Velocity micro workstation (so power supply and H11dsi-nt etc.) and transplanting it into this case. The reason I'm doing this is because I need a workstation immediately but in the meantime I can source all the components for the Phanteks build. https://www.newegg.com/black-phantek...VXK8cu1xVkPeBA

I was then planning on adding in two 360mm radiators with the Enermax liqtech version 2 (supposedly this fixed the issues with version 1) https://www.amazon.com/Enermax-ELC-L.../dp/B07H778NCW

I think that might be sufficient to cool the CPUs. May I ask how you achieved watercooling on the VRMs ? Looking at the H11 board it appears there is a heatsink installed on it already but the fins are perpendicular to the CPU sockets which is awkward for airflow especially if using a Noctua type cooler.

Do you think the H11 board can handle dual 48 core CPUs or would the H12 be a better option or perhaps another MB entirely. I know we sadly don't have too many options for MBs.

[flotus1]

I tend to recommend the Phanteks case because it is the ideal fit for aircooling a dual-socket SP3 workstation. It will also work for watercooling, it's a pretty solid case after all. But you have plenty more options to choose from if you want to do watercooling anyway. So feel free to pick whatever fits your radiator plans.

Two closed loop liquid coolers instead of a custom loop will also work. I don't know if these Enermax are particularly good or bad, but they will probably get the job done.

May I ask how you achieved watercooling on the VRMs ? Looking at the H11 board it appears there is a heatsink installed on it already but the fins are perpendicular to the CPU sockets which is awkward for airflow especially if using a Noctua type cooler.

1.jpeg
It's a small "universal" water block specifically for VRMs. The original heatsink is held in place with two pushpins, and easily removed. I sourced this from a company that seems to have gone out of business by now, so you would have to look for similar alternatives. There are a few, but you have to get the dimensions just right since there is not much wiggle room.
The original VRM heatsink is designed for strong front-to back airflow as in almost all server cases. Good airflow in a workstation case can be enough, but this remains the critical component to cool on the H11DSi with high TDP CPUs. Plenty of DIY builds ran into this problem. If you put water coolers on the CPUs, there is plenty of room to just put a fan on top of the original heatsink, which keeps temperatures below throttling territory. Zip-ties are your friend here
And in case of aircooling with Noctuas NH-U14S: there is just enough room to wedge two 40mm fans between the coolers and the VRM heatsink, which also helps a bit.

Do you think the H11 board can handle dual 48 core CPUs or would the H12 be a better option or perhaps another MB entirely. I know we sadly don't have too many options for MBs.

If you go to the H11DSi product page, it clearly states CPU support up to 240W TDP. Whether this means that the board won't even post with higher TDP CPUs, or you will just have to deal with the overheating CPU VRMs, I can't say. I would rather try to avoid this issue entirely.
Your 2 alternatives are:
Supermicro H12DSi: technically supports all the highest-end CPUs and also Milan+Milan-X. But given Supermicros track record of undersizing VRMs and their cooling, you will still have to put at least a fan on it.
Gigabyte MZ72-HB0: in my opinion the better choice for a workstation. It has some "neat" features like a built-in fan control setup, and the VRMs and their cooling aren't as undersized as on Supermicro boards. I recently got my hands on one, and from a DIY end-user perspective, it is just better than what Supermicro offers in almost every way.

[cbooks77]

Thanks so much for sharing some of your experience and wisdom! Super appreciated. I think I have a clear path forward now!