[Mr Jordan]

Hello there,

Im following this forum for quite some time as im always interested in hardware development, esp. for CFD performance. Now I managed to get funds (around ~5k€) for a CFD server which I will set up myself. I have a few questions about my current setup. I plan to use OpenFOAM with Ubuntu or Centos. Because of the infos provided in the openfoam-benchmark-thread (sticky) I decided to go for a 2 x AMD Epyc server.
This is what I got so far (with added comments/questions):

MB:
Supermicro H11DSi
Nothing to mention here because simply the only option for dual Epyc setup

CPUs:
2 x AMD Epyc 7301
Best bang for the buck (and within my budget) / price-performance-ratio for CFD applications

GPU:
MSI GeForce GTX 1060
Sufficient performance

SSD:
Samsung SSD 970 EVO 1TB, M.2
Am I right the motherboard does support one M.2 slot? I’m planning to install centos/Ubuntu on the M.2 SSD assuming I spend most of the time in this environment. If I want to install windows on separate ssd i would require a SATA-SSD (meaning no second M.2-ssd possible) ?!

PSU:
any platinum 650W? or 850W? PSU
I quickly estimated max workload from CPU+GPU is about: 2 x 170W (dual Epyc) + 120W (1060GTX) = 460W.
With the 650W PSU I would have 190W for all remaining components combined. Is that enough or do I really need an 850W PSU.
(as far as I know the sweet spot for power efficiency is reached while running at around 80-90% of the psu's max load)
Also: Does the PSU need to have ATX 2.4?

Tower:
Fractal Design Define XL R2
Mentioned by flotus1, so I assume everything fits in this case. Personally I know and like the silent case: Fractal Design Define R6 but this one ist too small for the MB right?!

RAM:
16 x Samsung DIMM 8GB, DDR4-2666, CL19-19-19, reg ECC
maybe 16x16, if fits in budget
This RAM is dual rank + 2666MHz, both tested to be positive for CFD performance (max. memory bandwidth),
Do I definitely need reg ECC memory?
(As non ECC RAM is cheaper/better timings at 2666MHz)

CPU-Fans: 2 x Noctua NH-U12S
Im a big fan of Noctua procduts. This fan fits on the CPUs and provides enough cooling power while being relatively silent as the server will be positioned inside the office room.


Case-Fans: 2-3 x Noctua NF-A14
Silent + enough air throughput

Is something missing/wrong on the hardware side?

Also: would you recommend Ubuntu 18.04 or CentOS 7 aiming for best performance for OpenFOAM?


Kind regards
M J

[flotus1]

SSD:
Samsung SSD 970 EVO 1TB, M.2
Am I right the motherboard does support one M.2 slot? I’m planning to install centos/Ubuntu on the M.2 SSD assuming I spend most of the time in this environment. If I want to install windows on separate ssd i would require a SATA-SSD (meaning no second M.2-ssd possible) ?!

Be aware that the onboard m.2 slot on this motherboard is only connected via 2 PCIe lanes. It makes absolutely no sense, but that is what Supermicro decided to do. Hence your super-fast SSD will be limited by the transfer speed of the interface.
One possible solution would be to use a carrier card for a normal PCIe slot that can then hold the m.2 SSDs. IIRC, the motherboard supports PCIe bifurcation which would be needed if you want more than one SSD per carrier card.
https://geizhals.eu/raidsonic-icy-bo...loc=at&hloc=de
https://geizhals.eu/supermicro-aoc-s...loc=at&hloc=de
https://geizhals.eu/asrock-ultra-qua...loc=at&hloc=de
Or simply use SSDs with a PCIe form factor, but they can be expensive.

PSU:
any platinum 650W? or 850W? PSU
I quickly estimated max workload from CPU+GPU is about: 2 x 170W (dual Epyc) + 120W (1060GTX) = 460W.
With the 650W PSU I would have 190W for all remaining components combined. Is that enough or do I really need an 850W PSU.
(as far as I know the sweet spot for power efficiency is reached while running at around 80-90% of the psu's max load)
Also: Does the PSU need to have ATX 2.4?

650W will be enough for this setup. Make sure your PSU has 2 CPU 8-Pin connectors, not all of them have it. The difference in efficiency between running at 50% or 80% load is negligible for most high-quality platinum or titanium PSUs. Don't worry about ATX standards as long as you get one that is not really ancient. I am still happy with my Seasonic Focus Plus Platinum 750W.

Tower:
Fractal Design Define XL R2
Mentioned by flotus1, so I assume everything fits in this case. Personally I know and like the silent case: Fractal Design Define R6 but this one ist too small for the MB right?!

Define R6 is too small for this board. For the Define XL R2, you will have to drill additional holes for 3 of the motherboard standoffs. I switched to Phanteks Enthoo Pro recently which requires no modifications for this motherboard and is very quiet as soon as you swap out the shitty included fans.

Do I definitely need reg ECC memory?

Yes, definitely.

CPU-Fans: 2 x Noctua NH-U12S

Make sure to get the TR4/SP3 variant of these coolers. The standard one won't fit.
I personally find the fans on this cooler a little loud by design and opted for the only slightly more expensive NH-U14s TR4/SP3.

Case-Fans: 2-3 x Noctua NF-A14

Good choice. Be prepared to change IPMI thresholds for these slow-spinning fans: https://calvin.me/quick-how-to-decre...fan-threshold/
This worked for me

ipmitool -I lan -U ADMIN -H 100.101.102.103 sensor thresh FANB lower 100 150 200

repeated for all fans 1-6+A+B and of course changing the IP address to whatever your machine gets.

Is something missing/wrong on the hardware side?

Maybe some local HDD storage? If you store all your data externally, maybe the NT version of the motherboard is worth it since it has built-in 10G Ethernet.

Also: would you recommend Ubuntu 18.04 or CentOS 7 aiming for best performance for OpenFOAM?

I have no objections here as long as you use a recent kernel >=4.12

[Mr Jordan]

Thank you flotus1!

Very detailed knowledge. I will make all recommended adjustments.

[codygo]

How do you determine that ECC memory is critical?

I plan on building a similar system using a single Epyc 7351 and 8 x 16GB ram, to later install a second processor and double the ram.

[flotus1]

You don't necessarily need ECC, but the system will not post with UDIMM. So you definitely need registered DIMMs which are very hard to come by without ECC capability. And ECC doesn't hurt, especially with that amount of RAM.

[Mr Jordan]

hi
unfortunately i still have a specific problem with my setup concerning the bifurcation of x8 PCIe slot into x4x4 with an addoncard.

i chose the Supermicro addon card AOC-SLG3-2M2 (despite the fact that my mainboard H11DSi is not in the list of supported boards for this addon card)



as flotus1 mentioned the mainboard supports bifurcation. In the BIOS i can choose bifurction options for every specific PCIe slot between AUTO and x4x4. I tested several x8 slots aswell as several bios settings but the server wont power on in any case with both ssds inserted in the addoncard. In a support forum for other (Intel based) Supermicro mainboards I read about the bios option OPROM -> EFI/UEFI has to be set to get bifurcation to work. Also for some mainboards only PCIe slot5 seems to be working. I tried all of these option without success.


With onlyone ssd populated in slot1 of the AOC the server powers on but the ssd is not recognized as a bootable medium.
Edit: Sorry this isnt right. The computer powers on with another 1slot AOC. With the 2slot AOC-SLG3-2M2 installed (with 1 or 2 ssds populated) it wont power on at all.

Has somebody experienced the same problems with the mainboard H11DSi?

I already updated bios to the latest verion 1.1c


kinds regards MJ

edit: i checked again the supermicro website for my mainboard. the AOC-SLG3-2M2 is listed there! with the description "Requires motherboard BIOS support", so i guess maybe i only got a specific bios option wrong