[Vestdam]

Hi there,

I am investing in a CFD workstation and I have found the following supplier in Germany, who builds dual EPYC workstations. From what I could read on the forum, and the budget of course, I have made the following configuration:

CPU
2 x AMD EPYC 7443
16 x 8 GB RAM (Micron MTA9ASF1G72PZ-3G2)
1 x Crucial MX500 CT2000MX500SSD1
1 x NVIDIA GeForce GTX 1050 Ti
1 x Crucial P5 Plus CT500P5PSSD8
https://www.deltacomputer.com/d20z-m4-zm-2xgpu.html
The configuration is 8.000 Euro - having 2x EPYC 74F3 would increase the price to 10.500 Euro.
I have (almost) unlimited core license avaiable the first year, and then possibly limited to 12 or 36 cores after that. I intend to work in OpenFoam on the side by that time, so that I can utilize more hardware. I will be simulating multiphase flow and heat transfer mainly on relatively small mesh sizes (<15 mil.).

My questions are:
- Any specific recommendations on the parts except for the CPU choice?
- Are there any rough estimates for, what 128 mb of extra chache and base speed would give in performance increase (compared to the cost of 2.500 Euro) with the 74F3?

Hope you can give some insight.

[flotus1]

Any specific recommendations on the parts except for the CPU choice?

I would recommend against this particular PC if you want it silent. The Supermicro cases and power supplies can be fairly noisy. The same SI has pretty much the same specs in a different case, with a regular ATX power supply.
Any particular reason why you picked two 500GB SSDs, and no hard drives at all? is 500GB all you will ever need for local storage? SSDs are cheap, I would recommend the second SSD with at least 2TB for a workstation in the 10k€ range.

Are there any rough estimates for, what 128 mb of extra chache and base speed would give in performance increase (compared to the cost of 2.500 Euro) with the 74F3?

Tough question... but since you will be limited to per-core licenses later, and some portion of the price for the whole workstation is overhead from the SI, I would say it is worth it.

Since you are in Germany, maybe try to get another quote from here: https://server-konfigurieren.de/prod...ntel-dual-epyc
I recently had the "pleasure" of going through an SI to source a workstation for a friend. They were rather accommodating with all the nitpicks I had for the configuration, just give them a call. And at least the second quote we got from them was fairly cheap.
The computer has not been shipped yet, so I can't say anything about the build quality or service. In fact, it has been delayed because they suddenly claimed they can't source the motherboard any more, and wanted to swap in a cheaper one. But apart from that, the whole ordeal wasn't too unpleasant so far.

[Vestdam]

Hi flotus,

First of all: thank you very much for your response.

Thanks for the case-information - I had no idea. Will go for the other case, if I choose this supplier.

The Crucial MX500 SSD is actually a 2TB SSD, I guess indicated by the "CT200MX...", however I see how this is confusing.

I will see if the budget can be stretched to include the 74F3's instead - the price increase compared to the aquisition of commercial CFD software is small percentages anyway.

I am in fact located in Denmark, but as I have found no supplier locally, I looked to Germany. I will try writing/calling both suppliers to have something to compare. Thanks for letting me know.

[flotus1]

Yeah, my mind went blank after reading "MX500"
Still, probably better to have the larger SSD as an NVMe for the increased read/write speeds. The price differences between SATA and NVMe SSDs are really small these days.

[Vestdam]

I sent a request for quotes on both types of CPUs to both suppliers. Deltacomputers came back with both quotes in two different cases. Appearantly, they do not build the 74F3 in the "silent" case, as this requires too much cooling.

I don't really mind a louder case, as I am going to put the workstation in a remote room, but it makes me think about what cooling issues could occur down the line with this supplier.

I'll see what the other supplier comes back with.

Originally Posted by flotus1

The price differences between SATA and NVMe SSDs are really small these days.

Good point. Thanks again

[flotus1]

I have to retract my recommendation for this particular SI.
They assured me that the system they build would be "super quiet". I made it very clear in a phone call that this was a priority. I also went off the menu and ordered some decent CPU coolers, leaving no room for interpretation. I initially ordered their "quiet package" for an extra 50€, but the sales person assured me that it would not have any impact on this particular build. What I get instead of a quiet system is the default fans the case comes with (3-pin connectors) plugged into 4-pin headers on the board. Thus running at max rpm all the time.
And they decided to swap out the power supply for a shitty one that costs half as much as the one I ordered. Just sneaking it in, without any notification.
FFS, why does this happen every time? It's not that hard to build a decent PC, and yet people who do it for a living always seem to find ways to screw it up.

[Vestdam]

Well, that doesn't sound good. I seems that the quality of the suppliers of custom built workstations varies quite a lot.

Good thing they never got back to me on my request.
In the meantime, I have found two suppliers in Denmark (where I'm located) with similar prices as the ones in Germany.

[Vestdam]

Hi again,

Having recieved the workstation, I was wondering if you have any thoughts on installing an OS? Should I be able to install windows 10/11 on this machine? Should I consider windows server?

Best regards.

[flotus1]

You should be able to install Windows 10, probably also Windows 11. Variants like "home" and "pro" have different limits for the number of threads and memory size they are allowed to handle, so choose wisely
When I did that for testing a few years ago with an H11DSI, I had to play around with IOMMU and above 4G decoding in bios to get it working. This might have improved by now.
Windows server doesn't add anything useful as far as I am concerned, "Pro" is probably enough.

But...if you want to use OpenFOAM, I strongly recommend using Linux natively. There are quite a few examples in our OpenFOAM benchmarking thread, where virtual machines and WSL lead to severe performance regression.
I guess you want to use that other software on Windows. Maybe dual boot is an option for you? If you want to start with Windows now, you can always install Linux as a second OS once you need it.

[Vestdam]

Hi Flotus,

Thanks for getting back to me. I was unable to succeed with installing Win 10 and 11 pro, but I also just discovered what you mention with IOMMU and 4g decoding, so I hope that will fix it.

I am mainly going to use Ansys, and I have never done much Linux, so I think I will start of with Windows (if possible) as you say, and then have Linux for future testing and possibly OpenFoam later.

[flotus1]

Which part of the installation are you stuck on? What's the problem?

[Vestdam]

Windows 11 reported that the hardware is not compatible.

Windows 10 got to the partitioning and choosing a drive. After that it said something about missing files. This could just be a bad USB-boot drive though.

I will try again when I get back to work and give an update. Now I am a bit more confident I should get it working.

[Vestdam]

Hi Flotus,

I got Win10 Pro working by disabling 4G decoding and using USB 2.0 instead (don't know which configuration solved it).

Do you have any tips on bios settings for a CFD machine like this other than disabling SMT?

Thanks again!

[flotus1]

Congratulations on getting it working.
I don't know which motherboard you have. Available bios settings and their names can vary between brands and models.
I recently got my hands on a workstation with a Gigabyte MZ72-HB0. I wrote down the bios settings here: OpenFOAM benchmarks on various hardware
You can spend a looong time trying to optimize bios settings, but from a productivity standpoint, it is rarely worth it.
The biggest factors are disabling SMT, and using NPS=4 mode for Rome and Milan CPUs. Then again, I don't know what the Windows scheduler does in conjunction with Ansys (Fluent?) when 8 NUMA nodes are present, so your mileage may vary. It's also a bit workload dependent, NPS=1 can be better for some applications.

[Vestdam]

The motherboard is a Supermicro H12DSI-NT6 with two 74F3 CPUs.

I will try to do some research on the NPS settings for this setup running ANSYS Fluent.

Thanks for everything!

[cramr5]

Originally Posted by flotus1

You should be able to install Windows 10, probably also Windows 11. Variants like "home" and "pro" have different limits for the number of threads and memory size they are allowed to handle, so choose wisely .

According to Wikipedia I don't think he should have any problems...
32 logical cores on IA-32
1280 logical cores (20 groups of 64 logical processors) on x86-64

[gilbert45]

For that specific motherboard (Gigabyte MZ32-AR0) it's a little hazy how to connect the case fans? The documentation is quite awful as far as what goes where - beside the CPU fan header.

[flopiflopi]

I am also building myself a workstation for CFD. I am using OpenFOAM mostly for turbomachinery simulations. I would need a configuration to work with meshes with 6-15 milion elements. Budget for my worksation is around 20k€ and I am thinking using two EPYC 7573X processores.
The configuration I curretly have in mind is:
- Tower SR10766 Silent AMD EPYC 7003 Dual CPU Server
- be quiet Dark Power PRO 12 1500W - 80 PLUS Platinum Silent ( BN312 )
- Supermicro Dual Mainboard H12DSI-NT6- 2 x 10GB LAN - AMD EPYC 7003 - (Milan CPU)
- AMD EPYC "MILAN X" 7573X - 2.8GHz / 3.6GHz (Boost) / 768MB Cache / 32 Cores / 280W
- 4x64GB DDR4 3200MHz - Reg. ECC RDIMM
- 1.6TB Samsung PM1735 - NVME PCIe 4.0 - Read 7000MB/s - Write 2400MB/s
- GeForce RTX 3060 - 12GB - GDDR6X PCIe 4.0 x16

I am wondering if that would be enough RAM? Is Epyx 7573X applicable decision?
Hope you can give your toughts.

[cramr5]

Originally Posted by flopiflopi

- AMD EPYC "MILAN X" 7573X - 2.8GHz / 3.6GHz (Boost) / 768MB Cache / 32 Cores / 280W
- 4x64GB DDR4 3200MHz - Reg. ECC RDIMM

You will be using 2 CPUs that each have 8 memory channels.

If you want 256 Gb of RAM you would be better with 16 sticks of 16GB each to use the full potential of the memory bandwith than only using 4 sticks

[flopiflopi]

Thank you for that remark. Any other thoughts maybe?