What to look at for a HPC to run 3-D meshes with chemistry?
 

At my work we are trying to find hardware in the ballpark of 10-15K that can effectively run a parallelized simulation that will at some point be asked to run refined 3-D cases with a chemistry model to model the mixing and combustion of fast moving gasses (near sonic velocities some of the times).

I understand that there is diminishing returns on the number of processor cores available to run in parallel, and I know there is a lot more to it than just that, so I figured I'd post some specs of a HPC we were quoted and see what you guys think:

Model: ACTserv x2110
Base system: Dual socket Xeon SP 2U system with 8x 2.5" drive bays
Omni-Path fabric integrated into processor: No
Processor: 2x Intel 12-Core Xeon Gold 6126 2.6GHz - 125W
Memory: 96GB - 12x 8GB DDR4 2666MHz
Storage configuration: 8x 2.5" SATA drives (software RAID only)
Storage 2.5" SATA: 2TB SATA 6Gb/s Enterprise 2.5" Hard Drive 7200RPM w/ 128MB Cache
Boot / OS drives: 2x 240GB Data Center SATA 2.5" Solid State Drive - RAID 1
Boot / OS drive location: Installed inside the system, not externally accessible
Networking: 2x RJ45 10Gb ethernet ports
OCP networking expansion: None
GPU configuration: 1x GPU
Management: Remote iKVM with in-band management
Power supply: Dual 1300W PSU (redundant)
Power cables: 2x 6ft NEMA 5-15P
Warranty: 3 year standard warranty

Do you already have a server rack in a dedicated room for this machine?
Does it need to be a 2U form factor?
Is this supposed to be a workstation or a compute node to log into remotely?
Which software will you be using?
Approximate cell count for the simulations?
Steady or transient simulations?
Do you have enough HDD capacity somewhere else to store your results?

Thanks for the reply,

1) Yes, we have a dedicated server rack for this machine.

2) We are not restrained by the 2U form factor.

3) We currently are using a workstation type setup, but we may end up wanting a setup where we can login remotely. It hasn't been decided yet. We only have one user that is currently needing the power to make these runs.

4) We use OpenFOAM primarily.

5) I could see the maximum cell count reaching into the 50-70M territory but it really comes down to when we can obtain grid independence on a simulation by simulation basis.

6) These will be almost exclusively transient simulations.

7) We do have several external HDD which we use to store results.

Then there are two things I would recommend:
1) AMD Epyc. Way more bang for the buck than Intel Xeon-SP at the moment, especially for software with no license cost. 2x AMD Epyc 7351 should fit comfortably into your budget. See sticky post in this subforum.
2) SSD-based storage for intermediate results. Since you have external disk space for long-term storage, I would not use spinning disks at all. Fast storage helps when dealing with lots of transient data files.

I appreciate the reply. The general rule of thumb I've been presented is that we'll need 2.5 GB of RAM per one million elements, which would place us in the 170 GB of RAM ballpark. I guess the first question is (1) is that rule of thumb really all that accurate and how much more RAM is needed to run through the equations dictating reaction when compared to just the turbulent mixing problem and (2) when storing information on an SSD, is there a noticeable speed difference when using swap versus only RAM.


Again, thanks for the help.

There is a performance difference between swapping on a regular HDD, a SATA-SSD and NVMe-SSDs. However, I would not design the workstation to swap. 256GB DDR4 cost around 3000$ which should be covered by your budget.
No matter how fast your SSDs are, running in-core is still much faster. And SSDs that can handle being abused as a swap drive for longer periods of time are not cheap either.

2.5GB per million cells should be a good estimate for standard simulations. Maybe someone with more experience in OpenFOAM can confirm. But adding chemistry with lots of species and reactions can increase memory requirements beyond that. You are in the best position to estimate how much RAM your simulations need per million cells. Just run a representative job with lower cell count on the hardware you already have.