What to look at for a HPC to run 3-D meshes with chemistry?

trl0007 · April 10, 2018, 19:07

At my work we are trying to find hardware in the ballpark of 10-15K that can effectively run a parallelized simulation that will at some point be asked to run refined 3-D cases with a chemistry model to model the mixing and combustion of fast moving gasses (near sonic velocities some of the times).

I understand that there is diminishing returns on the number of processor cores available to run in parallel, and I know there is a lot more to it than just that, so I figured I'd post some specs of a HPC we were quoted and see what you guys think:

Model: ACTserv x2110
Base system: Dual socket Xeon SP 2U system with 8x 2.5" drive bays
Omni-Path fabric integrated into processor: No
Processor: 2x Intel 12-Core Xeon Gold 6126 2.6GHz - 125W
Memory: 96GB - 12x 8GB DDR4 2666MHz
Storage configuration: 8x 2.5" SATA drives (software RAID only)
Storage 2.5" SATA: 2TB SATA 6Gb/s Enterprise 2.5" Hard Drive 7200RPM w/ 128MB Cache
Boot / OS drives: 2x 240GB Data Center SATA 2.5" Solid State Drive - RAID 1
Boot / OS drive location: Installed inside the system, not externally accessible
Networking: 2x RJ45 10Gb ethernet ports
OCP networking expansion: None
GPU configuration: 1x GPU
Management: Remote iKVM with in-band management
Power supply: Dual 1300W PSU (redundant)
Power cables: 2x 6ft NEMA 5-15P
Warranty: 3 year standard warranty

flotus1 · April 11, 2018, 03:40

Do you already have a server rack in a dedicated room for this machine?
Does it need to be a 2U form factor?
Is this supposed to be a workstation or a compute node to log into remotely?
Which software will you be using?
Approximate cell count for the simulations?
Steady or transient simulations?
Do you have enough HDD capacity somewhere else to store your results?

trl0007 · April 11, 2018, 09:34

Thanks for the reply,

1) Yes, we have a dedicated server rack for this machine.

2) We are not restrained by the 2U form factor.

3) We currently are using a workstation type setup, but we may end up wanting a setup where we can login remotely. It hasn't been decided yet. We only have one user that is currently needing the power to make these runs.

4) We use OpenFOAM primarily.

5) I could see the maximum cell count reaching into the 50-70M territory but it really comes down to when we can obtain grid independence on a simulation by simulation basis.

6) These will be almost exclusively transient simulations.

7) We do have several external HDD which we use to store results.

flotus1 · April 11, 2018, 09:50

Then there are two things I would recommend:
1) AMD Epyc. Way more bang for the buck than Intel Xeon-SP at the moment, especially for software with no license cost. 2x AMD Epyc 7351 should fit comfortably into your budget. See sticky post in this subforum.
2) SSD-based storage for intermediate results. Since you have external disk space for long-term storage, I would not use spinning disks at all. Fast storage helps when dealing with lots of transient data files.

trl0007 · April 11, 2018, 11:03

I appreciate the reply. The general rule of thumb I've been presented is that we'll need 2.5 GB of RAM per one million elements, which would place us in the 170 GB of RAM ballpark. I guess the first question is (1) is that rule of thumb really all that accurate and how much more RAM is needed to run through the equations dictating reaction when compared to just the turbulent mixing problem and (2) when storing information on an SSD, is there a noticeable speed difference when using swap versus only RAM.

Again, thanks for the help.

flotus1 · April 11, 2018, 12:01

There is a performance difference between swapping on a regular HDD, a SATA-SSD and NVMe-SSDs. However, I would not design the workstation to swap. 256GB DDR4 cost around 3000$ which should be covered by your budget.
No matter how fast your SSDs are, running in-core is still much faster. And SSDs that can handle being abused as a swap drive for longer periods of time are not cheap either.

2.5GB per million cells should be a good estimate for standard simulations. Maybe someone with more experience in OpenFOAM can confirm. But adding chemistry with lots of species and reactions can increase memory requirements beyond that. You are in the best position to estimate how much RAM your simulations need per million cells. Just run a representative job with lower cell count on the hardware you already have.

April 10, 2018, 19:07	What to look at for a HPC to run 3-D meshes with chemistry?	#1
trl0007 New Member Tyler Ross Lambert Join Date: Jun 2017 Location: Huntsville, AL Posts: 8 Rep Power: 8	At my work we are trying to find hardware in the ballpark of 10-15K that can effectively run a parallelized simulation that will at some point be asked to run refined 3-D cases with a chemistry model to model the mixing and combustion of fast moving gasses (near sonic velocities some of the times). I understand that there is diminishing returns on the number of processor cores available to run in parallel, and I know there is a lot more to it than just that, so I figured I'd post some specs of a HPC we were quoted and see what you guys think: Model: ACTserv x2110 Base system: Dual socket Xeon SP 2U system with 8x 2.5" drive bays Omni-Path fabric integrated into processor: No Processor: 2x Intel 12-Core Xeon Gold 6126 2.6GHz - 125W Memory: 96GB - 12x 8GB DDR4 2666MHz Storage configuration: 8x 2.5" SATA drives (software RAID only) Storage 2.5" SATA: 2TB SATA 6Gb/s Enterprise 2.5" Hard Drive 7200RPM w/ 128MB Cache Boot / OS drives: 2x 240GB Data Center SATA 2.5" Solid State Drive - RAID 1 Boot / OS drive location: Installed inside the system, not externally accessible Networking: 2x RJ45 10Gb ethernet ports OCP networking expansion: None GPU configuration: 1x GPU Management: Remote iKVM with in-band management Power supply: Dual 1300W PSU (redundant) Power cables: 2x 6ft NEMA 5-15P Warranty: 3 year standard warranty

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Job submit code in HPC for Workbench	beyonder1	ANSYS	0	January 4, 2016 08:58
Pinning the processors in cluster to run a solver	coolcrasher	OpenFOAM Running, Solving & CFD	0	November 5, 2015 06:11
Can I run ANSYS FLUENT on Amazon HPC?	federvo.mala	ANSYS	2	March 5, 2012 18:34
Solving for two distinct meshes in same run.	Fransje	OpenFOAM Running, Solving & CFD	0	November 16, 2010 11:08
What's the best order to run this simulation in?	siw	CFX	1	November 4, 2009 19:42

April 11, 2018, 03:40		#2
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,400 Rep Power: 47	Do you already have a server rack in a dedicated room for this machine? Does it need to be a 2U form factor? Is this supposed to be a workstation or a compute node to log into remotely? Which software will you be using? Approximate cell count for the simulations? Steady or transient simulations? Do you have enough HDD capacity somewhere else to store your results?

April 11, 2018, 09:34		#3
trl0007 New Member Tyler Ross Lambert Join Date: Jun 2017 Location: Huntsville, AL Posts: 8 Rep Power: 8	Thanks for the reply, 1) Yes, we have a dedicated server rack for this machine. 2) We are not restrained by the 2U form factor. 3) We currently are using a workstation type setup, but we may end up wanting a setup where we can login remotely. It hasn't been decided yet. We only have one user that is currently needing the power to make these runs. 4) We use OpenFOAM primarily. 5) I could see the maximum cell count reaching into the 50-70M territory but it really comes down to when we can obtain grid independence on a simulation by simulation basis. 6) These will be almost exclusively transient simulations. 7) We do have several external HDD which we use to store results.

April 11, 2018, 09:50		#4
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,400 Rep Power: 47	Then there are two things I would recommend: 1) AMD Epyc. Way more bang for the buck than Intel Xeon-SP at the moment, especially for software with no license cost. 2x AMD Epyc 7351 should fit comfortably into your budget. See sticky post in this subforum. 2) SSD-based storage for intermediate results. Since you have external disk space for long-term storage, I would not use spinning disks at all. Fast storage helps when dealing with lots of transient data files.

April 11, 2018, 11:03		#5
trl0007 New Member Tyler Ross Lambert Join Date: Jun 2017 Location: Huntsville, AL Posts: 8 Rep Power: 8	I appreciate the reply. The general rule of thumb I've been presented is that we'll need 2.5 GB of RAM per one million elements, which would place us in the 170 GB of RAM ballpark. I guess the first question is (1) is that rule of thumb really all that accurate and how much more RAM is needed to run through the equations dictating reaction when compared to just the turbulent mixing problem and (2) when storing information on an SSD, is there a noticeable speed difference when using swap versus only RAM. Again, thanks for the help.

April 11, 2018, 12:01		#6
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,400 Rep Power: 47	There is a performance difference between swapping on a regular HDD, a SATA-SSD and NVMe-SSDs. However, I would not design the workstation to swap. 256GB DDR4 cost around 3000$ which should be covered by your budget. No matter how fast your SSDs are, running in-core is still much faster. And SSDs that can handle being abused as a swap drive for longer periods of time are not cheap either. 2.5GB per million cells should be a good estimate for standard simulations. Maybe someone with more experience in OpenFOAM can confirm. But adding chemistry with lots of species and reactions can increase memory requirements beyond that. You are in the best position to estimate how much RAM your simulations need per million cells. Just run a representative job with lower cell count on the hardware you already have.