[ethan.le]

Hi folks,

I am building a workstation to run OpenFoam. I will be running a modified version of InterFoam with around 15-20 mil cells. Below is the specs:

Case SX3 Black - Velocity Micro Classic mid Tower Case
Power Supply 1600 Watt EVGA SuperNOVA Power Supply, 80Plus
Motherboard Dual SP5 Socket LGA 6096 Epyc Chipset, Motherboard
Processor Dual AMD EPYCTM 9654 Processors, 96-core/192-thread (192-core/384-thread total) @ 2.40GHz (3.70GHz Turbo)
CPU cooling 2 x Velocity Micro® LiquiCoolTM 8 - Custom loop - liquid cooling system

Memory 1.5TBGB DDR5-4800MHz (24 x 64GB), low latency, ECC
Video Card 12GB PNY® NVIDIA® RTX A2000 Workstation GPU
Hard Drive 1 4TB Crucial® P3 Plus NVMe M.2 Solid State Drive
up to 4800MBps Read/4100MBps write speed
Hard Drive 2-5 4x Seagate 16TB Exos X18 7200 rpm SAS III 12 Gb/s 3.5

Please let me know is there anything that I can improve/modify. Any suggestions is appreciated.

Thank you.

Ethan

[CFDfan]

Originally Posted by ethan.le

Hi folks,

I am building a workstation to run OpenFoam. I will be running a modified version of InterFoam with around 15-20 mil cells. Below is the specs:

Case SX3 Black - Velocity Micro Classic mid Tower Case
Power Supply 1600 Watt EVGA SuperNOVA Power Supply, 80Plus
Motherboard Dual SP5 Socket LGA 6096 Epyc Chipset, Motherboard
Processor Dual AMD EPYCTM 9654 Processors, 96-core/192-thread (192-core/384-thread total) @ 2.40GHz (3.70GHz Turbo)
CPU cooling 2 x Velocity Micro® LiquiCoolTM 8 - Custom loop - liquid cooling system

Memory 1.5TBGB DDR5-4800MHz (24 x 64GB), low latency, ECC
Video Card 12GB PNY® NVIDIA® RTX A2000 Workstation GPU
Hard Drive 1 4TB Crucial® P3 Plus NVMe M.2 Solid State Drive
up to 4800MBps Read/4100MBps write speed
Hard Drive 2-5 4x Seagate 16TB Exos X18 7200 rpm SAS III 12 Gb/s 3.5

Please let me know is there anything that I can improve/modify. Any suggestions is appreciated.

Thank you.

Ethan

Impressive spec..., how much was the total cost of the system?

[NhanLe]

Originally Posted by CFDfan

Impressive spec..., how much was the total cost of the system?

Thank you. It's $48k, no tax due to R&D purpose.

[wkernkamp]

We should start a second OpenFOAM Benchmark with 10x the number of cells just for you

[CFDfan]

Originally Posted by NhanLe

Thank you. It's $48k, no tax due to R&D purpose.

The spec is great but the system looks like greatly overpriced. The parts' cost is about a half of the US$48k you quoted

[trampoCFD]

Hi Ethan, unless you know that the 9654 is much faster than the 9454 on the particular models you run, I would take this fact into account:
The AMD Epyc Genoa 9654 is roughly 23% faster than a 9454 but you have to pay $15-20K extra.

If you want to keep things simple in a single workstation or need the 1.5TB RAM in a single computer, I'd go for the 9684X. It will only be marginally more expensive than the 9654, roughly 40% faster on your models, and up to 110% faster on small models.

I'd recommend a mini cluster with 2x 2x 9454 if you want peak performance for a similar budget.

PS: all the above performance figures are from composite benchmarks using Star-CCM+, which should roughly match Openfoam figures.

[ethan.le]

Yeah, you are correct. I switched for the 9684X in the final config. We are considering mini cluster option also.

[emmaagro]

In my experience I've been able to get away with an 18 core, 64GB RAM machine for 2D simulations, albeit slowly (6-7 days for the largest runs).You get a 16-core Epyc CPU with 64GB of RAM. That amount will be enough to run cases with 20M cells. And I included a relatively large NVMe SSD as an intermediate storage Core Ball medium for the operating system and the transient simulation results.OpenFOAM scales reasonably well up to thousands of cores, upper limit order of thousands of cores [1] We are looking at achieving radical scalability of cases of 100's of millions / billions of cell on 10K-100K cores.The reactingFoam solver is a transient solver and ANSYS Fluent was simulated at steady state condition. Standard k-ε model was used to predict the turbulence effect in both computational fluid dynamics codes. OpenFOAM gave a higher mixing level compared to ANSYS Fluent.