[LJA_Koers]

Hi all,

As the only CFD-engineer at my workplace, I recently convinced some people to acquire a new Ansys CFD Premium license to replace the old one. I'm now tasked with finding new appropriate hardware to install this license on. I've tried to look into the necessary specs but I still feel like a need some advice, hopefully some of you can help me out.

some things to point out:
1. The software needs to be installed on a server which I will access through a remote desktop connection. So no laptop workstation.
2. It was decided to not get the High Performance Computing license so I'm limited to 4 cores according to the Ansys CFD Premium License.
3. The price range is up to €12.000.

The most computationally challenging simulation Ill be doing are combustions simulation, multiphase flow/boiling, and heat-transfer simulations up to about 40million cells (meshed by Ansys 14).

To optimize the design of our products, I want to prepare a few simulations during the day and let them run over night, review the results the next day.

I've read a bit about the importance of RAM for CFD simulations. In short, it is better to spend extra cash on more RAM than on a more powerful CPU, Am I correct there?

Also read an SSD Drive helps with solving while an HDD is just good for storage. What amounts of storage for both would be sufficient?

If there is anything I forgot, or what you think would help me, I'd gladly hear it.

Thanks in advance!

[flotus1]

I've read a bit about the importance of RAM for CFD simulations. In short, it is better to spend extra cash on more RAM than on a more powerful CPU, Am I correct there?

You need enough RAM to fit your largest models. Once you are there, adding even more doesn't help. Given your requirements and the rather generous budget, I would recommend 192-256GB, depending on the CPU choice.
For the CPU, there is no need to cut corners. The goal is to make the most of your limited licenses. It's either Intel Xeon Gold 6250 or AMD Epyc 72F3.
But overall, don't expect any miracles from your hardware. 4 threads is not a lot for transient simulations with 40m cells.

Also read an SSD Drive helps with solving while an HDD is just good for storage. What amounts of storage for both would be sufficient?

The amount of storage is difficult to pin down from afar. You know best how much storage you need for each project, and how many recent projects you need to retain in full detail.
With 40 million cells and mostly transient simulation, I would say SSD all the things. And not just SATA SSDs, but proper NVMe. At least for the project you are currently working on. If you don't have external storage for long-term archiving of old projects, you can start adding hard drives.

[LJA_Koers]

Thank you!

I'll take your suggestions into consideration. The budget was suggested by some IT guy, having looked into it a little, I feel like it is an overestimation.

[flotus1]

It may sound like an overestimation at first. But honestly, that's where hardware budgets should be in a professional setting.
First of all, the licenses are expensive. So making the most out of them with capable hardware makes sense.
More importantly, your time is expensive. It probably costs your company somewhere north of 100€ per hour. Over the 3-5 year lifespan of that workstation, that would be a lot of money wasted if you have to wait longer than absolutely necessary for your workstation to do its job.
If you can e.g. shave off 30 seconds each time you load a model by spending 1000€ on faster storage, that is a financially viable solution.

[danbence]

40M cells on 4 cores may take on the scale of 15 mins per time step, so I hope you don't need to do many time steps.

[pbasis]

A multiphase simulation with 40 million cells (or really even single phase) on 4-cores sounds wildly optimistic. Unless you are willing to wait months for the results, you're best off either getting the HPC packs or limiting yourself to smaller problems in the 1-5 million cell range depending on simulation complexity.

Fluent is meant to scale in parallel. You can only get so much performance out of one chip even if it is very high end. Your bottleneck will likely be memory access and cache access.

[LJA_Koers]

40 mil is about the maximum amount in a problem that I can subdivide into several problems, and most of my simulations are steady state. We are however considering using this license for consultancy projects, some of those might require transient simulation so its probably wise to spend the extra euro for some more processing power.

[LJA_Koers]

To continue on this, I talked to some people and made them clear that the extra HPC pack was a necessity. Perhaps even a second one with help of subsidy.

I found this set of Ansys benchmarks today and the 11th of them, the 71million cells combustor, is similar to what I'm expected to conduct soon:

https://www.ansys.com/it-solutions/b...overview#terms

so a few follow up questions:

1) Are the results of this benchmark accurate? In my limited CFD experience, solving a problem with 71 million cells with complex physics in a transient simulation on 64 cores 39 times a day seems too good to be true. Could this be be result of say a perfect mesh and perfect settings + only a few time steps?

2) If I have 1 HPC license to use and can solve on 12 cores. Does the core solver rating keep scaling linearly to 38.7896/64*12=7.27. Assuming the same 2.95GHz.

3) I keep finding that parallel solving gives the fastest results. So for 1 HPC pack I was thinking of having 2 Xeon CPUs with 8 cores each. For a 2 HPC pack(36 cores for solving), perhaps 4 CPUs with 10 cores each. That's based on my intuition but does 4*10 cores solve faster than 2*20 cores? Leaving extra cores for meshing or other things.

4) Is it possible to have two sets of CPU's in a server? So 2x 8-core CPU and 2x 10-core CPU for example?

Thank you in advance

[flotus1]

1) Are the results of this benchmark accurate? In my limited CFD experience, solving a problem with 71 million cells with complex physics in a transient simulation on 64 cores 39 times a day seems too good to be true. Could this be be result of say a perfect mesh and perfect settings + only a few time steps?

You would have to download the benchmark and check what stopping criteria they actually used.
But overall, their "core solver rating" can be a bit misleading. IIRC, for steady-state flows the benchmarks just run for a fixed number of iterations. A number of iterations that is usually way too small to get convergence. I would not expect the transient benchmarks to be any different. The benchmark result is only useful as a relative metric, not an absolute one. I.e. don't expect to run several of this kind of simulation per day, producing meaningful results.

2) If I have 1 HPC license to use and can solve on 12 cores. Does the core solver rating keep scaling linearly to 38.7896/64*12=7.27. Assuming the same 2.95GHz.

Running the same case on the same hardware, but with a lower thread count, should actually produce super-linear "scaling". I.e. Faster than 64/12*solution time.
The reason is that with 64 cores on 16 memory channels, memory bandwidth starts to become a limiting factor. Also, CPU frequency will be higher when running on fewer cores.

3) I keep finding that parallel solving gives the fastest results. So for 1 HPC pack I was thinking of having 2 Xeon CPUs with 8 cores each. For a 2 HPC pack(36 cores for solving), perhaps 4 CPUs with 10 cores each. That's based on my intuition but does 4*10 cores solve faster than 2*20 cores? Leaving extra cores for meshing or other things.

More CPUs means more shared resources like L3 cache and memory bandwidth. Which usually results in faster solver times. By how much depends on the exact hardware setup and simulation.
General rule of thumb for CFD solvers with expensive per-core licenses: 2-3 cores per memory channel. If you use more than that, scaling drops due to memory bandwidth limitations. In which case you would have been better off adding another machine instead of choosing CPUs with higher core counts.
Further reading: General recommendations for CFD hardware [WIP]

4) Is it possible to have two sets of CPU's in a server? So 2x 8-core CPU and 2x 10-core CPU for example?

Sure. You can also see it in the first-party benchmarks from Ansys. The column "machines" denotes how many of these boxes were used to solve the model in distributed parallel, connected via Infiniband.
You need some kind of interconnect for running a single simulation distributed across both nodes though. Infiniband is the de-facto standard for this, but for only 2 nodes, you can probably get away with 10Gigabit Ethernet. If scaling across both nodes turns out to be insufficient, you can still drop in two IB cards. No expensive switch required for only 2 nodes.
Intel also offers solutions with 4 or even 8 CPUs in a shared memory system, if you want to avoid distributed memory for whatever reason. But that's generally not necessary for CFD.
That being said, for something in the order of 32-48 cores, I would definitely use a single system with two CPUs. If money for hardware is not a huge issue, Epyc 73F3 (16 cores) or 74F3 (24 cores) would be my preferred options. Otherwise Epyc 7313 or 7443 are more budget-friendly options. They have lower clock speeds and only half the L3 cache, but cost significantly less.

[LJA_Koers]

Thank you flotus!

You've been a great help.