[Andrea23]

Hi,

can you suggest a server configuration optimized for finite volume method (openfoam, fluent) and finite element method (comsol) simulations?
3-4 people have to work on this server simultaneously, it is better to maximize physical CPU or virtual CPU? We don't need a very good GPU for graphic.
the budget is around 35,000-45,000 €

[flotus1]

That's not a lot of information to base a hardware recommendation on
Comsol is the difficult part here. Knowing what type of simulation and problem size you want to run is crucial.
They have a write-up here: https://www.comsol.de/support/knowledgebase/866
Also make sure to follow some of the links there for further reading, e.g. https://www.comsol.com/blogs/much-me...-comsol-models
Some of the info in there is no longer relevant, in particular the part about memory population affecting memory frequency. But the rest sounds quite reasonable, and should be a good starting point to narrow things down.
You might even end up with a heterogeneous cluster: one node specialized for (large) Comsol runs, and more cost-efficient nodes for the rest.

it is better to maximize physical CPU or virtual CPU?

Pretty much all x86 CPUs that could be considered for this application have 2 logical threads per core. So you can't specialize a configuration towards either of these metrics. And you should not be using more than one thread per core with these applications anyway.

[Andrea23]

Originally Posted by flotus1

That's not a lot of information to base a hardware recommendation on
Comsol is the difficult part here. Knowing what type of simulation and problem size you want to run is crucial.
They have a write-up here: https://www.comsol.de/support/knowledgebase/866
Also make sure to follow some of the links there for further reading, e.g. https://www.comsol.com/blogs/much-me...-comsol-models
Some of the info in there is no longer relevant, in particular the part about memory population affecting memory frequency. But the rest sounds quite reasonable, and should be a good starting point to narrow things down.
You might even end up with a heterogeneous cluster: one node specialized for (large) Comsol runs, and more cost-efficient nodes for the rest.


Pretty much all x86 CPUs that could be considered for this application have 2 logical threads per core. So you can't specialize a configuration towards either of these metrics. And you should not be using more than one thread per core with these applications anyway.

Thanks for your answer.

I am not the one using Comsol but I know they are running compressible flow in porous media simulation.
With Openfoam I am running a steady-state/transient incompressible with boussinesq approximation fluid-flow simulation, with a mesh of 10 million elements.

Now we are using a machine with 256gb RAM, 2 psychical processors, 36 (2X18) cores and 72 logical processors/threads; with HDD.
my first question is when I decompose an OpenFoam simulation in X processors, in using X cores or X logical processors ? As you said it is better so use X cores and 1 thread per core.

as for ram it is better to have, as an example, 8X64 gb or 16X32 gb?

To help you dimensionize the server and understand the computational load, I can say that with had no problem running 1 openfoam simulation (16 processors) and 1 comsol simulation (I think 8-12 processors); but when a third person logged to run another simulation with 12 Millions element (I don't know how many processors) the server started slowing down.

I don't know if these information can help you, I have no idea how to select the optimal configuration.

If you can give me an initial configuration then I can ask other person in my institution how to improve the configuration and it is needed to modify something to insert it in the network.

[flotus1]

my first question is when I decompose an OpenFoam simulation in X processors, in using X cores or X logical processors ? As you said it is better so use X cores and 1 thread per core.

With a recent kernel for your operating system, the scheduler should be clever enough to use physical CPU cores first, if you don't over-subscribe. I.e. not requesting more threads than the number of physical cores on your system.
"should" is the caveat here. That's why to this day, the recommendation to just disable SMT on the bios level is still around.
It's about load balancing. A parallel mesher and solver will attempt to distribute work units of similar weight to each process. And usually all processes have to wait at some point until the last process has finished its task. Putting two instead of one work unit on a physical core will break all assumptions for load balancing. This core will take significantly longer than all cores with only one work unit, which drastically increases overall run time.
Even in a best-case scenario -which we don't have for CFD workloads- SMT2 only increases performance of a core by ~30%. Which is never enough to offset twice the amount of work.

as for ram it is better to have, as an example, 8X64 gb or 16X32 gb?

You are looking for compute nodes with two physical CPUs, and the latest generations of CPUs from AMD or Intel. They all have 8 memory channels, which means you need 16 DIMMs for each compute node.

To help you dimensionize the server and understand the computational load, I can say that with had no problem running 1 openfoam simulation (16 processors) and 1 comsol simulation (I think 8-12 processors); but when a third person logged to run another simulation with 12 Millions element (I don't know how many processors) the server started slowing down.

Can be caused by a few different scenarios:
1) Too many tasks led to oversubscribing cores, see above
2) System ran out of physical memory. Swap needs to be avoided at all cost, i.e. make sure you have enough memory on each node. And maybe make nodes exclusive to a single job, or enforce memory limits via a queuing system.
3) Running out of memory bandwidth or other shared CPU resources like L3 cache General recommendations for CFD hardware [WIP]
Not much you can do about the last one, other than not allowing everyone to run a simulation on the same node

If you can give me an initial configuration then I can ask other person in my institution how to improve the configuration and it is needed to modify something to insert it in the network.

Making the most out of your budget, I would probably get a few relatively cheap nodes for CFD. 2x Epyc 7413 (2x 24cores) and 16x16GB DDR4-3200 reg ECC.
Depending on what the actual requirements from your FEM-colleagues are for Comsol, and which priority it has, you have a few options:
1) just use the same node for Comsol too
2) bump up the memory on one of these nodes if they need more than 256GB
3) possibly in addition to 2: get a RAID0 array of some of the fastest NVMe-SSDs for the Comsol node if they need even more "memory" for out-of-core simulations
4) Get faster CPUs for the Comsol node. Not all of their solver variants can run distributed parallel (a single simulation running on several compute nodes), so having the fastest CPU available is the only option to speed things up further.
edit: forgot about 5, but really, the possibilities are endless) use the machine you currently have for Comsol exclusively, maybe with a memory upgrade