CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > General Forums > Hardware

Server configuration suggestion (CFD, FEM)

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   September 24, 2021, 16:19
Default Server configuration suggestion (CFD, FEM)
  #1
New Member
 
Join Date: Mar 2019
Posts: 28
Rep Power: 4
Andrea23 is on a distinguished road
Hi,

can you suggest a server configuration optimized for finite volume method (openfoam, fluent) and finite element method (comsol) simulations?
3-4 people have to work on this server simultaneously, it is better to maximize physical CPU or virtual CPU? We don't need a very good GPU for graphic.
the budget is around 35,000-45,000
Andrea23 is offline   Reply With Quote

Old   September 24, 2021, 16:48
Default
  #2
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 2,864
Rep Power: 40
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
That's not a lot of information to base a hardware recommendation on
Comsol is the difficult part here. Knowing what type of simulation and problem size you want to run is crucial.
They have a write-up here: https://www.comsol.de/support/knowledgebase/866
Also make sure to follow some of the links there for further reading, e.g. https://www.comsol.com/blogs/much-me...-comsol-models
Some of the info in there is no longer relevant, in particular the part about memory population affecting memory frequency. But the rest sounds quite reasonable, and should be a good starting point to narrow things down.
You might even end up with a heterogeneous cluster: one node specialized for (large) Comsol runs, and more cost-efficient nodes for the rest.

Quote:
it is better to maximize physical CPU or virtual CPU?
Pretty much all x86 CPUs that could be considered for this application have 2 logical threads per core. So you can't specialize a configuration towards either of these metrics. And you should not be using more than one thread per core with these applications anyway.
flotus1 is offline   Reply With Quote

Old   September 27, 2021, 06:30
Default
  #3
New Member
 
Join Date: Mar 2019
Posts: 28
Rep Power: 4
Andrea23 is on a distinguished road
Quote:
Originally Posted by flotus1 View Post
That's not a lot of information to base a hardware recommendation on
Comsol is the difficult part here. Knowing what type of simulation and problem size you want to run is crucial.
They have a write-up here: https://www.comsol.de/support/knowledgebase/866
Also make sure to follow some of the links there for further reading, e.g. https://www.comsol.com/blogs/much-me...-comsol-models
Some of the info in there is no longer relevant, in particular the part about memory population affecting memory frequency. But the rest sounds quite reasonable, and should be a good starting point to narrow things down.
You might even end up with a heterogeneous cluster: one node specialized for (large) Comsol runs, and more cost-efficient nodes for the rest.


Pretty much all x86 CPUs that could be considered for this application have 2 logical threads per core. So you can't specialize a configuration towards either of these metrics. And you should not be using more than one thread per core with these applications anyway.
Thanks for your answer.

I am not the one using Comsol but I know they are running compressible flow in porous media simulation.
With Openfoam I am running a steady-state/transient incompressible with boussinesq approximation fluid-flow simulation, with a mesh of 10 million elements.

Now we are using a machine with 256gb RAM, 2 psychical processors, 36 (2X18) cores and 72 logical processors/threads; with HDD.
my first question is when I decompose an OpenFoam simulation in X processors, in using X cores or X logical processors ? As you said it is better so use X cores and 1 thread per core.

as for ram it is better to have, as an example, 8X64 gb or 16X32 gb?

To help you dimensionize the server and understand the computational load, I can say that with had no problem running 1 openfoam simulation (16 processors) and 1 comsol simulation (I think 8-12 processors); but when a third person logged to run another simulation with 12 Millions element (I don't know how many processors) the server started slowing down.

I don't know if these information can help you, I have no idea how to select the optimal configuration.

If you can give me an initial configuration then I can ask other person in my institution how to improve the configuration and it is needed to modify something to insert it in the network.
Andrea23 is offline   Reply With Quote

Old   September 28, 2021, 07:15
Default
  #4
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 2,864
Rep Power: 40
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
Quote:
my first question is when I decompose an OpenFoam simulation in X processors, in using X cores or X logical processors ? As you said it is better so use X cores and 1 thread per core.
With a recent kernel for your operating system, the scheduler should be clever enough to use physical CPU cores first, if you don't over-subscribe. I.e. not requesting more threads than the number of physical cores on your system.
"should" is the caveat here. That's why to this day, the recommendation to just disable SMT on the bios level is still around.
It's about load balancing. A parallel mesher and solver will attempt to distribute work units of similar weight to each process. And usually all processes have to wait at some point until the last process has finished its task. Putting two instead of one work unit on a physical core will break all assumptions for load balancing. This core will take significantly longer than all cores with only one work unit, which drastically increases overall run time.
Even in a best-case scenario -which we don't have for CFD workloads- SMT2 only increases performance of a core by ~30%. Which is never enough to offset twice the amount of work.

Quote:
as for ram it is better to have, as an example, 8X64 gb or 16X32 gb?
You are looking for compute nodes with two physical CPUs, and the latest generations of CPUs from AMD or Intel. They all have 8 memory channels, which means you need 16 DIMMs for each compute node.

Quote:
To help you dimensionize the server and understand the computational load, I can say that with had no problem running 1 openfoam simulation (16 processors) and 1 comsol simulation (I think 8-12 processors); but when a third person logged to run another simulation with 12 Millions element (I don't know how many processors) the server started slowing down.
Can be caused by a few different scenarios:
1) Too many tasks led to oversubscribing cores, see above
2) System ran out of physical memory. Swap needs to be avoided at all cost, i.e. make sure you have enough memory on each node. And maybe make nodes exclusive to a single job, or enforce memory limits via a queuing system.
3) Running out of memory bandwidth or other shared CPU resources like L3 cache General recommendations for CFD hardware [WIP]
Not much you can do about the last one, other than not allowing everyone to run a simulation on the same node

Quote:
If you can give me an initial configuration then I can ask other person in my institution how to improve the configuration and it is needed to modify something to insert it in the network.
Making the most out of your budget, I would probably get a few relatively cheap nodes for CFD. 2x Epyc 7413 (2x 24cores) and 16x16GB DDR4-3200 reg ECC.
Depending on what the actual requirements from your FEM-colleagues are for Comsol, and which priority it has, you have a few options:
1) just use the same node for Comsol too
2) bump up the memory on one of these nodes if they need more than 256GB
3) possibly in addition to 2: get a RAID0 array of some of the fastest NVMe-SSDs for the Comsol node if they need even more "memory" for out-of-core simulations
4) Get faster CPUs for the Comsol node. Not all of their solver variants can run distributed parallel (a single simulation running on several compute nodes), so having the fastest CPU available is the only option to speed things up further.
edit: forgot about 5, but really, the possibilities are endless) use the machine you currently have for Comsol exclusively, maybe with a memory upgrade
flotus1 is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
CFD Online Back on Main Server pete Site News & Announcements 0 March 31, 2011 19:14
CFD Online Running on Backup Server pete Site News & Announcements 0 March 25, 2011 15:51
CFD Online Server Up Again Jonas Larsson Main CFD Forum 2 November 27, 2006 18:02
CFD Online Server Move Jonas Larsson Main CFD Forum 0 January 15, 2005 08:42
CFD to FEM V. Worbington Main CFD Forum 1 July 21, 1998 16:40


All times are GMT -4. The time now is 20:24.