[cleanflow]

Hi guys,

can you help me with the hardware configuration for CFD simulation? I would buy 1 node (2U rack server with 2 CPUs, for example Lenovo [s]SR655[/s] SR665 or Dell R7525) to start and then I would buy more if it would be necessary.

Which software do you intend to use?
Fluent meshing and Openfoam solver.

Operating system?
Linux.

Are you limited by license constraints? I.e. does your software license only allow you to run on N threads?
No, for now.

What type of simulations do you want to run? And what's the maximum cell count?
CFD simulations, mostly RANS for now. Not sure about the cell count, roughly estimate around 150M cells.

If there is a budget, how high is it?
Let's say there are two configurations: cheaper under 10k euros and expensive under 20k euros.

What kind of setting are you in? Hobbyist? Student? Academic research? Engineer?
CFD Engineer.

Where can you source your new computer? Buying a complete package from a large OEM? Assemble it yourself from parts? Are used parts an option?
Large OEM, new parts only.

Which part of the world are you from? It's cool if you don't want to tell, but since prices and availability vary depending on the region, this can sometimes be relevant. Particularly if it's not North America or Europe.
Europe.

Anything else that people should know to help you better?
I would like to build on the AMD EPYC platform because I think you get more for the money. Please correct me if I am wrong.
Which server rack manufacturer do you recommend (Dell, HP, Lenovo etc.)?

Thank you in advance.

[wkernkamp]

flotus is the resident expert for this type of system, especially for Europe. He will probably respond soon enough. In the mean time, I noticed that the Lenovo SR655 is a single socket system. It is probably more efficient for you to select a dual socket system. The problem size is fairly large and dual socket basically doubles your speed.


The SR655 is a good choice for a storage server, or a mixed cpu/gpu server. However, in your case, you want the two cpus and all channels filled with at least one DIMM.

[cleanflow]

Originally Posted by wkernkamp

flotus is the resident expert for this type of system, especially for Europe. He will probably respond soon enough. In the mean time, I noticed that the Lenovo SR655 is a single socket system. It is probably more efficient for you to select a dual socket system. The problem size is fairly large and dual socket basically doubles your speed.


The SR655 is a good choice for a storage server, or a mixed cpu/gpu server. However, in your case, you want the two cpus and all channels filled with at least one DIMM.

sorry my bad, it was a typo I meant SR665

[flotus1]

Thanks for the endorsement @wkernkamp, but I am pretty far from being an expert on "which server to buy". All I know is "which hardware to put in".

For lower range of the budget, we are probably looking at 3rd gen Epyc Milan with 2x24 cores or 2x32 cores. Since we also want 512GB (16x32GB) of memory. And other stuff like storage and maybe a graphics card? Depends on whether this is supposed to be just a compute node, or if it should also handle remote visualization.
7543 would be nice because it is a fully enabled package, i.e. 256MB of L3 cache.
Other options if the budget doesn't allow it: 7513 (32C) or 7413 (24C).
It's a bit hard to judge right now, because retail prices for this generation are heavily discounted.

For upper end of the budget, it's either Milan-X: 2x 7573X with 16x32GB of DDR4-3200.
Or 4th gen: 2x 9354 with 24x16GB of RAM.
Not quite sure how much memory Fluent meshing really needs for 150 million cells. This would be worth checking with a representative test case.

[cleanflow]

Originally Posted by flotus1

Thanks for the endorsement @wkernkamp, but I am pretty far from being an expert on "which server to buy". All I know is "which hardware to put in".

For lower range of the budget, we are probably looking at 3rd gen Epyc Milan with 2x24 cores or 2x32 cores. Since we also want 512GB (16x32GB) of memory. And other stuff like storage and maybe a graphics card? Depends on whether this is supposed to be just a compute node, or if it should also handle remote visualization.
7543 would be nice because it is a fully enabled package, i.e. 256MB of L3 cache.
Other options if the budget doesn't allow it: 7513 (32C) or 7413 (24C).
It's a bit hard to judge right now, because retail prices for this generation are heavily discounted.

For upper end of the budget, it's either Milan-X: 2x 7573X with 16x32GB of DDR4-3200.
Or 4th gen: 2x 9354 with 24x16GB of RAM.
Not quite sure how much memory Fluent meshing really needs for 150 million cells. This would be worth checking with a representative test case.

Regarding the GPU I am not sure, I would do post-processing on another machine. But also if I need to check something quickly in Fluent then I would like to do it on the node. So this means I would need a GPU on it, right?

2x9354 is in our upper budget, is this the best option for now? What do I lose if I fill only 12 slots (6 slots per CPU) with 32 GB instead of 24 slots (12 slots per CPU) with 16 GB (total 384GB) for the beginning and then add more if it is necessary?

PS. Sorry for the late answer

[flotus1]

With CCM+, we can connect to simulations running on compute nodes without a GPU. I would be surprised if that was not possible with Fluent.

Leaving half the memory channels empty on a fancy new machine for 20000€ stings a bit.
It won't be as bad, since that still yields the same theoretical maximum bandwidth as last-gen. But then you might as well save the money and buy last gen.

[cleanflow]

Originally Posted by flotus1

With CCM+, we can connect to simulations running on compute nodes without a GPU. I would be surprised if that was not possible with Fluent.

Leaving half the memory channels empty on a fancy new machine for 20000€ stings a bit.
It won't be as bad, since that still yields the same theoretical maximum bandwidth as last-gen. But then you might as well save the money and buy last gen.

Thank you Alex. I get the point, I will fill all the slots with 16Gb or 32Gb. I want to go with genoa 2x9354. I have found out today there are only 24 slots per 2 cpu on genoa vs 32 slots per 2 cpu on milan servers (Lenovo and Dell, didn't check others). 1DPC has better performance than 2DPC and they don't have a physical place on the server I suppose this is the reason.

[flotus1]

If rumors are to be believed, AMD is working on supporting 2DPC on Genoa.
For Milan, motherboards with both 1DPC ans 2DPC exist. I'd say the majority of dual-socket boards come with "only" 16 DIMM slots.
In the server market, this is probably less about performance. There just aren't too many customers that actually need 32 DIMMs, i.e. more than 4TB of RAM.