Best HW for ANSYS 2 HPC in CFX and Mechanical
 

Hello Everybody,

First, thanks o lot for all the informations there are on this forum. :)

We are working since more than a decade with ANSYS CFX and MECHANICAL (more often iterative solver so without AVX, than sparse solver).
Our design office uses 2 HPC so the maximum number of useable cores for us is 36.
The cluster we have (3 x dual socket Xeon 6 cores so exactly 36 cores) since a little bit more than 10 years works well but time is coming to upgrade our hardware for speedup.
The linearity of our cluster is very good, thanks a good balance between RAM bandwidth and CPU performance. :cool:

We know that the ideal hardware for FEA is a little bit in opposite with the CFD needs but it is not reasonable to have 2 hardwares for each type of calculation. :(

1/ First question :

Some examples of potential hardwares that could match our needs :

- 1 dual socket Xeon 6242R (2 x 20) : 12 memory channels
- 1 dual EPYC 7F72 (2 x 24) : 16 memory channels
- 1 dual EPYC 7F52 (2 x 16) : 16 memory channels but 4 cores less than our 36 cores licence capacity... but more L3 Cache 256MB instead of 192MB for the 7F72 and more TDP and clockrate CPU

What is your opinion with these 3 configurations that are the "standard" proposal for professionnal needs with 2 HPC today ?
I personnaly think that the dual EPYC 7F52 could be the best choice for MECHANICAL but will this configuration be slower in CFX than the dual EPYC 7F72 because of the lack of 4 cores ?

Page 22 of this presentation, we can see that EPYC Rome is a good candidate for CFX and not a bad CPU for MECHANICAL (with the iterative solver, EPYC is a little bit faster than INTEL and for sparse solver it is pretty much the same...) :

https://www.cmc.ca/wp-content/upload...Mechanical.pdf

I think an F EPYC can only be better for MECHANICAL than the 7502 that is present in the document ?

2/ Second question :

If an EPYC 74F3 would be available before this summer, would be judicious to wait the zen 3 generation ?
I think the next XEON SP will be available after this summer, no ?

3/ Third question :

We also have a look on a "no standard" approach :

3 x High End mainstream CPU in an Infiniband cluster, for example 3 x 12 cores.

In this way, the most important spec nice to have is : frequency for MECHANICAL and memory bandwidth for CFX.

- 3 x i9 9920X or 10920X overclocked (some company makes very good configurations and warranty) : same memory channels than a dual Xeon 6242R.
- 3 x Threadripper Pro 3945WX : 3 x 8 memory channels but only 2 CCD per CPU, I don't know if this will make an important bottleneck (sure there will be a bottlneck but we have only 1.5 core per memory channel...) ?

I think that the 3 x 3945WX could be the best setup for MECHANICAL.
Is there a chance that this setup could be as fast as a dual EPYC 7F72 in CFX ?

We know that ECC memory is unfortunately not possible with this cluster.

Thanks a lot for the future answers and every different proposal for our future hardware is welcome. ;)

This is the key point here. I would not buy last-gen at this point.
Third gen AMD Epyc 7Fxx CPUs are way superior compared to second gen. And they should already be available in OEM/SI systems, since I can already buy them separately.
Same for Intel 3rd gen "scalable" CPUs. For applications like FEA and CFD, they offer a substantial improvement compared to second gen. Same here, they are already available.

It's more or less a coin-toss which CPU manufacturer you go with. It will probably come down to which one your OEM/SI offers, or which offer is cheaper.
Low core count Thradripper Pro CPUs are a no-go in my opinion. Without at least 4 dies enabled, they are no better than a regular 2nd gen Threadripper CPU. And get beaten by third gen Epyc 7Fxx CPUs in pretty much every metric. AMD copied a page out of Intels playbook, titled "unnecessary market segmentation"

Thank you flotus1 for your answer. :)

I saw in an other post here that with EPYCs, the number of CCDs is important for scalability.
The 7F72 has 6 CCDs, so a 36 cores calculation is optimal (3 threads x 6 per socket).
Do you think, the 74F3, with 8 CCDs (2 x 8 per socket = 32 cores or 3 x 8 per socket = 40 cores) will be faster with 36 cores than the 7F72 ?

Thank you by advance.

Yes, it will be faster.
In terms of "threads per CCD", the maximum counts. Which in both cases is 3. Those CCDs that only got 2 threads will just finish their tasks faster, and then wait for the other threads to catch up.
Comparing Epyc 7F72 to 74F3, we are looking at a generation difference of Zen2 vs. Zen3. The latter will provide somewhere around 20-30% higher performance. The individual cores are faster (even clock for clock), and the infinity fabric finally runs at a 1:1 ratio with DDR4-3200 memory.

Perfectly clear. Thank you again. :)

Since we are licence limited (36 cores), if we take a look on the EPYC Rome generation, what would you suggest between :

--> dual 7F72
--> dual 7F52

With the 7F52 we will loose 4 cores but we will add 33% more CPU Cache, 0.3 gHz (base) and 0.2 gHz (Turbo).

What is your opinion for CFD and for Mechanical.
I have a little idea for Mechanical (the 7F52 could be better because of the frequency) but no idea for CFX.

If the calculation speed will be nearly the same, we will go to the 7F72 because, one day we could appreciate a HW with 48 cores...

Another question that comes directly after the previous one, do you have a frequencial stepping document (like INTEL) for AMD ?
I have no idea if a 7F72 will run at 3.7 gHz, or 3.6 gHz or 3.5 gHz with 36 cores instead of 48 (with no thermal problem).
And I have also no idea what will the 7F52 frequency at 32 cores (with no thermal problem). :confused: