[xuegy]

Hello all,

My lab needs a 256GB workstation, running OpenFOAM and Fluent cases around 200GB. So it will be definitely memory bounded.

AMD EPYC has 8*DDR4 channels but with ccx which is an issue on Ryzen.
Intel Skylake-EP has 6*DDR4 channels.

I have just persuaded my professor to skip Broadwell and wait for the next generation. But we need the 256GB before the end of summer. I guess we won't be able to see any CFD benchmark before the decision.

[kyle]

I don't imagine the CCX latency issue will be a huge deal for CFD in two socket systems. Simulations will iterate at the speed of the slowest domain, which for both Epyc and Skylake would usually be the domain with the most inter-socket communication. We still don't know much about Skylake-EP's socket-to-socket connectivity.

Really it's a coin toss if you don't have time to wait for benchmarks.

[danbence]

Here is a video showing a speed comparison between intel and AMD EPYC on a fluent sim.

https://www.youtube.com/watch?v=gdYYRRDJDUc

[Goh77]

Not epyc but threadripper is doing great
http://www.tomshardware.com/reviews/...u,5167-12.html

[hpvd]

Originally Posted by danbence

Here is a video showing a speed comparison between intel and AMD EPYC on a fluent sim.

https://www.youtube.com/watch?v=gdYYRRDJDUc

2:18min: System Intel E5-2699v4 @ 2,2Ghz 2x22core (Broadwell)
not skylake...

[danbence]

Well we can do some rough extrapolations.

This intel paper shows Xeon Gold 6148 being ~30% faster than E5-2697v4 using fluent

https://www.intel.com/content/dam/ww...uent-brief.pdf

and we have AMD saying that EPYC is 78% faster than e5-2699v4

E5-2697v4 is 18 cores @2.3Gz
E5-2699v4 is 22 cores @2.2Gz

To be generous to Intel, say that the E5-2699v4 is 22/18=22% faster than E5-2697v4.

So Gold 6148 is 1.3/1.22=6% faster than E5-2699v4 and the implication is Epyc is 1.78/1.06= 68% faster than Gold 6148

There are bigger Xeons than the Gold 6148 (22 cores). There is the Platinum 8180 (28 cores)

28/22=27% , so EPYC has a good chance of being faster than the biggest Xeon.

Pete

[Micael]

I looked at the video demonstration with the FLUENT benchmark and did the same test with our 32-core cluster:
- 4 x (dual E5-2637v3 (4-core, 3.5 GHz), 64GB DDR4-2133)
- interconnect = FDR infiniband

See attached picture.

Result surprises me quite a lot and calls for a paradigm shift. Looks like a single EPYC-7601 32-core would be faster than my cluster by at least 75%.

[danbence]

Which of the standard tests is it ?

[Micael]

it is aircraft_wing_14m.tar in below page

https://support.ansys.com/AnsysCusto...ent+Benchmarks

Note that you need to login in Ansys customer portal

[flotus1]

I don't think this qualifies as a paradigm-shift. It is rather a sign that there is something seriously wrong with your cluster configuration. Maybe you should start by running the benchmark on a single node to see if the error is related to the infiniband configuration or the configuration of the nodes themselves.

For comparison, here is the aircraft_14m benchmark run on a single Xeon E5-1650v3 (6 cores, slightly overclocked, same generation as your cluster) taking ~27 seconds per iteration. Fluent 18.1, Windows 10
aircraft_14m.png

[Micael]

thanks for doing this test flotus1, indeed it looks there is something wrong on my side.

[Micael]

Ok, now I think I got good results. There was a warning I overlooked about cache so I used this launcher option: -cflush. Attached are results and everything is as expected, no paradigm shift

Basically, my 4-node 32-core cluster is 50% of the speed of the 2-node 128-core EPYC

[Echidna]

Originally Posted by Micael

Ok, now I think I got good results. There was a warning I overlooked about cache so I used this launcher option: -cflush. Attached are results and everything is as expected, no paradigm shift

Basically, my 4-node 32-core cluster is 50% of the speed of the 2-node 128-core EPYC

Your 32-core cluster is very close in performance to the 44-core Intel cluster...

[Micael]

Originally Posted by Echidna

Your 32-core cluster is very close in performance to the 44-core Intel cluster...

Sounds normal to me given my cluster has higher CPU frequency and memory channel per core.

[flotus1]

Of course, that's the whole point of getting a cluster with many lower core count CPUs instead of a single node with few high core count CPUs. The higher memory bandwidth (and clock speed) leads to higher performance per core. If you have to pay for Ansys licenses this pays off within the first year.
The advantage will be even higher when running in double precision. This benchmark uses single precision.

[Echidna]

Originally Posted by Micael

Sounds normal to me given my cluster has higher CPU frequency and memory channel per core.

Thanks! Maybe you could write your opinion on my thread (Threadripper vs Dual-Xeon)?

[danbence]

PADT inc are now recommending EPYC for CFD

http://www.padtinc.com/products/hard...13_04_18-1.pdf

[flotus1]

Where exactly do they mention AMD Epyc processors?
The link you gave seems to be pretty old, at least when looking at the name of the document "PADT_CubeBrochure_2013_04_18-1.pdf" and the outdated "Kepler"-based Nvidia graphics cards.

[danbence]

Good point