[flotus1]

Right now there are two retailers in Germany that have 2 different dual-socket Epyc boards listed. They are not in stock, but the date of delivery is 11.08.2017.
Additionally, a single Epyc processor (AMD EPYC 7351, 16 cores) is available. Compatible coolers are already announced, e.g. from Noctua.
Skylake-SP is a different story. The CPUs have been widely available for quite a while now, but no dual-socket boards are in sight yet. Compatible coolers
It should be a matter of days until you can build your own Epyc system from retail parts.

[naveedgt]

Based on the following benchmarks for OpenFoam, although limited on the information of the setup, it appears Threadripper in this case is quite inferior to to the Intel competition.

http://pclab.pl/art75073-19.html

Not sure how applicable this is to the Xeon/Epyc conversation, but I assume its quite relevant.

[hpvd]

pretty interesting!

what might be the reasons for this? Inter Die Latency?

Or simply a non optimized setup/config...

[Kaskade]

According to Google Translate the author calls OpenFOAM a FEM-tool. So chances are, while he got interFoam running (which can be daunting at first), he may lack some knowledge of simulation.

As long as we don't know the exact nature of the test case, I would take this benchmark with a large grain of salt.

[Goh77]

Tom's hardware benchmarks

http://www.tomshardware.com/reviews/...u,5167-12.html

[RobertB]

@naveedgt

The 4 memory channel TR will suffer against the 6 memory channel Intels.

Epyc has 8 memory channels and thus will tend to win - having more cores probably helps too.

Someone else posted

www.youtube.com/watch?v=gdYYRRDJDUc

but the Intels here only have four memory channels and thus would be expected to perform worse.

In general it would seem both manufacturers have put out a series of CPUs that will perform significantly better on CFD applications in this generation.

Now if only our understanding progressed as fast.....

[flotus1]

The 4 memory channel TR will suffer against the 6 memory channel Intels

The TR counterpart from Intel (X299 platform, leaving out Kaby-Lake X) also has a 4-channel memory controller.

[hpvd]

Originally Posted by flotus1

The TR counterpart from Intel (X299 platform, leaving out Kaby-Lake X) also has a 4-channel memory controller.

just an addition: only Xeon SP have 6-Channel

[dybuk]

Originally Posted by Kaskade

According to Google Translate the author calls OpenFOAM a FEM-tool. So chances are, while he got interFoam running (which can be daunting at first), he may lack some knowledge of simulation.

As long as we don't know the exact nature of the test case, I would take this benchmark with a large grain of salt.

They probably use damBreak tutorial from OpenFoam (based on they description of the case). If cell count from tutorial was not changed(from original 2268 cells) whole analysis fits in L3, maybe even in L2 cache on AMD and Intel systems. If so, pclab benchmark is far away from typical simulation.
I will ask pclab for more information.

[chad]

Out of curiosity, has anyone got their hands on the new AMD CPUs?

I will hopefully be ordering the 7281 and a Supermicro board in the coming weeks but was wondering if anyone had experience with them yet.

[Kaskade]

Ansys released a White Paper in which they directly compare the per-core-performance of a Xeon Gold 6148 to the previous two generations for a simulation on 32 cores in total: http://www.ansys.com/Resource-Librar...ssor?tli=en-us

Skylake-SP is faster of course, but despite 50% more memory channels and faster memory it is "only" between 15% and 36% faster.

So does that mean that the memory bandwidth stops being the bottleneck and the simulations become CPU limited? Meaning that you would profit from a higher clock rate? Or that there is still room for a boost due to the implementation of AVX-512 support in Fluent?

(There is a second white paper about the benefits of OPA instead of EDR: http://www.ansys.com/Resource-Librar...s-fluent-intel)

[flotus1]

I guess this document is based on the same information, but readily available https://www.intel.com/content/dam/ww...uent-brief.pdf
I would interpret the results in a different way. 35% more performance per core is pretty substantial.
Remember: when talking about "memory bottlenecks" this does not mean that performance is entirely limited by memory performance. You will get a disproportional performance increase from using higher core count CPUs. And since these high core count CPUs are disproportionally more expensive they are usually not worth it. Especially when on an expensive "per-core" licensing model for for commercial CFD packages.
Maybe look at it this way: in order to get a similar improvement in performance per core on a 32-core license with Broadwell-EP, you would have to replace a single dual-socket workstation (2x16cores) with two nodes (2x8 cores each). That would be quite expensive and inconvenient.
To conclude: I don't think CFD will stop being memory sensitive. The gap between memory and raw CPU performance will continue to widen. We will see "new" technologies to compensate this in the future, e.g. another level of cache or HBM in the CPU package...

[Kaskade]

It's "up to" 35%. And the price pro core does not really increase with core number if you correct for the influence of clock frequency.

Dell called it 'saturation' in their benchmarks. In the sense of diminishing returns. In the end it's about deciding how much performance one needs and can afford.

[hpvd]

just some more benchmarks of cheap 2S Epycs in comparison to some others (more expensive Epycs, XEON SPs...)
https://www.servethehome.com/dual-am...nsive-2p-epyc/

I'm not quite sure yet if one can take any of these numbers to compare CFD-performance

[flotus1]

I finally got to do a quick&dirty performance comparison for one of our our in-house CFD codes.
Linux kernel versions were not the same and testing had to be done really quick without diving too much into details, so the numbers are just a rough estimate.
The code is an OpenMP parallel Lattice Boltzmann solver, so scaling on large numbers of cores is not ideal.

Intel System: 2x Xeon E5-2650v4 (2x12 cores), 8x16GB DDR4-2400 dual-rank, SMT off
time per iteration on 22 cores: ~63s

AMD System: 2x Epyc 7301 (2x16 cores), 16x16GB DDR4-2133 dual-rank, SMT off
time per iteration on 16 cores of one CPU: ~68s
time per iteration on 32 cores, both CPUs: ~43s

Especially when keeping in mind that the AMD system will translate higher memory speeds (up to DDR4-2666 supported) into better performance, these results are quite impressive.
At least promising enough for me to make the decision for my next workstation: -> dual AMD Epyc 7301.
Edit: There will be a more in-depth comparison once my new Epyc workstation is deployed.

[eric]

Thanks for sharing, flotus! Will be interesting to see the in-depth benchmarks. Did you end up going for the build you posted in this thread? I am looking at something similar, but looks like no one here has the Epyc processors in stock yet.

[flotus1]

It will be similar to this setup.
The main difference in terms of performance is memory: I currently only have enough DDR4-2133 memory to fill all the slots and no plans to upgrade with the current memory prices. First thing I will try is running them at DDR4-2400 which would match AMDs (Supermicros) official specifications for dual-rank RDIMM.
Extrapolating benchmark results I have seen for Ryzen CPUs this should even yield slightly better performance than DDR4-2666 single rank.

I ended up ordering the CPUs from Alternate, one of the largest hardware retailers here in Germany. The CPUs are available but not in stock, they order them from their supplier once a customer places an order. Hence the 9 days of delivery. Fingers crossed, the status went to "not available" again today
One comparison that Intel clearly wins: availability of both CPUs and motherboards.

Edit: allegedly the CPUs should be ready to ship on Friday.

[naffrancois]

Many thanks flotus for sharing your benchmarks, I am really interested to see how it goes

[flotus1]

So am I. CPUs should arrive today, motherboard perhaps next week. Who knows
Shopping for AMD parts is quite an adventure.

Edit: CPUs arrived last thursday. Motherboard came today. I got a different version than I ordered because of supply problems, but I stopped being picky.

[tombo]

What motherboard did they end up bringing you?

You may have seen this: https://www.anandtech.com/show/12084...ur-analysis-/4

An interesting comparison considering the source, but I'd like to see the comparison between the CPUs you chose or perhaps the 24c, they probably won't do much worse that the 7601 of memory bandwidth is limiting.