[fluidsong]

Hi All，
I was an experimentalist doing convections. Now I am trying to do some CFD and
compare the results with lab data. I plan to carry out DNS using Nek5000 or OpenFoam on a cluster.


The budget is ~100K US$, the plan is as following:


work nodes:
1.
10 nodes :
2 x E5-2680v4 (14 cores each, 2.4-3.3GHz, 35MB cache)
8*16G DDR4 2400MHz ECC REG RAM.


OR


2.
14 nodes:
2x E5-2650v4 (12 cores each, 2.2-2.9GHz, 30MB cache)
8*16G DDR4 2400MHz ECC REG RAM.


which one you think is better for the DNS task, or any other suggestions on the CPU/RAM?


And I want to thank evcelica, scipy, flotus1, bindesboll, RobertB... and many others, I learned a lot from your posts.


Many thanks!
Song

[flotus1]

Maybe you did not get an answer yet because the question is a rather tricky one.
Both options have their pros and cons and it is hard to declare a clear winner.
Option 2 is better in the total amount of cores, total amount of memory and total memory bandwidth. Option 1 has slightly faster cores and since the amount of nodes is lower the overhead for communication (both in terms of money spent on network hardware and time spent during a simulation) will be lower.
Option 2 will obviously allow you to run larger simulations in less time assuming the software has a good weak scaling. Option 1 should be slightly better in cases where strong scaling is relevant.
If both options have about the same price tag I would recommend number 2. The Xeon E5-2650 v4 has an excellent price/performance ratio for CFD applications and therefore I see no better option.

[fluidsong]

Originally Posted by flotus1

Maybe you did not get an answer yet because the question is a rather tricky one.
Both options have their pros and cons and it is hard to declare a clear winner.
Option 2 is better in the total amount of cores, total amount of memory and total memory bandwidth. Option 1 has slightly faster cores and since the amount of nodes is lower the overhead for communication (both in terms of money spent on network hardware and time spent during a simulation) will be lower.
Option 2 will obviously allow you to run larger simulations in less time assuming the software has a good weak scaling. Option 1 should be slightly better in cases where strong scaling is relevant.
If both options have about the same price tag I would recommend number 2. The Xeon E5-2650 v4 has an excellent price/performance ratio for CFD applications and therefore I see no better option.

Thanks for your reply, Alex.
I agree with your analysis, and considering the increasing RAM need, I now prefer the second.

[akidess]

If you are planning to use all nodes for a single simulation, don't forget an Infiniband interconnect. An 18-port Mellanox switch will set you back roughly $10k, and an IB-card for each node will go for something around $800.

[fluidsong]

Originally Posted by akidess

If you are planning to use all nodes for a single simulation, don't forget an Infiniband interconnect. An 18-port Mellanox switch will set you back roughly $10k, and an IB-card for each node will go for something around $800.

Thanks, Kidess.
Yes, the infiniband switcher is necessary to simulate this problem.