Adding compute nodes - new nodes faster than existing ones?

SLC · November 13, 2019, 14:47

I've currently got a 2 node compute setup based on Skylake SP that I run ANSYS Fluent and CFX on (see Fluent benchmarks here AMD Epyc CFD benchmarks with Ansys Fluent).

Existing nodes based on Skylake SP:

2 x Intel Xeon Gold 6146, 12c, all-core turbo 3.9 GHz
Six channel memory @ 2666 MHz, theoretical bandwidth 119.21 GiB/s

I am expanding my compute setup with an additional 4 nodes. Now I have the choice of purchasing identical Skylake SP nodes as the ones I already own, or going for the the latest and greatest Cascade Lake SP release. The price is actually the same (have to purchase from Dell), irrespective of which Intel generation I go for.

Potential additional nodes based on Cascade Lake SP:

2 x Intel Xeon Gold 6246, 12c, all core turbo 4.1 GHz
Six channel memory @ 2933 MHz, theoretical bandwidth 131.13 GiB/s

Going for the Cascade Lake SP generation means there is an effective 5 % increase in all-core turbo clock speed, and a 10 % increase in theoretical memory bandwidth.

Logic dictates that I go for the newer and faster Cascade Lake setup when purchasing the 4 additional nodes, especially when the price is the same.

I will however be running the 4 nodes in parallel alongside my existing 2 nodes (adding infiniband as interconnect). Will the fact that 4 of my nodes are a smidge faster than the 2 older nodes have any significance when running a single simulation across all the nodes? I assume everything will end up running at the lowest common denominator, and that the four faster nodes will end up having to wait slightly for the two slower nodes to finish at every iteration.

I see a potential benefit to having four nodes that are faster, because I will not always be running one large simulation across all six nodes. I will at times be running one simulation on the four newer nodes, and then a second concurrent simulation on the two older nodes. So in that circumstance I will benefit from the generational speedup of the Cascade Lake release.

I appreciate your thoughts!

flotus1 · November 13, 2019, 17:23

At the same cost, I would consider this a no-brainer.
You already covered one potential benefit of having a few slightly faster nodes.
And if you are willing to put in additional effort, you might be able to adjust load balancing in a way that uses all nodes to their full potential, even for a single simulation. I.e. distributing slightly larger partitions to the faster nodes.

SLC · November 14, 2019, 12:47

Quote:

Originally Posted by flotus1

At the same cost, I would consider this a no-brainer.
You already covered one potential benefit of having a few slightly faster nodes.
And if you are willing to put in additional effort, you might be able to adjust load balancing in a way that uses all nodes to their full potential, even for a single simulation. I.e. distributing slightly larger partitions to the faster nodes.

Thanks flotus, appreciate your input!

evcelica · November 14, 2019, 15:35

I agree completely with getting the new ones.

I know in CFX you can set the relative speeds of each one of the nodes in the hostinfo.ccl file.

November 13, 2019, 14:47	Adding compute nodes - new nodes faster than existing ones?	#1
SLC Member Join Date: Jul 2011 Posts: 53 Rep Power: 14	I've currently got a 2 node compute setup based on Skylake SP that I run ANSYS Fluent and CFX on (see Fluent benchmarks here AMD Epyc CFD benchmarks with Ansys Fluent). Existing nodes based on Skylake SP: 2 x Intel Xeon Gold 6146, 12c, all-core turbo 3.9 GHz Six channel memory @ 2666 MHz, theoretical bandwidth 119.21 GiB/s I am expanding my compute setup with an additional 4 nodes. Now I have the choice of purchasing identical Skylake SP nodes as the ones I already own, or going for the the latest and greatest Cascade Lake SP release. The price is actually the same (have to purchase from Dell), irrespective of which Intel generation I go for. Potential additional nodes based on Cascade Lake SP: 2 x Intel Xeon Gold 6246, 12c, all core turbo 4.1 GHz Six channel memory @ 2933 MHz, theoretical bandwidth 131.13 GiB/s Going for the Cascade Lake SP generation means there is an effective 5 % increase in all-core turbo clock speed, and a 10 % increase in theoretical memory bandwidth. Logic dictates that I go for the newer and faster Cascade Lake setup when purchasing the 4 additional nodes, especially when the price is the same. I will however be running the 4 nodes in parallel alongside my existing 2 nodes (adding infiniband as interconnect). Will the fact that 4 of my nodes are a smidge faster than the 2 older nodes have any significance when running a single simulation across all the nodes? I assume everything will end up running at the lowest common denominator, and that the four faster nodes will end up having to wait slightly for the two slower nodes to finish at every iteration. I see a potential benefit to having four nodes that are faster, because I will not always be running one large simulation across all six nodes. I will at times be running one simulation on the four newer nodes, and then a second concurrent simulation on the two older nodes. So in that circumstance I will benefit from the generational speedup of the Cascade Lake release. I appreciate your thoughts!

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
chtMultiRegionSimpleFoam	samiam1000	OpenFOAM Running, Solving & CFD	39	March 31, 2016 08:43
chtMultiRegionSimpleFoam: strange error	samiam1000	OpenFOAM Running, Solving & CFD	26	December 29, 2015 22:14
chtMultiRegionFoam, adding a member to an existing class (regionProperties)	romain.h	OpenFOAM Running, Solving & CFD	1	October 10, 2012 15:05
Adding equations to existing solvers	treima	OpenFOAM Programming & Development	0	July 9, 2012 09:46
adding features to elements of an existing mesh?	bunni	OpenFOAM	1	November 4, 2010 13:03

November 13, 2019, 17:23		#2
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	At the same cost, I would consider this a no-brainer. You already covered one potential benefit of having a few slightly faster nodes. And if you are willing to put in additional effort, you might be able to adjust load balancing in a way that uses all nodes to their full potential, even for a single simulation. I.e. distributing slightly larger partitions to the faster nodes.

November 14, 2019, 15:35		#4
evcelica Senior Member Erik Join Date: Feb 2011 Location: Earth (Land portion) Posts: 1,167 Rep Power: 23	I agree completely with getting the new ones. I know in CFX you can set the relative speeds of each one of the nodes in the hostinfo.ccl file.