MPI nodes limit

badoumba · April 2, 2017, 05:41

Hi friends,

I found this report about testing OF on Raspberry.
https://krex.k-state.edu/dspace/bits...pdf?sequence=1

There is an interesting conclusion at pages 32-35 explaining the reasons beyond the increase in computational time when the job is shared between more than 4 nodes.

Well, I feel concern about this as I was considering equip myself with a 64 Intel cores rack. So do we have to stick to the 4 nodes in our decomposeParDict file? This would mean that with my happy 64 cores, I would simply be able to run more cases simultaneously, not solving one case at the speed of light.

Has anyone some feedback about MPI distribution rules?

Happy sunday!

badoumba · April 2, 2017, 18:20

Well, after further readings
http://www.prace-ri.eu/IMG/pdf/Curre...Clusters-2.pdf
http://www.hpcadvisorycouncil.com/pd...015_PakLui.pdf
https://blog.syncious.com/2017/03/24...-hpc-clusters/
I think I mixed cores and clusters. Actually the first experiment was made with RP1 with only 1 core. So 1 cluster = 1 core and the bottleneck is indeed the Ethernet cable as intercommunication between cores is required in OpenFOAM.

From these studies, I understand that as long as we have RDMA clusters with InfinityBand connections, we simply need to carefully select the number of cores we attribute according to the mesh size (not solving a medium size project in a hight number of cores).

hokhay · June 26, 2017, 21:13

Interesting to know RP can be used for Openfoam as well. Thanks for sharing

Sent from my LG-H818 using CFD Online Forum mobile app

April 2, 2017, 05:41	MPI nodes limit	#1
badoumba Member badoumba Join Date: Aug 2013 Posts: 68 Rep Power: 12	Hi friends, I found this report about testing OF on Raspberry. https://krex.k-state.edu/dspace/bits...pdf?sequence=1 There is an interesting conclusion at pages 32-35 explaining the reasons beyond the increase in computational time when the job is shared between more than 4 nodes. Well, I feel concern about this as I was considering equip myself with a 64 Intel cores rack. So do we have to stick to the 4 nodes in our decomposeParDict file? This would mean that with my happy 64 cores, I would simply be able to run more cases simultaneously, not solving one case at the speed of light. Has anyone some feedback about MPI distribution rules? Happy sunday!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
mpirun, best parameters	pablodecastillo	Hardware	18	November 10, 2016 12:36
MPI code on multiple nodes, scalability and best practice	t.teschner	Hardware	0	October 7, 2014 05:07
Error using LaunderGibsonRSTM on SGI ALTIX 4700	jaswi	OpenFOAM	2	April 29, 2008 10:54
Is Testsuite on the way or not	lakeat	OpenFOAM Installation	6	April 28, 2008 11:12
CFX4.3 -build analysis form	Chie Min	CFX	5	July 12, 2001 23:19

April 2, 2017, 18:20		#2
badoumba Member badoumba Join Date: Aug 2013 Posts: 68 Rep Power: 12	Well, after further readings http://www.prace-ri.eu/IMG/pdf/Curre...Clusters-2.pdf http://www.hpcadvisorycouncil.com/pd...015_PakLui.pdf https://blog.syncious.com/2017/03/24...-hpc-clusters/ I think I mixed cores and clusters. Actually the first experiment was made with RP1 with only 1 core. So 1 cluster = 1 core and the bottleneck is indeed the Ethernet cable as intercommunication between cores is required in OpenFOAM. From these studies, I understand that as long as we have RDMA clusters with InfinityBand connections, we simply need to carefully select the number of cores we attribute according to the mesh size (not solving a medium size project in a hight number of cores).

June 26, 2017, 21:13		#3
hokhay Member Join Date: Nov 2014 Posts: 92 Rep Power: 11	Interesting to know RP can be used for Openfoam as well. Thanks for sharing Sent from my LG-H818 using CFD Online Forum mobile app