OpenFOAM 2012 and MPI (openmpi 4.0.3)

m_a_s · April 1, 2021, 16:07

Has anyone had any problems with running OF with open mpi 4.0.3.

I had been using OF1912 which came with Open MPI 1.10.7.
Today I tried using OF2012 which comes with Open MPI 4.0.3.

I am getting the "There are not enough slots available in the system...." and it appears that the number of slots the Open MPI sees is just the number of cores, not threads. Same decomposeParDict.

I am using Ubuntu 20.04 on two different boxes. One is a native install, the other is WSL2 on a Win10 box. Both boxes are AMD Threadripper workstations.

Thoughts?

KTG · April 1, 2021, 20:21

I don't think you will see a speedup if you run the number of processors equal to the number of threads, rather than cores, but you should try it and report back to us! I think you can do this by running, for example with 4 cores and 2 threads each:

mpirun -np 8 --bind-to hwthread

verses:

mpirun -np 4 --bind-to core

I could be wrong, but give it a try and see what happens.

m_a_s · April 1, 2021, 23:17

results:
> np 32 / hwthread: 3m-17s
> np 16 / core: 2m-38s

Wow!

KTG · April 1, 2021, 23:21

Yay! Now you can move on with your life until the next problem!

DominikMuetter · April 2, 2021, 04:56

You can also oversubscribe the number of slots in openmpi with "mpirun -np x --oversubscribe"

April 1, 2021, 16:07	OpenFOAM 2012 and MPI (openmpi 4.0.3)	#1
m_a_s New Member Mark Join Date: Jun 2017 Posts: 4 Rep Power: 8	Has anyone had any problems with running OF with open mpi 4.0.3. I had been using OF1912 which came with Open MPI 1.10.7. Today I tried using OF2012 which comes with Open MPI 4.0.3. I am getting the "There are not enough slots available in the system...." and it appears that the number of slots the Open MPI sees is just the number of cores, not threads. Same decomposeParDict. I am using Ubuntu 20.04 on two different boxes. One is a native install, the other is WSL2 on a Win10 box. Both boxes are AMD Threadripper workstations. Thoughts?

April 1, 2021, 23:21		#4
KTG Senior Member Abe Join Date: May 2016 Posts: 119 Rep Power: 10	Yay! Now you can move on with your life until the next problem! DominikMuetter likes this.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
openFOAM mpirun error on cluster	cheng1988sjtu	OpenFOAM Running, Solving & CFD	1	November 14, 2012 22:13

April 1, 2021, 20:21		#2
KTG Senior Member Abe Join Date: May 2016 Posts: 119 Rep Power: 10	I don't think you will see a speedup if you run the number of processors equal to the number of threads, rather than cores, but you should try it and report back to us! I think you can do this by running, for example with 4 cores and 2 threads each: mpirun -np 8 --bind-to hwthread verses: mpirun -np 4 --bind-to core I could be wrong, but give it a try and see what happens.

April 1, 2021, 23:17		#3
m_a_s New Member Mark Join Date: Jun 2017 Posts: 4 Rep Power: 8	results: > np 32 / hwthread: 3m-17s > np 16 / core: 2m-38s Wow!

April 2, 2021, 04:56		#5
DominikMuetter New Member Dominik Mütter Join Date: Mar 2021 Location: Europe Posts: 4 Rep Power: 5	You can also oversubscribe the number of slots in openmpi with "mpirun -np x --oversubscribe"