OpenFOAM-2.0.x and MVAPICH2

kpsl · August 9, 2011, 11:03

Dear Foamers,

I've been trying to install OF-2.0.x on a cluster running MVAPICH2.

Somehow, the install script is unable to properly locate or use the pre-installed MPI.

I get errors such as

Code:

ptscotchDecomp.C(125): catastrophic error: cannot open source file "mpi.h"

despite having set

Code:

#- MPI implementation:
#    WM_MPLIB = SYSTEMOPENMPI | OPENMPI | MPICH | MPICH-GM | HPMPI
#               | GAMMA | MPI | QSMPI
export WM_MPLIB=MVAPICH2

in the bashrc file and having added

Code:

case "$WM_MPLIB" in
MVAPICH2)
export MPI_HOME=/sw/comm/mvapich2/1.5.0-gcc
export MPI_ARCH_PATH=$MPI_HOME
_foamAddPath $MPI_ARCH_PATH/bin
_foamAddLib $MPI_ARCH_PATH/lib
export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/MVAPICH2
;;

in settings.sh.

I have also checked to see that the file mpi.h actually exists

Code:

ls $MPI_ARCH_PATH/include
mpi_base.mod       mpicxx.h  mpi.h    mpiof.h  mpi_sizeofs.mod  opa_primitives.h  opa_util.h
mpi_constants.mod  mpif.h    mpi.mod  mpio.h   opa_config.h     opa_queue.h       primitives

and also opened it in an editor just to be sure.

I have also tried calling my MPI implementation MPI-MVAPICH2 since I have heard that some scripts check for the presence of MPI in the name. If I do this I get the same errors about not being able to find mpi.h, however, I get these much sooner when building PStream and scotch rather early on. I have also tried both gcc and Icc with similar results.

I don't quite understand why the install script cannot find the neccessary files. Does anybody have any ideas?

wyldckat · August 9, 2011, 12:03

Greetings Kris,

I think you only missed one detail: in the folder wmake/rules/General you will also need a file named mplibMVAPICH2. Check the other "mplib*" files there for additional references, since you might need more than simply libmpi.so (the -lmpi entry) with MVAPICH2.

Best regards and good luck!
Bruno

kpsl · August 10, 2011, 04:16

That was it!!! Three cheers for Bruno

wyldckat · August 10, 2011, 10:15

Awww, you're making me blush

I'm glad it's working

Some other people haven't had such an easy success with SGI MPI

kpsl · August 12, 2011, 13:45

I did have some minor trouble with ptscotchDecomp failing to compile, but I don't plan on using it so I'm not that bothered

scotth2o · August 16, 2012, 13:04

I just went through this exercise for a new install of OF 2.1.x and I can confirm that the “Allwmake” script in pstream does look for a “*MPI*” string in the $WM_MPLIB variable. Also my attempt to use the mplibMPICH file as a template for MVAPICH didn’t work and I had to use the link/include flags returned by mpicc -cc= -compile-info & mpicc -cc= -link-info . This might be a system specific thing. Regardless, the information provided above was very useful.

August 9, 2011, 11:03	OpenFOAM-2.0.x and MVAPICH2	#1
kpsl New Member Kris Join Date: Nov 2010 Posts: 21 Rep Power: 15	Dear Foamers, I've been trying to install OF-2.0.x on a cluster running MVAPICH2. Somehow, the install script is unable to properly locate or use the pre-installed MPI. I get errors such as Code: ptscotchDecomp.C(125): catastrophic error: cannot open source file "mpi.h" despite having set Code: #- MPI implementation: # WM_MPLIB = SYSTEMOPENMPI \| OPENMPI \| MPICH \| MPICH-GM \| HPMPI # \| GAMMA \| MPI \| QSMPI export WM_MPLIB=MVAPICH2 in the bashrc file and having added Code: case "$WM_MPLIB" in MVAPICH2) export MPI_HOME=/sw/comm/mvapich2/1.5.0-gcc export MPI_ARCH_PATH=$MPI_HOME _foamAddPath $MPI_ARCH_PATH/bin _foamAddLib $MPI_ARCH_PATH/lib export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/MVAPICH2 ;; in settings.sh. I have also checked to see that the file mpi.h actually exists Code: ls $MPI_ARCH_PATH/include mpi_base.mod mpicxx.h mpi.h mpiof.h mpi_sizeofs.mod opa_primitives.h opa_util.h mpi_constants.mod mpif.h mpi.mod mpio.h opa_config.h opa_queue.h primitives and also opened it in an editor just to be sure. I have also tried calling my MPI implementation MPI-MVAPICH2 since I have heard that some scripts check for the presence of MPI in the name. If I do this I get the same errors about not being able to find mpi.h, however, I get these much sooner when building PStream and scotch rather early on. I have also tried both gcc and Icc with similar results. I don't quite understand why the install script cannot find the neccessary files. Does anybody have any ideas?

August 9, 2011, 12:03		#2
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,974 Blog Entries: 45 Rep Power: 128	Greetings Kris, I think you only missed one detail: in the folder wmake/rules/General you will also need a file named mplibMVAPICH2. Check the other "mplib" files there for additional references, since you might need more than simply libmpi.so* (the -lmpi entry) with MVAPICH2. Best regards and good luck! Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

August 10, 2011, 10:15		#4
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,974 Blog Entries: 45 Rep Power: 128	Awww, you're making me blush I'm glad it's working Some other people haven't had such an easy success with SGI MPI __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
OpenFOAM 1.7.x using intel compiler and MVAPICH2	kpsl	OpenFOAM Installation	8	August 12, 2011 13:44
OF 141 and MVAPICH2	msrinath80	OpenFOAM Installation	3	November 7, 2007 12:10
MVAPICH2 SIGEMT error	kumar2	OpenFOAM Running, Solving & CFD	0	May 26, 2007 18:24

August 10, 2011, 04:16		#3
kpsl New Member Kris Join Date: Nov 2010 Posts: 21 Rep Power: 15	That was it!!! Three cheers for Bruno

August 12, 2011, 13:45		#5
kpsl New Member Kris Join Date: Nov 2010 Posts: 21 Rep Power: 15	I did have some minor trouble with ptscotchDecomp failing to compile, but I don't plan on using it so I'm not that bothered

August 16, 2012, 13:04		#6
scotth2o New Member Scott Haynes Join Date: May 2009 Posts: 10 Rep Power: 16	I just went through this exercise for a new install of OF 2.1.x and I can confirm that the “Allwmake” script in pstream does look for a “MPI” string in the $WM_MPLIB variable. Also my attempt to use the mplibMPICH file as a template for MVAPICH didn’t work and I had to use the link/include flags returned by mpicc -cc= -compile-info & mpicc -cc= -link-info . This might be a system specific thing. Regardless, the information provided above was very useful.