CFD Online Discussion Forums - [OpenFOAM.org] OpenFOAM 3.01 on Centos 5.11

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM Installation (https://www.cfd-online.com/Forums/openfoam-installation/)

- - [OpenFOAM.org] OpenFOAM 3.01 on Centos 5.11 - MPI Issue (https://www.cfd-online.com/Forums/openfoam-installation/176872-openfoam-3-01-centos-5-11-mpi-issue.html)

OpenFOAM 3.01 on Centos 5.11 - MPI Issue

Have been struggling with this for some time. If anyone has any thoughts would be appreciated.

I've been running 3.0.1 on Centos 5.11 successfully on a number of nodes here. Today I encountered one issue with a node that has Intel composer xe installed on it and is defaulting to intels MPI.

All my other nodes run on OpenMPI with success and I would like to execute of301 on this node using the same config.

mpi-selector --list returns:
lam-i386
lam-x86_64
openmpi-1.4-gcc-x86_64

mpi-selector --query returns:
default: openmpi-1.4-gcc-x86_64
level:system

but

mpirun -which returns
/opt/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64/mpirun
and throws an error saying /opt/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64/mpivars.sh: No such file or directory.

Foam is configured OpenFOAM-3.0.1/etc/bashrc configures WM_MPLIB to SYSTEMOPENMPI. I have tried setting it to OPENMPI but with no luck.

but after initializing with of301 typing $WM_MPLIB at the terminal returns bash: SYSTEMOPENMPI: command not found.

I note in the path statement that the intel path statements appear before the/usr/lib64/openmpi/1.4-gcc/bin directory.

I'd rather not uninstall the intel compiler but am unsure how to force the system to see openMPI as the default.

Any suggestions appreciated.

Britt

Quick answer: Looks like Intel's path is loaded automatically into the PATH variable automatically.

The trick I usually suggest on the wiki for OpenFOAM on Ubuntu, is to use the following symbolic linking commands:

Code:

ln -s /usr/bin/mpicc.openmpi OpenFOAM-3.0.1/bin/mpicc

ln -s /usr/bin/mpirun.openmpi OpenFOAM-3.0.1/bin/mpirun

Therefore, in the nodes where you don't have Intel-MPI installed, you can use these commands to find the actual location of each file:

Code:

which mpirun

which mpicc

and then adjust accordingly the symbolic linking commands.

edit: I deleted you're previous thread that had this same exact question.

Thank you for the quick response. Apologies for asking again!

I followed your suggestion.

ln -s /usr/bin/mpirun.openmpi /usr/lib64/openmpi/1.4-gcc/bin/mpirun
ln -s /usr/bin/mpicc.openmpi /usr/lib64/openmpi/1.4-gcc/bin/mpicc

links established correctly but:
which mpirun still returns:
/opt/intel/composer_xe_2013_sp1.1.106/mpirt/bin/intel64/mpirun

but:

which mpicc is correct. /usr/lib64/openmpi/1.4-gcc/bin/mpicc

clearly still something happening with the path statements. I'll keep at it but let me know if anything else jumps out at you.

Thanks for the assistance.

Quick answer: I don't know if you've solved this issue or not, but another possible solution is to run the following:

Code:

echo PATH= /usr/lib64/openmpi/1.4-gcc/bin/:$PATH

If will enforce that the path to to the Open-MPI binaries is picked up first when you try to run them.

The other possible solutions that I can think of are regarding the environment variables. The following command should give us a few ideas:

Code:

export | grep MPI

There are a few variables that I can think of that may be giving you problems:

Code:

I_MPI_ROOT

MPI_ROOT

OPAL_PREFIX

the last one is commonly used for Open-MPI.
To unset them, run for example:

Code:

unset I_MPI_ROOT

@WYLDCKAT thanks for the follow up:/

After a bit of trial and error and some support from Intel I got around this by adding the following lines at the end of my .bashrc

export PATH=/usr/lib64/openmpi/1.4-gcc/bin:$PATH
export LD_LIBRARY_PATH=/usr/lib64/openmpi/1.4-gcc/lib:$LD_LIBRARY_PATH

Hope this helps anyone who encounters similar issues.