CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)

- OpenFOAM (http://www.cfd-online.com/Forums/openfoam/)

- - mpirun fails (http://www.cfd-online.com/Forums/openfoam/93579-mpirun-fails.html)

I'm trying to install OpenFOAM 1.7.1 on a RedHat EL 5.5 machine.
After installing the ThirdParty-1.7.1 stuff ending in
========================================
Done ThirdParty Allwmake
========================================

when trying to run e.g. mpirun w/o any argument I get:

mpirun: error while loading shared libraries: .../OpenFOAM/ThirdParty-1.7.1/platforms/linux64Gcc/openmpi-1.4.1/lib/libopen-rte.so.0: ELF file OS ABI invalid

Any idea what went wrong here?

Greetings Achim and welcome to the forum!

Check your Linux-machine architecture:

Code:

uname -m

If it shows "i686" or similar (i386/486/586), then the problem you are having is because you didn't set the variable WM_ARCH_OPTION to 32 in the file "OpenFOAM-1.7.1/etc/bashrc" or "OpenFOAM-1.7.1/etc/cshrc".

Best regards,
Bruno

Quote:

Originally Posted by wyldckat (Post 328700)

Greetings Achim and welcome to the forum!

Check your Linux-machine architecture:

Code:

uname -m

Thanks Bruno,

my uname -m is x86_64.

Starting a trivial hello world mpi program ends up with:
ipath_wait_for_device: The /dev/ipath device failed to appear after 30.0 seconds: Connection timed out

so it seems that the IB support is not built in the openmpi correctly.

Hi Achim,

Mmm, that's strange... The error message "ELF file OS ABI invalid" indicates that the problem is something like running a 64bit application in a 32bit processor or vice-versa. And the solution I described was related to the most usual occurrence of this problem.
The other possibility is if you are trying to build an application in x86_64 to run in a IA64 (Itanium) or something like that.

As for IB support, you have two options:

Use the Open-MPI version that comes with your Linux distribution, which should be already optimized for your machines. For that, "WM_MPLIB=SYSTEMOPENMPI" should do the trick.
Build (again) Open-MPI that comes with OpenFOAM, but this time edit the file "ThirdParty-*/Allwmake" and search for the area for building Open-MPI and find the block of code:

Code:

# Infiniband support # if [ -d /usr/local/ofed -a -d /usr/local/ofed/lib64 ] # then # configOpt="$configOpt --with-openib=/usr/local/ofed" # configOpt="$configOpt --with-openib-libdir=/usr/local/ofed/lib64" # fi

In bold are the hash tags to be removed, so it will build with IB.

If you check that "Allwmake" script, you'll find out that if you remove the files "$MPI_ARCH_PATH/lib/libmpi.*", you'll force it to rebuild Open-MPI with the new options :)

This might also come in handy:

Quote:

Originally Posted by pkr (Post 292700)

When using MPI_reduce, the OpenMPI was trying to establish TCP through a different interface. The problem is solved if the following command is used:
mpirun --mca btl_tcp_if_exclude lo,virbr0 -hostfile machines -np 2 /home/rphull/OpenFOAM/OpenFOAM-1.6/bin/foamExec interFoam -parallel

The above command will restrict MPI to use certain networks (lo, vibro in this case).

Best regards,
Bruno

Quote:

Originally Posted by wyldckat (Post 328801)

Use the Open-MPI version that comes with your Linux distribution, which should be already optimized for your machines. For that, "WM_MPLIB=SYSTEMOPENMPI" should do the trick.
Build (again) Open-MPI that comes with OpenFOAM, but this time edit the file "ThirdParty-*/Allwmake" and search for the area for building Open-MPI and find the block of code:

Code:

# Infiniband support # if [ -d /usr/local/ofed -a -d /usr/local/ofed/lib64 ] # then # configOpt="$configOpt --with-openib=/usr/local/ofed" # configOpt="$configOpt --with-openib-libdir=/usr/local/ofed/lib64" # fi

In bold are the hash tags to be removed, so it will build with IB.

Thanks Bruno,

now, the openmpi installation seems to be in a non-standard place and I don't have root rights.
# Is there a way to specify the location of the open-mpi installation?

I have:
/usr/mpi/gcc/openmpi-1.4.3 $ ls -l
total 28
drwxr-xr-x 2 root root 4096 Aug 10 15:59 bin
drwxr-xr-x 2 root root 4096 Aug 10 15:59 etc
drwxr-xr-x 4 root root 4096 Aug 10 15:59 include
drwxr-xr-x 3 root root 4096 Aug 10 15:59 lib64
drwxr-xr-x 2 root root 4096 Aug 10 15:59 openmpi_gcc-1.4.3
drwxr-xr-x 5 root root 4096 Aug 10 15:59 share
drwxr-xr-x 5 root root 4096 Aug 10 15:59 tests

How can I tell openfoam to use this installation?

Regards
Achim

Quote:

Originally Posted by achim (Post 328849)

------------------------------------------------------------------------

So far I edited etc/settings.sh:

...
# Communications library
# ~~~~~~~~~~~~~~~~~~~~~~

unset MPI_ARCH_PATH MPI_HOME

case "$WM_MPLIB" in
OPENMPI)
# mpi_version=openmpi-1.4.1
mpi_version=openmpi-1.4.3
# export MPI_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$mpi_version
export MPI_ARCH_PATH=/usr/mpi/gcc/openmpi-1.4.3

# Tell OpenMPI where to find its install directory
...

and etc/bashrc:

# WM_MPLIB = SYSTEMOPENMPI | OPENMPI | MPICH | MPICH-GM | HPMPI | MPI | QSMPI
# : ${WM_MPLIB:=OPENMPI}; export WM_MPLIB
: ${WM_MPLIB:=SYSTEMOPENMPI}; export WM_MPLIB

and - it seems to work ... (I can test currently only on one node, will get more during the weekend.)

Thanks a lot Bruno!

Hi Achim,

You're welcome!
But it's interesting... you made a redundant fix ;) By choosing SYSTEMOPENMPI, it automatically tries to detect where Open-MPI is installed. But since you also modified in "settings.sh" the entry OPENMPI, you can now probably use either mode, may it be WM_MPLIB=OPENMPI or SYSTEMOPENMPI :D

Good luck with setting up all nodes!
Bruno