mpirun fails
 

I'm trying to install OpenFOAM 1.7.1 on a RedHat EL 5.5 machine.
After installing the ThirdParty-1.7.1 stuff ending in
========================================
Done ThirdParty Allwmake
========================================

when trying to run e.g. mpirun w/o any argument I get:

mpirun: error while loading shared libraries: .../OpenFOAM/ThirdParty-1.7.1/platforms/linux64Gcc/openmpi-1.4.1/lib/libopen-rte.so.0: ELF file OS ABI invalid

Any idea what went wrong here?

Greetings Achim and welcome to the forum!

Check your Linux-machine architecture:If it shows "i686" or similar (i386/486/586), then the problem you are having is because you didn't set the variable WM_ARCH_OPTION to 32 in the file "OpenFOAM-1.7.1/etc/bashrc" or "OpenFOAM-1.7.1/etc/cshrc".

Best regards,
Bruno

uname -m
 

Thanks Bruno,

my uname -m is x86_64.

Starting a trivial hello world mpi program ends up with:
ipath_wait_for_device: The /dev/ipath device failed to appear after 30.0 seconds: Connection timed out

so it seems that the IB support is not built in the openmpi correctly.

Hi Achim,

Mmm, that's strange... The error message "ELF file OS ABI invalid" indicates that the problem is something like running a 64bit application in a 32bit processor or vice-versa. And the solution I described was related to the most usual occurrence of this problem.
The other possibility is if you are trying to build an application in x86_64 to run in a IA64 (Itanium) or something like that.

As for IB support, you have two options:

• Use the Open-MPI version that comes with your Linux distribution, which should be already optimized for your machines. For that, "WM_MPLIB=SYSTEMOPENMPI" should do the trick.
• Build (again) Open-MPI that comes with OpenFOAM, but this time edit the file "ThirdParty-*/Allwmake" and search for the area for building Open-MPI and find the block of code:
  
  Code:

  ---

          # Infiniband support
        # if [ -d /usr/local/ofed -a -d /usr/local/ofed/lib64 ]
        # then
        #    configOpt="$configOpt --with-openib=/usr/local/ofed"
        #    configOpt="$configOpt --with-openib-libdir=/usr/local/ofed/lib64"
        # fi

  ---

  In bold are the hash tags to be removed, so it will build with IB.

If you check that "Allwmake" script, you'll find out that if you remove the files "$MPI_ARCH_PATH/lib/libmpi.*", you'll force it to rebuild Open-MPI with the new options :)

This might also come in handy:Best regards,
Bruno

open mpi
 

Thanks Bruno,

now, the openmpi installation seems to be in a non-standard place and I don't have root rights.
# Is there a way to specify the location of the open-mpi installation?

I have:
/usr/mpi/gcc/openmpi-1.4.3 $ ls -l
total 28
drwxr-xr-x 2 root root 4096 Aug 10 15:59 bin
drwxr-xr-x 2 root root 4096 Aug 10 15:59 etc
drwxr-xr-x 4 root root 4096 Aug 10 15:59 include
drwxr-xr-x 3 root root 4096 Aug 10 15:59 lib64
drwxr-xr-x 2 root root 4096 Aug 10 15:59 openmpi_gcc-1.4.3
drwxr-xr-x 5 root root 4096 Aug 10 15:59 share
drwxr-xr-x 5 root root 4096 Aug 10 15:59 tests

How can I tell openfoam to use this installation?

Regards
Achim

------------------------------------------------------------------------

So far I edited etc/settings.sh:

...
# Communications library
# ~~~~~~~~~~~~~~~~~~~~~~

unset MPI_ARCH_PATH MPI_HOME

case "$WM_MPLIB" in
OPENMPI)
# mpi_version=openmpi-1.4.1
mpi_version=openmpi-1.4.3
# export MPI_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$mpi_version
export MPI_ARCH_PATH=/usr/mpi/gcc/openmpi-1.4.3

# Tell OpenMPI where to find its install directory
...

and etc/bashrc:

# WM_MPLIB = SYSTEMOPENMPI | OPENMPI | MPICH | MPICH-GM | HPMPI | MPI | QSMPI
# : ${WM_MPLIB:=OPENMPI}; export WM_MPLIB
: ${WM_MPLIB:=SYSTEMOPENMPI}; export WM_MPLIB

and - it seems to work ... (I can test currently only on one node, will get more during the weekend.)

Thanks a lot Bruno!

Hi Achim,

You're welcome!
But it's interesting... you made a redundant fix ;) By choosing SYSTEMOPENMPI, it automatically tries to detect where Open-MPI is installed. But since you also modified in "settings.sh" the entry OPENMPI, you can now probably use either mode, may it be WM_MPLIB=OPENMPI or SYSTEMOPENMPI :D

Good luck with setting up all nodes!
Bruno

Hi guys,

i know this is an old thread, but i have a related question:

1) I want to run with IB support on a cluster - is it enough to simply change $WM_MPLIB = SYSTEMOPENMPI in $WM_PROJECT_DIR/etc/bashrc, where the system openmpi has been compiled with IB support, or do i need to completely recompile OF as well.

I know if i want to use $WM_MPLIB = OPENMPI with IB support from ThirdParty, i need to change some switches and recompile, but if i want to use an already compiled version of openmpi with IB support, i shouldn't have to recompile anything should i???

many thanks and best regards
Jonathan

Greetings Jonathan,

If you change the option to "WM_MPLIB=SYSTEMOPENMPI", save the file, start a new terminal and run Allwmake once again. OpenFOAM's build system will only build the missing stuff, such as "libptscotch" and Pstream. Once it's done building, it should be ready to go!

Best regards,
Bruno

Hi Bruno,

Firstly - thanks for your reply and very extensive help to everyone on the forum - its much appreciated!

May i just clarify then:

1) so even though the SYSTEMOPENMPI has already been compiled by the cluster administrators with IB support, i still need to run Allwmake for OF to make some additional libraries?

Is the $WM_MPLIB variable not just an environment variable which tells OF to use the system mpi library at runtime, rather than the one built during the OF install?

Many thanks
Best regards
Jonathan

Hi Bruno,

so - i just answered my own question! :D (and confirmed your post :p)

yes, there are missing libraries which you need to build when / if you want to use SYSTEMOPENMPI.

Out of interest then, i wonder why these Pstream etc are not needed when using the bundled mpi libraries! i don't know much about it, so just learning as much as possible at this stage :)

anyway, thanks for your help - apologies for the double post!

best regards
Jonathan

The idea is that any and all MPI toolboxes that you want to build OpenFOAM with, will need "Pstream", "libptscotch" and similar libraries built accordingly. Because these are the libraries that link directly to each MPI toolbox. This is also because OpenFOAM already has these kinds of situations in mind, namely having a single installation that can operate with any number of MPI toolboxes... albeit only one MPI per shell environment.

If you run the following command, you'll see what I mean:

Hi Bruno,

Thanks very much for that explanation - i see it now.

I wonder if you could help me though with my build?

I ran as instructed, but no libraries where produced in:

/OpenFOAM/OpenFOAM-2.1.1/platforms/linux64GccDPOpt/lib/openmpi-system

and attached is the first part of my build log - as you can see, it can't find ptscotch.h - which is strange, since (it does exist), but not where OF obviously expects it to be. However, its strange since i have simply downloaded and unzipped the source code from the website.

My previous OPENMPI build went along just fine for what its worth ...

EDIT: I am not so sure now whether my first compile with OPENMPI went ok - in my /lib/openmpi-1.5.3 directory, i only have libPstream.so and libptscotchDecomp.so, not libptscotch.so as Bruno suggested. However, i have been able to run jobs over 30 -> 60 processors on the cluster which ran ok!

many thanks and regards
Jonathan

Hi Jonathan,

"ptscotch" is only needed for performing decomposition in parallel runs, such as when using snappyHexMesh in parallel, where it is able to shift cells between processors for keeping a good balance. It is not usable directly in decomposePar and it's not usually used by solvers.

As for the error you got, the block of output that matters is this one:
From it, these lines are the most important:
It's mpicc that is saying that when building the application dummysizes, it's not able to find the library "libnuma.so". This is a bit tricky, because mpicc should have also stated where this library is located, not just mention that it needed it by stating "-lnuma".
You can search for the library "libnuma.so" in your system and add the path to the folder it is in, by editing the file and adding the path to the variable "LDFLAGS"; for example, if the path was "/usr/mpi/lib", then the line would be:
If this doesn't work, you might need to ask assistance to your systems administrator, since this seems to be specific to your cluster or MPI toolbox.

Best regards,
Bruno

Dear Bruno,

thank you very much for your post:

I have tried the changes, and they are reflected in the compile log -

however, i still bet -lnuma not found errors ...

What i have realised, is that, perhaps i am not supposed to be using mpicc, but rather one of the system versions (there are a couple on the cluster (i.e. mpiCC, mpiCC-vt). I am querying the system admins as to what the differences are, however, if i look in the OF wmake rules, and run: mpicc --showme:link, it does list -lnuma.

but i dont know if that means it knows where it is?

btw, the library libnuma.so is found in the dir /usr/lib64 as shown above.

any more ideas?

thanks very much again...

best regards and thanks
jonathan

Hi Jonathan,

Run this command:
It will tell you the complete command line that will be used in place of mpicc.

For example, on my Ubuntu 12.04, the default Open-MPI mpicc gives me this:
In your case, it will likely mention "/usr/lib" but not "/usr/lib64", before "-lnuma" appears, because only some Linux Distributions use explicitly the "lib64" folder, e.g. openSUSE.

Based on Open-MPI's man page for mpicc, e.g.: http://manpages.ubuntu.com/manpages/...openmpi.1.html - it gives several hints:

• We can edit the reference wrapper file that has this list of options. You'll have to find the one for your compiler.
• We can use the output from mpicc indirectly, like this:
  
  Code:

  ---

  gcc `mpicc -showme:compile` -L/usr/lib64 `mpicc -showme:link`

  ---

  The idea is to replace the references to mpicc for the command above.
• Or simply export the value to the environment variable "LDFLAGS", before running Allwmake:
  
  Code:

  ---

  export LDFLAGS="-L/usr/lib64"

  ---

Best regards,
Bruno

thanks
 

Dear Bruno,

thanks very much for your help - i think its all sorted now.

adding -L/usr/lib64 seems to have gotten rid of the -lnuma issue, but i think the actual solution was also to run ThirdParty-2.1.1/Allwmake, and then OpenFoam-2.1.1/Allwmake.

Previously i was just running ./Allwmake in OpenFoam-2.1.1, and i was still getting a whole lot of errors.

I'm not sure what has changed, but running wmake in ThirdParty ended up building the ptscotch etc libraries which were missing. These are all now in OpenFoam/platforms/../../openmpi-system/ as well.

Running:
did as you said - and listed the correct lib locations, including /usr/lib64.

so i'm not sure what the change actually was, but as i said above (i dont know if this is possible!), running wmake in ThirdParty first and then in OpenFoam, seemed to do the trick!

I just need to test now, and check that the Infiniband is actually being used in multiple node jobs.

again, thanks very much for your help,
its much appreciated,
best regards
jonathan