[wyldckat]

OK, then there must be a conflict of MPI versions... what do these commands output:

mpirun --version
echo $FOAM_MPI
echo $PINC
echo $PLIBS
echo $MPI_ARCH_PATH

[linch]

Hi,

sorry, I didn't notice your answer because of the new page.

mpirun (Open MPI) 1.5.3

echo $FOAM_MPI
openmpi-system

echo $PINC
-I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread

echo $PLIBS
-pthread -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl

echo $MPI_ARCH_PATH
/usr/lib/openmpi

Best,
Ilya

[wyldckat]

Hi Ilya,

OK, then the "LD_LIBRARY_PATH" variable must be contaminated - what does this output :

echo $LD_LIBRARY_PATH

If you can see the "local/mpich" path appear somewhere in the middle of that variable, then that would be the reason for this whole problem!

Best regards,
Bruno

[linch]

Hi,
here it comes:

echo $LD_LIBRARY_PATH
/home/shevchuk/OpenFOAM/ThirdParty-2.1.x/platforms/linux64Gcc/paraview-3.12.0/lib/paraview-3.12:/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/openmpi-system:/home/shevchuk/OpenFOAM/ThirdParty-2.1.x/platforms/linux64GccDPOpt/lib/openmpi-system:/usr/lib/openmpi/lib:/home/shevchuk/OpenFOAM/shevchuk-2.1.x/platforms/linux64GccDPOpt/lib:/home/shevchuk/OpenFOAM/site/2.1.x/platforms/linux64GccDPOpt/lib:/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib:/home/shevchuk/OpenFOAM/ThirdParty-2.1.x/platforms/linux64GccDPOpt/lib:/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/dummy:/opt/intel/Compiler/11.1/069/lib/intel64:/usr/lib/:/usr/lib64/:/opt/intel/Compiler/11.1/069/lib/intel64/:/usr/local/mpich/lib/

You're right. How can I clean it?

Best regards,
Iyla

[wyldckat]

Hi Ilya,

Took me a while to get back to you... OK, edit OpenFOAM's "etc/bashrc" file once again and search for these lines:

#- Clean LD_LIBRARY_PATH
cleaned=`$foamClean "$LD_LIBRARY_PATH"` && LD_LIBRARY_PATH="$cleaned"

Add below those two lines this new one:

cleaned=`$foamClean "$LD_LIBRARY_PATH" "/usr/local/mpich/lib/"` && LD_LIBRARY_PATH="$cleaned"

I think taking out the "intel64" paths is also a good idea, for which you can use instead this line (I hope I'm not mistaken on this one):

cleaned=`$foamClean "$LD_LIBRARY_PATH" "/usr/local/mpich/lib/:/opt/intel/Compiler/11.1/069/lib/intel64:/opt/intel/Compiler/11.1/069/lib/intel64/"` && LD_LIBRARY_PATH="$cleaned"

If it doesn't work, use spaces instead of ":".

Again, better start a new terminal. Rebuilding only "Pstream" is also a good idea, after you check if the variable mentioned before is cleaned.

If you still have problems, the environment variable "PATH" is also a suspect

Best regards,
Bruno

[linch]

Hi Bruno,

thanks a lot, step by step we are getting closer. After cleaning the LD_LIBRARY_PATH the error reduced to

mpirun: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory

Then I cleaned the PATH varible:

cleaned=`$foamClean "$PATH" "/usr/local/mpich/bin"` && PATH="$cleaned"
cleaned=`$foamClean "$PATH" "/opt/intel/Compiler/11.1/069/bin/intel64"` && PATH="$cleaned"

But now there is no mpi-path left and mpirun can't be found any more Now we have to add the "right" PATH, don't we? But which one is the right one?

Best regards,
Ilya

[wyldckat]

Do any of these appear:

which mpirun
which mpicc
ls -l `which mpirun`
ls -l `which mpicc`
mpirun --version
mpicc --help

Number 3 and 4 might reveal if there is a symbolic link.

If not, then try building again OpenFOAM's own Open-MPI 1.5.3
Don't forget to switch back from SYSTEMOPENMPI to OPENMPI in "etc/bashrc".

[linch]

mpirun -> negative; but mpicc (2,4,6) is still there.

ls -l `which mpicc`
lrwxrwxrwx 1 root root 23 3. Aug 2011 /usr/bin/mpicc -> /etc/alternatives/mpicc

Is it a good or a bad news? Should I still try to recompile the OF-internal openmpi?

[wyldckat]

Try these again, still with the SYSTEMOPENMPI:

Originally Posted by wyldckat

OK, then there must be a conflict of MPI versions... what do these commands output:

mpirun --version
echo $FOAM_MPI
echo $PINC
echo $PLIBS
echo $MPI_ARCH_PATH

edit: Additionally, if you can:

sudo update-alternatives --config mpirun

Check if there are other options already installed in the system.

[linch]

Ok, here it is:

mpirun --version
bash: mpirun: command not found.

$ echo $FOAM_MPI
openmpi-system

$ echo $PINC
-I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread

$ echo $PLIBS
-pthread -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl

$ echo $MPI_ARCH_PATH
/usr/lib/openmpi

[wyldckat]

You can also check:

update-alternatives --config mpirun

And see also:

ls -l /usr/bin/mpirun.*

[linch]

it doesn't look good:

update-alternatives --config mpirun
update-alternatives: error: no alternatives for mpirun.

ls -l /usr/bin/mpirun.*
ls: /usr/bin/mpirun.* not found

[wyldckat]

Then switch back to the normal OPENMPI option in OpenFOAM's etc/bashrc file and start a new terminal.

Then:

foam3rdParty
./Allwmake > make.log 2>&1

Let's see if Open-MPI 1.5.3 will play nice this time around

[linch]

Unfortunately not. There are still the same errors.

[wyldckat]

I think it is mpicc's fault here... but I don't know how to disable it without removing it

By the way, does this output anything:

echo $GREP_OPTIONS

In the mean time I'm going to test on the VM how to add mpicc to the mix to see if it bungles something up...

[linch]

Originally Posted by wyldckat

By the way, does this output anything:

echo $GREP_OPTIONS

nope, nothing there

[wyldckat]

Hi Ilya,

It's taking me a while in trying to reproduce the same error. And the "debian-security" server seems to be offline for a couple of minutes to help...

I did find something interesting: seems that you might have installed on your machine the package "mpi-default-dev" but not the "mpi-default-bin". Without the second one, it's not possible to use the system Open-MPI properly. To check if they are installed:

dpkg-query -s mpi-default-bin
dpkg-query -s mpi-default-dev

If both commands provide the line:

Status: install ok installed

then something strange is going on... if only one appears, then that's why you have half of it.

This command can help figure out if there are other surrounding packages:

dpkg-query -W "*mpi*"

Then use the "-s" for each package to confirm if they are installed. Or simply use Synaptic as root:

synaptic

I'll make another post when I have more information!

Best regards,
Bruno

[wyldckat]

Hi Ilya,

And I'm out of ideas. I haven't been able to reproduce the problem you're having. My guess would be that the other application to blame could be the Icc compiler that is installed in your machine.

Nonetheless, my suggestion is to:

1. Make sure to install those two packages:
   

   sudo apt-get install mpi-default-bin mpi-default-dev

   They will make sure Open-MPI is properly installed. You'll need to reboot the machine.
2. Switch to SYSTEMOPENMPI in OpenFOAM's "etc/bashrc" file.
3. Start a new terminal and try again building Pstream!

Best regards,
Bruno

[linch]

Hi Bruno,

first thanks a lot for the time you spent trying to figure out, what's goin on here.

None of the mpi-default-... packages is installed. Unfortunately though, I don't even have the sudo privileges, so I'll have to beg our admins at Monday to install them.

I wish you a nice weekend!
Ilya

[wyldckat]

Hi Ilya,

Maybe instead of begging for installing those packages, then perhaps it would be best to request more information about what's installed in the machine and also ask which packages could be affecting OpenFOAM's proper installation!

Over this long attempt to get things running, we've found that:

• The system's mpicc seems to be from one package.
• The system's mpirun is somewhat elusive, because sometimes it's present, others seems to be missing... although I might not be remembering this one properly.
• Intel's compiler is also in sight, which could either be good or bad.
  • I wonder if Intel MPI is installed in there somewhere as well!?

Knowing more about what's installed would help, because with the default packages, I don't have any problems whatsoever!

Nonetheless, in the end, you might have to switch to building with Icc instead of Gcc, which OpenFOAM supports, as long as the Icc version isn't too old...

edit: and have a nice weekend as well

Best regards,
Bruno