CFD Online Discussion Forums

CFD Online Discussion Forums (
-   OpenFOAM Installation (
-   -   Configuration of MPICH (

mer September 7, 2005 12:52

Hi, How can I set the MPICH e
How can I set the MPICH env_variables:
I'm using a Pc with linux red Hat 9.0
I need this, because I have received in test installation that MPICH is not set.

mattijs September 7, 2005 23:06

Is this the foamInstallationTe
Is this the foamInstallationTest script? This script is no longer valid for OpenFOAM 1.2 and the warning message is probably not relevant.

By default OpenFOAM uses lam and not mpich so none of the MPICH_ROOT etc. are set.

mer September 8, 2005 12:50

Hi, Then, what is the problem
Then, what is the problem? When I compile the exemple turboFOAM with wmake and compare the results on my screen I can't find lines in compilation related to lam/MPI and mpich as expected by the user guide.

mattijs September 8, 2005 13:25

There might be no problem, the
There might be no problem, the text in the user guide might just describe a slightly different environment.

The question is does it run in parallel?

mer September 9, 2005 05:09

Hi, As you can see, I'm a new
As you can see, I'm a new user of openFOAM and I try to execute this exemple in parallel with 2 Pcs on red hat 9.0 environment. When I got a result, I will be back.

anne May 18, 2006 07:07

Hello, I would like to run

I would like to run OpenFoam with Mpich and not LAM.
By default I know that the LAM MPI is used by OpenFoam.
I would like to change the variable WM_MPLIB to
MPICH, however I couldn't find the
file where you can modify the default
value of WM_MPLIB. I looked in wmake
directory but so many files that
I didn't find out it.



theof May 18, 2006 07:37

Look in the file $WM_PROJECT_
Look in the file
and set the WM_MPLIB.

Then go to
and set the path to your mpich installation.


hjasak May 18, 2006 07:41

Look in: ~/OpenFOAM/OpenFOA
Look in:

~/OpenFOAM/OpenFOAM-1.3/.cshrc or .bashrc towards the end.


anne May 18, 2006 08:58

Sorry, I am not very good i

I am not very good in shell.
What I can see is that in the .cshrc there are a lot of options of MPI (lam, mpich ..)
alll with a il/else if
Please, could you be
a little more explicit on what I have to write
to force MPICH.

Do I have just to write in the .csrhc:




theof May 19, 2006 02:49

Ok, look in ~/OpenFOAM/OpenFOA
Ok, look in ~/OpenFOAM/OpenFOAM-1.3/.OpenFOAM/cshrc:
Go to section corresponding to your operating system
(linux, sun or silicon graphics workstation). Then comment the line corresponding to lam, and uncomment the line for mpich.

Then go to ~/OpenFOAM/OpenFOAM-1.3/.cshrc:
in the communications library section, set the variables corresponding to your installation of mpich.


anne May 19, 2006 03:29

K, Thank you, I had just

Thank you,

I had just missed the correct path of the first directory you wrote me so that I couldn't find the cshrc file and was looking always in the .cshrc.

Thanks a lot,

I will do it now,


hani July 14, 2006 03:48

I'm trying to link OpenFOAM 1.
I'm trying to link OpenFOAM 1.3 to a locally installed MPICH 1.2.7 and then run simpleFoam trough PBS (and actually also through Swegrid). I have compiled OpenFOAM locally with gcc 4.1.1 and I have produced a for MPICH using the procedure described below.

When I run simpleFoam it seems like MPICH starts the tasks (mpiexec: All 4 tasks started.), but then OpenFOAM aborts:
0] MPI Abort by user Aborting program !
[0] Aborting program!
p0_3256: p4_error: : 1
rm_l_3_30702: (0.004464) net_send: could not write to fd=9, errno = 32

Does anyone have a clue? I have tried to find the answer in the forum, without success.


I have done the following to compile for MPICH:
In .OpenFOAM-1.3/cshrc:
In )openFOAM-1.3/.cshrc:
setenv MPICH_VERSION 1.2.7
setenv MPICH_PATH /usr/local/mpich



Then in src/Pstream/mpi:
wmake libso

And I get a mpich-1.2.7/

There are no error messages when doing this except the warning: use of old-style cast

Result from ldd `which simpleFoam`: => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libincompressibleTurb ulen (0x40001000) => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libincompressibleTran sport (0x401a7000) => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/ (0x401d8000) => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/ (0x40946000) => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/ (0x40a96000) => /users/unicc/hani/OpenFOAM/linux/gcc-4.1.1/lib/ (0x40d64000) => /lib/i686/ (0x49c85000) => /users/unicc/hani/OpenFOAM/linux/gcc-4.1.1/lib/ (0x40e49000) => /lib/i686/ (0x49d16000) => /lib/i686/ (0x49b43000) => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/mpich-1.2.7/libPstrea (0x40e54000) => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/ (0x40ec7000) => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/ (0x40f37000) => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/ (0x40f3a000) => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/ (0x40f3c000)
/lib/ => /lib/ (0x00643000)

hani July 14, 2006 08:36

It seems like my compilation w
It seems like my compilation with MPICH was alright all the time since I am now able to run it in parallel locally. The problem is when I submit my job.

When running using mpiexec through PBS there are lots of arguments added. Does anyone know how to make OpenFOAM not care about those:

/users/unicc/hani/OpenFOAM/OpenFOAM-1.3/applications/bin/linuxGcc4DPOpt/simpleFo am /users/unicc/hani/OpenFOAM/hani-1.3/run/pitzDaily/.. pitzDaily -parallel -p4wd /users/unicc/hani/OpenFOAM/hani-1.3/run/pitzDaily -execer_id mpiexec -master_host node101 -my_hostname node101 -my_nodenum 0 -my_numprocs 1 -total_numnodes 4 -master_port 32854 -remote_info node063 1 node061 1 node041 1

Error message:
[0] --> FOAM FATAL ERROR : Wrong number of arguments, expected 2 found 15


mattijs July 14, 2006 12:41

Pstream/mpi/Pstream.C Pstre


anne July 17, 2006 10:05

Hello Hakan, I have compi
Hello Hakan,

I have compiled Mpich_1.2.7. While I am
running an application: mpirun -np 2 `which application` ....

I have the following error as you get earlier (see below)
BUT it is because tehere is a problem with
a shared library that is not found.

I have just installed mpich-1.2.7 in Src of openFoam1-3, and then went to
Pstream/mpi and made wclean, wmake libso.

Did you also have an error with shared library ??
Also, I has the warning about " Pstream.C:279: warning: use of old-style cast" but I don't know how far it is important.
Did you succeed in runnning mpich ?

I am fighting a while with this. On which
station do you run it ??

Thanks for Your help,


Exec : /afs/ is_ufixed /afs/ case2D_par -p4pg /afs/ -p4wd /afs/
/afs/ is_ufixed: error while loading shared libraries: cannot open shared object file: No such file or directory
p0_22339: p4_error: Child process exited while making connection to remote process on 0
p0_22339: (9.173624) net_send: could not write to fd=5, errno = 32

hani July 18, 2006 03:44

Hello Anne, I am able to ru
Hello Anne,

I am able to run the locally installed MPICH. You have to make sure that all processes/processors have the correct PATHs and LD_LIBRARY_PATHs. You also have to be sure that they all have access to the shared library. When submitting jobs through a que system it is also important to think about where your submit script will be run. It is best to give full paths to the <root>. It is not for sure that the processes inherit all the environment variables, paths etc from the submit process.

I don't think we have to bother about old-style cast. It's just a warning. I think that it has been discussed before.

Right now I'm at vacation, so I didn't have time to test the suggestion by Mattijs (in fact I don't understand it yet).

As I said, I am able to run MPICH locally with mpirun, but when I submit through our que system I still have problems. Will have a look at it in August.


anne July 18, 2006 04:00

Hello Hakan, Thank you for
Hello Hakan,

Thank you for your answer.

I will check again if the paths were correctly declared. How do you check that the paths is correctly declared for all processes.
When I check some of the variables it is OK, at least in the the home directory where I am working.
I dont't know the way to check this on the processes where I want to run my jobs.
The test I have done were locally, I didn't try through the queue system.

well, I know you are on holidays,
so enjoy your time.

Thanks again,


hani August 22, 2006 06:25

Thanks Mattijs for your answer
Thanks Mattijs for your answer on July 14, it almost solved my problem.

I still have problems with the final flag:

-remote_info node063 1 node061 1 node041 1

This flag lists information about the other nodes involved in the parallel run. As you can see there are more than one argument to the flag, in fact twice the number of other nodes so it will depend on how many nodes are involved. How can this flag be added as valid?

I understand that for instance
validParOptions.insert("p4wd", "directory");
makes the flag -p4wd valid and I guess that the directory listed after -p4wd may be found in OpenFOAM somehow using the keyword "directory".
In my case I am not interested in finding out the arguments, I just want to make the computation run.

I have tried different ways of adding more arguments without success. Do you or anyone else have any idea?


hani August 23, 2006 10:04

After some more research it se
After some more research it seems like when running OpenFOAM using MPICH under PBS and using mpiexec, the extra flags added should actually be used by MPI_Init inside OpenFOAM.

Being unable to make the flag -remote_info and its arguments valid using validParOptions, I tried removing the FatalError.exit(); at line 193 in src/OpenFOAM/global/argList/argList.C (OF1.3)

The parallel run then works properly. However, I have then lost the nice check of arguments.

I would still like to have hints on how to make -remote_info and its arguments valid in a correct way. Note that -remote_info has a number of arguments that depends on the number of nodes involved.


mattijs August 24, 2006 03:54

- maybe there is an option for
- maybe there is an option for mpiexec to read arguments from a file? There is one for the LAM mpiexec.

- the correct way would be to modify argList.C to do two passes over the arguments. First pass checks for validParOptions and calls parRunControl_.runPar which should filter the command line args. Then the second pass works as before and checks for correct OF arguments.

All times are GMT -4. The time now is 11:08.