[mer]

Hi,
How can I set the MPICH env_variables:
MPICH_PATH, MPICH_ROOT, etc.
I'm using a Pc with linux red Hat 9.0
I need this, because I have received in test installation that MPICH is not set.
THANKS

[mattijs]

Is this the foamInstallationTest script? This script is no longer valid for OpenFOAM 1.2 and the warning message is probably not relevant.

By default OpenFOAM uses lam and not mpich so none of the MPICH_ROOT etc. are set.

[mer]

Hi,
Then, what is the problem? When I compile the exemple turboFOAM with wmake and compare the results on my screen I can't find lines in compilation related to lam/MPI and mpich as expected by the user guide.
thanks

[mattijs]

There might be no problem, the text in the user guide might just describe a slightly different environment.

The question is does it run in parallel?

[mer]

Hi,
As you can see, I'm a new user of openFOAM and I try to execute this exemple in parallel with 2 Pcs on red hat 9.0 environment. When I got a result, I will be back.
thanks.

[anne]

Hello,

I would like to run OpenFoam with Mpich and not LAM.
By default I know that the LAM MPI is used by OpenFoam.
I would like to change the variable WM_MPLIB to
MPICH, however I couldn't find the
file where you can modify the default
value of WM_MPLIB. I looked in wmake
directory but so many files that
I didn't find out it.

Thanks

Anne

Look in the file
$WM_PROJECT_INST_DIR/OpenFOAM-$VERSION/.OpenFOAM/{cshrc , bashrc}
and set the WM_MPLIB.

Then go to
$WM_PROJECT_INST_DIR/OpenFOAM-$VERSION/.{cshrc, bashrc}
and set the path to your mpich installation.

theophane

[hjasak]

Look in:

~/OpenFOAM/OpenFOAM-1.3/.cshrc or .bashrc towards the end.

Hrv

[anne]

Sorry,

I am not very good in shell.
What I can see is that in the .cshrc there are a lot of options of MPI (lam, mpich ..)
alll with a il/else if
Please, could you be
a little more explicit on what I have to write
to force MPICH.

Do I have just to write in the .csrhc:

setenv WM_MPILIB MPICH

Thanks,

Anne

Ok, look in ~/OpenFOAM/OpenFOAM-1.3/.OpenFOAM/cshrc:
Go to section corresponding to your operating system
(linux, sun or silicon graphics workstation). Then comment the line corresponding to lam, and uncomment the line for mpich.

Then go to ~/OpenFOAM/OpenFOAM-1.3/.cshrc:
in the communications library section, set the variables corresponding to your installation of mpich.

theophane

[anne]

K,

Thank you,

I had just missed the correct path of the first directory you wrote me so that I couldn't find the cshrc file and was looking always in the .cshrc.


Thanks a lot,

I will do it now,

Anne

[hani]

I'm trying to link OpenFOAM 1.3 to a locally installed MPICH 1.2.7 and then run simpleFoam trough PBS (and actually also through Swegrid). I have compiled OpenFOAM locally with gcc 4.1.1 and I have produced a libPstream.so for MPICH using the procedure described below.

When I run simpleFoam it seems like MPICH starts the tasks (mpiexec: All 4 tasks started.), but then OpenFOAM aborts:
0] MPI Abort by user Aborting program !
[0] Aborting program!
p0_3256: p4_error: : 1
rm_l_3_30702: (0.004464) net_send: could not write to fd=9, errno = 32

Does anyone have a clue? I have tried to find the answer in the forum, without success.

Håkan.

--------------------------------------------
I have done the following to compile libPstream.so for MPICH:
In .OpenFOAM-1.3/cshrc:
setenv WM_MPLIB MPICH
In )openFOAM-1.3/.cshrc:
setenv MPICH_VERSION 1.2.7
setenv MPICH_PATH /usr/local/mpich
setenv MPICH_ARCH_PATH $MPICH_PATH
setenv MPICH_ROOT $MPICH_ARCH_PATH

AddLib $MPICH_ARCH_PATH/lib
AddPath $MPICH_ARCH_PATH/bin

setenv FOAM_MPI_LIBBIN $FOAM_LIBBIN/mpich-$MPICH_VERSION

Then in src/Pstream/mpi:
wclean
wmake libso

And I get a mpich-1.2.7/libPstream.so

There are no error messages when doing this except the warning: use of old-style cast

------------------------------------------
Result from ldd `which simpleFoam`:
libincompressibleTurbulenceModels.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libincompressibleTurb ulen
ceModels.so (0x40001000)
libincompressibleTransportModels.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libincompressibleTran sport
Models.so (0x401a7000)
libfiniteVolume.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libfiniteVolume.so (0x401d8000)
libmeshTools.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libmeshTools.so (0x40946000)
libOpenFOAM.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libOpenFOAM.so (0x40a96000)
libstdc++.so.6 => /users/unicc/hani/OpenFOAM/linux/gcc-4.1.1/lib/libstdc++.so.6 (0x40d64000)
libm.so.6 => /lib/i686/libm.so.6 (0x49c85000)
libgcc_s.so.1 => /users/unicc/hani/OpenFOAM/linux/gcc-4.1.1/lib/libgcc_s.so.1 (0x40e49000)
libpthread.so.0 => /lib/i686/libpthread.so.0 (0x49d16000)
libc.so.6 => /lib/i686/libc.so.6 (0x49b43000)
libPstream.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/mpich-1.2.7/libPstrea m.so (0x40e54000)
libtriSurface.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libtriSurface.so (0x40ec7000)
liblagrangian.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/liblagrangian.so (0x40f37000)
libfoamUser.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libfoamUser.so (0x40f3a000)
libz.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libz.so (0x40f3c000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00643000)

[hani]

It seems like my compilation with MPICH was alright all the time since I am now able to run it in parallel locally. The problem is when I submit my job.

When running using mpiexec through PBS there are lots of arguments added. Does anyone know how to make OpenFOAM not care about those:

/users/unicc/hani/OpenFOAM/OpenFOAM-1.3/applications/bin/linuxGcc4DPOpt/simpleFo am /users/unicc/hani/OpenFOAM/hani-1.3/run/pitzDaily/.. pitzDaily -parallel -p4wd /users/unicc/hani/OpenFOAM/hani-1.3/run/pitzDaily -execer_id mpiexec -master_host node101 -my_hostname node101 -my_nodenum 0 -my_numprocs 1 -total_numnodes 4 -master_port 32854 -remote_info node063 1 node061 1 node041 1

Error message:
[0] --> FOAM FATAL ERROR : Wrong number of arguments, expected 2 found 15

Håkan.

[mattijs]

Pstream/mpi/Pstream.C

Pstream::addValidParOptions

[anne]

Hello Hakan,


I have compiled Mpich_1.2.7. While I am
running an application: mpirun -np 2 `which application` ....


I have the following error as you get earlier (see below)
BUT it is because tehere is a problem with
a shared library that is not found.

I have just installed mpich-1.2.7 in Src of openFoam1-3, and then went to
Pstream/mpi and made wclean, wmake libso.

Did you also have an error with shared library ??
Also, I has the warning about " Pstream.C:279: warning: use of old-style cast" but I don't know how far it is important.
Did you succeed in runnning mpich ?


I am fighting a while with this. On which
station do you run it ??

Thanks for Your help,

Anne


--------------------------------------------
Exec : /afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/applications/bin/linuxGcc4DPOpt/po is_ufixed /afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/run case2D_par -p4pg /afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/run/PI22250 -p4wd /afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/run
/afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/applications/bin/linuxGcc4DPOpt/po is_ufixed: error while loading shared libraries: libfiniteVolume.so: cannot open shared object file: No such file or directory
p0_22339: p4_error: Child process exited while making connection to remote process on lince.ciemat.es: 0
p0_22339: (9.173624) net_send: could not write to fd=5, errno = 32

[hani]

Hello Anne,

I am able to run the locally installed MPICH. You have to make sure that all processes/processors have the correct PATHs and LD_LIBRARY_PATHs. You also have to be sure that they all have access to the shared library. When submitting jobs through a que system it is also important to think about where your submit script will be run. It is best to give full paths to the <root>. It is not for sure that the processes inherit all the environment variables, paths etc from the submit process.

I don't think we have to bother about old-style cast. It's just a warning. I think that it has been discussed before.

Right now I'm at vacation, so I didn't have time to test the suggestion by Mattijs (in fact I don't understand it yet).

As I said, I am able to run MPICH locally with mpirun, but when I submit through our que system I still have problems. Will have a look at it in August.

Håkan.

[anne]

Hello Hakan,

Thank you for your answer.

I will check again if the paths were correctly declared. How do you check that the paths is correctly declared for all processes.
When I check some of the variables it is OK, at least in the the home directory where I am working.
I dont't know the way to check this on the processes where I want to run my jobs.
The test I have done were locally, I didn't try through the queue system.

well, I know you are on holidays,
so enjoy your time.

Thanks again,

Anne

[hani]

Thanks Mattijs for your answer on July 14, it almost solved my problem.

I still have problems with the final flag:

-remote_info node063 1 node061 1 node041 1

This flag lists information about the other nodes involved in the parallel run. As you can see there are more than one argument to the flag, in fact twice the number of other nodes so it will depend on how many nodes are involved. How can this flag be added as valid?

I understand that for instance
validParOptions.insert("p4wd", "directory");
makes the flag -p4wd valid and I guess that the directory listed after -p4wd may be found in OpenFOAM somehow using the keyword "directory".
In my case I am not interested in finding out the arguments, I just want to make the computation run.

I have tried different ways of adding more arguments without success. Do you or anyone else have any idea?

Håkan.

[hani]

After some more research it seems like when running OpenFOAM using MPICH under PBS and using mpiexec, the extra flags added should actually be used by MPI_Init inside OpenFOAM.

Being unable to make the flag -remote_info and its arguments valid using validParOptions, I tried removing the FatalError.exit(); at line 193 in src/OpenFOAM/global/argList/argList.C (OF1.3)

The parallel run then works properly. However, I have then lost the nice check of arguments.

I would still like to have hints on how to make -remote_info and its arguments valid in a correct way. Note that -remote_info has a number of arguments that depends on the number of nodes involved.

Håkan.

[mattijs]

- maybe there is an option for mpiexec to read arguments from a file? There is one for the LAM mpiexec.

- the correct way would be to modify argList.C to do two passes over the arguments. First pass checks for validParOptions and calls parRunControl_.runPar which should filter the command line args. Then the second pass works as before and checks for correct OF arguments.