CFD Online Discussion Forums

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)
-   OpenFOAM Installation (https://www.cfd-online.com/Forums/openfoam-installation/)
-   -   Configuration of MPICH (https://www.cfd-online.com/Forums/openfoam-installation/57536-configuration-mpich.html)

mer September 7, 2005 13:52

Hi, How can I set the MPICH e
 
Hi,
How can I set the MPICH env_variables:
MPICH_PATH, MPICH_ROOT, etc.
I'm using a Pc with linux red Hat 9.0
I need this, because I have received in test installation that MPICH is not set.
THANKS

mattijs September 8, 2005 00:06

Is this the foamInstallationTe
 
Is this the foamInstallationTest script? This script is no longer valid for OpenFOAM 1.2 and the warning message is probably not relevant.

By default OpenFOAM uses lam and not mpich so none of the MPICH_ROOT etc. are set.

mer September 8, 2005 13:50

Hi, Then, what is the problem
 
Hi,
Then, what is the problem? When I compile the exemple turboFOAM with wmake and compare the results on my screen I can't find lines in compilation related to lam/MPI and mpich as expected by the user guide.
thanks

mattijs September 8, 2005 14:25

There might be no problem, the
 
There might be no problem, the text in the user guide might just describe a slightly different environment.

The question is does it run in parallel?

mer September 9, 2005 06:09

Hi, As you can see, I'm a new
 
Hi,
As you can see, I'm a new user of openFOAM and I try to execute this exemple in parallel with 2 Pcs on red hat 9.0 environment. When I got a result, I will be back.
thanks.

anne May 18, 2006 08:07

Hello, I would like to run
 
Hello,

I would like to run OpenFoam with Mpich and not LAM.
By default I know that the LAM MPI is used by OpenFoam.
I would like to change the variable WM_MPLIB to
MPICH, however I couldn't find the
file where you can modify the default
value of WM_MPLIB. I looked in wmake
directory but so many files that
I didn't find out it.

Thanks

Anne

theof May 18, 2006 08:37

Look in the file $WM_PROJECT_
 
Look in the file
$WM_PROJECT_INST_DIR/OpenFOAM-$VERSION/.OpenFOAM/{cshrc , bashrc}
and set the WM_MPLIB.

Then go to
$WM_PROJECT_INST_DIR/OpenFOAM-$VERSION/.{cshrc, bashrc}
and set the path to your mpich installation.

theophane

hjasak May 18, 2006 08:41

Look in: ~/OpenFOAM/OpenFOA
 
Look in:

~/OpenFOAM/OpenFOAM-1.3/.cshrc or .bashrc towards the end.

Hrv

anne May 18, 2006 09:58

Sorry, I am not very good i
 
Sorry,

I am not very good in shell.
What I can see is that in the .cshrc there are a lot of options of MPI (lam, mpich ..)
alll with a il/else if
Please, could you be
a little more explicit on what I have to write
to force MPICH.

Do I have just to write in the .csrhc:

setenv WM_MPILIB MPICH

Thanks,

Anne

theof May 19, 2006 03:49

Ok, look in ~/OpenFOAM/OpenFOA
 
Ok, look in ~/OpenFOAM/OpenFOAM-1.3/.OpenFOAM/cshrc:
Go to section corresponding to your operating system
(linux, sun or silicon graphics workstation). Then comment the line corresponding to lam, and uncomment the line for mpich.

Then go to ~/OpenFOAM/OpenFOAM-1.3/.cshrc:
in the communications library section, set the variables corresponding to your installation of mpich.

theophane

anne May 19, 2006 04:29

K, Thank you, I had just
 
K,

Thank you,

I had just missed the correct path of the first directory you wrote me so that I couldn't find the cshrc file and was looking always in the .cshrc.


Thanks a lot,

I will do it now,

Anne

hani July 14, 2006 04:48

I'm trying to link OpenFOAM 1.
 
I'm trying to link OpenFOAM 1.3 to a locally installed MPICH 1.2.7 and then run simpleFoam trough PBS (and actually also through Swegrid). I have compiled OpenFOAM locally with gcc 4.1.1 and I have produced a libPstream.so for MPICH using the procedure described below.

When I run simpleFoam it seems like MPICH starts the tasks (mpiexec: All 4 tasks started.), but then OpenFOAM aborts:
0] MPI Abort by user Aborting program !
[0] Aborting program!
p0_3256: p4_error: : 1
rm_l_3_30702: (0.004464) net_send: could not write to fd=9, errno = 32

Does anyone have a clue? I have tried to find the answer in the forum, without success.

Håkan.

--------------------------------------------
I have done the following to compile libPstream.so for MPICH:
In .OpenFOAM-1.3/cshrc:
setenv WM_MPLIB MPICH
In )openFOAM-1.3/.cshrc:
setenv MPICH_VERSION 1.2.7
setenv MPICH_PATH /usr/local/mpich
setenv MPICH_ARCH_PATH $MPICH_PATH
setenv MPICH_ROOT $MPICH_ARCH_PATH

AddLib $MPICH_ARCH_PATH/lib
AddPath $MPICH_ARCH_PATH/bin

setenv FOAM_MPI_LIBBIN $FOAM_LIBBIN/mpich-$MPICH_VERSION

Then in src/Pstream/mpi:
wclean
wmake libso

And I get a mpich-1.2.7/libPstream.so

There are no error messages when doing this except the warning: use of old-style cast

------------------------------------------
Result from ldd `which simpleFoam`:
libincompressibleTurbulenceModels.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libincompressibleTurb ulen
ceModels.so (0x40001000)
libincompressibleTransportModels.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libincompressibleTran sport
Models.so (0x401a7000)
libfiniteVolume.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libfiniteVolume.so (0x401d8000)
libmeshTools.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libmeshTools.so (0x40946000)
libOpenFOAM.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libOpenFOAM.so (0x40a96000)
libstdc++.so.6 => /users/unicc/hani/OpenFOAM/linux/gcc-4.1.1/lib/libstdc++.so.6 (0x40d64000)
libm.so.6 => /lib/i686/libm.so.6 (0x49c85000)
libgcc_s.so.1 => /users/unicc/hani/OpenFOAM/linux/gcc-4.1.1/lib/libgcc_s.so.1 (0x40e49000)
libpthread.so.0 => /lib/i686/libpthread.so.0 (0x49d16000)
libc.so.6 => /lib/i686/libc.so.6 (0x49b43000)
libPstream.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/mpich-1.2.7/libPstrea m.so (0x40e54000)
libtriSurface.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libtriSurface.so (0x40ec7000)
liblagrangian.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/liblagrangian.so (0x40f37000)
libfoamUser.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libfoamUser.so (0x40f3a000)
libz.so => /users/unicc/hani/OpenFOAM/OpenFOAM-1.3/lib/linuxGcc4DPOpt/libz.so (0x40f3c000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00643000)

hani July 14, 2006 09:36

It seems like my compilation w
 
It seems like my compilation with MPICH was alright all the time since I am now able to run it in parallel locally. The problem is when I submit my job.

When running using mpiexec through PBS there are lots of arguments added. Does anyone know how to make OpenFOAM not care about those:

/users/unicc/hani/OpenFOAM/OpenFOAM-1.3/applications/bin/linuxGcc4DPOpt/simpleFo am /users/unicc/hani/OpenFOAM/hani-1.3/run/pitzDaily/.. pitzDaily -parallel -p4wd /users/unicc/hani/OpenFOAM/hani-1.3/run/pitzDaily -execer_id mpiexec -master_host node101 -my_hostname node101 -my_nodenum 0 -my_numprocs 1 -total_numnodes 4 -master_port 32854 -remote_info node063 1 node061 1 node041 1

Error message:
[0] --> FOAM FATAL ERROR : Wrong number of arguments, expected 2 found 15

Håkan.

mattijs July 14, 2006 13:41

Pstream/mpi/Pstream.C Pstre
 
Pstream/mpi/Pstream.C

Pstream::addValidParOptions

anne July 17, 2006 11:05

Hello Hakan, I have compi
 
Hello Hakan,


I have compiled Mpich_1.2.7. While I am
running an application: mpirun -np 2 `which application` ....


I have the following error as you get earlier (see below)
BUT it is because tehere is a problem with
a shared library that is not found.

I have just installed mpich-1.2.7 in Src of openFoam1-3, and then went to
Pstream/mpi and made wclean, wmake libso.

Did you also have an error with shared library ??
Also, I has the warning about " Pstream.C:279: warning: use of old-style cast" but I don't know how far it is important.
Did you succeed in runnning mpich ?


I am fighting a while with this. On which
station do you run it ??

Thanks for Your help,

Anne


--------------------------------------------
Exec : /afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/applications/bin/linuxGcc4DPOpt/po is_ufixed /afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/run case2D_par -p4pg /afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/run/PI22250 -p4wd /afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/run
/afs/ciemat.es/users/u5303/OpenFOAM/u5303-1.3/applications/bin/linuxGcc4DPOpt/po is_ufixed: error while loading shared libraries: libfiniteVolume.so: cannot open shared object file: No such file or directory
p0_22339: p4_error: Child process exited while making connection to remote process on lince.ciemat.es: 0
p0_22339: (9.173624) net_send: could not write to fd=5, errno = 32

hani July 18, 2006 04:44

Hello Anne, I am able to ru
 
Hello Anne,

I am able to run the locally installed MPICH. You have to make sure that all processes/processors have the correct PATHs and LD_LIBRARY_PATHs. You also have to be sure that they all have access to the shared library. When submitting jobs through a que system it is also important to think about where your submit script will be run. It is best to give full paths to the <root>. It is not for sure that the processes inherit all the environment variables, paths etc from the submit process.

I don't think we have to bother about old-style cast. It's just a warning. I think that it has been discussed before.

Right now I'm at vacation, so I didn't have time to test the suggestion by Mattijs (in fact I don't understand it yet).

As I said, I am able to run MPICH locally with mpirun, but when I submit through our que system I still have problems. Will have a look at it in August.

Håkan.

anne July 18, 2006 05:00

Hello Hakan, Thank you for
 
Hello Hakan,

Thank you for your answer.

I will check again if the paths were correctly declared. How do you check that the paths is correctly declared for all processes.
When I check some of the variables it is OK, at least in the the home directory where I am working.
I dont't know the way to check this on the processes where I want to run my jobs.
The test I have done were locally, I didn't try through the queue system.

well, I know you are on holidays,
so enjoy your time.

Thanks again,

Anne

hani August 22, 2006 07:25

Thanks Mattijs for your answer
 
Thanks Mattijs for your answer on July 14, it almost solved my problem.

I still have problems with the final flag:

-remote_info node063 1 node061 1 node041 1

This flag lists information about the other nodes involved in the parallel run. As you can see there are more than one argument to the flag, in fact twice the number of other nodes so it will depend on how many nodes are involved. How can this flag be added as valid?

I understand that for instance
validParOptions.insert("p4wd", "directory");
makes the flag -p4wd valid and I guess that the directory listed after -p4wd may be found in OpenFOAM somehow using the keyword "directory".
In my case I am not interested in finding out the arguments, I just want to make the computation run.

I have tried different ways of adding more arguments without success. Do you or anyone else have any idea?

Håkan.

hani August 23, 2006 11:04

After some more research it se
 
After some more research it seems like when running OpenFOAM using MPICH under PBS and using mpiexec, the extra flags added should actually be used by MPI_Init inside OpenFOAM.

Being unable to make the flag -remote_info and its arguments valid using validParOptions, I tried removing the FatalError.exit(); at line 193 in src/OpenFOAM/global/argList/argList.C (OF1.3)

The parallel run then works properly. However, I have then lost the nice check of arguments.

I would still like to have hints on how to make -remote_info and its arguments valid in a correct way. Note that -remote_info has a number of arguments that depends on the number of nodes involved.

Håkan.

mattijs August 24, 2006 04:54

- maybe there is an option for
 
- maybe there is an option for mpiexec to read arguments from a file? There is one for the LAM mpiexec.

- the correct way would be to modify argList.C to do two passes over the arguments. First pass checks for validParOptions and calls parRunControl_.runPar which should filter the command line args. Then the second pass works as before and checks for correct OF arguments.

mattijs August 24, 2006 13:39

Hi Håkan, can you try attac
 
Hi Håkan,

can you try attached argList.C and let me know whether that helps?

http://www.cfd-online.com/OpenFOAM_D...hment_icon.gif argList.C

hani August 24, 2006 14:43

Hi Mattijs, mpiexec can rea
 
Hi Mattijs,

mpiexec can read the arguments from a file, and I tried that. However, the arguments in the file are just added as flags to the mpiexec command and we're back where we started.

I'll test your argList.C on monday when I get back to work.

If it doesn't work, is there any problem just removing the FatalError.exit() as long as I make sure that I start my computations correctly (giving the correct flags and arguments)?

Thanks for your efforts.
Håkan.

mattijs August 25, 2006 04:34

>removing the FatalError. It
 
>removing the FatalError.
It probably will set the root and case incorrectly.

hani August 28, 2006 04:24

Hi Mattijs, Actually, remov
 
Hi Mattijs,

Actually, removing FatalError seemed to work and set the root and case correctly.

Your argList.C above also seems to work, so I will stick to that one. What I am not really sure about is if you actually truncate argv? Then the extra flags are not needed by MPI_Init inside OpenFOAM? Or is MPI_Init run before argList.C?

It now seems like I don't have to do addValidParOptions for the extra flags, so there is still no check of consistency for those flags. But I can live with that.

Thanks for your help!
Håkan.

mattijs August 29, 2006 04:52

Thanks for testing. The 'ru
 
Thanks for testing.

The 'rule' is that MPI_Init strips off any of its arguments (which it added to start up the job). So after MPI_Init we should have a clean argument list.

You should now not need those extra addValidParOptions in your mpi/Pstream.C

hsieh September 9, 2006 08:58

Hi, What is the difference
 
Hi,

What is the difference between MPICH and Lam? Does MPICH has a better performance than LAM?
Thanks!

Pei

rafal December 5, 2006 14:37

Hi I got today the same messa
 
Hi
I got today the same message
error. Without PBS everything
runs perfectly but when using
queue system than it is not so
pinky. Did you find any solution?
Is thre any trick to solve or
bypass the error?
thanks in advance
rafal

error:
[0] MPI Abort by user Aborting program !
[0] Aborting program

hani December 6, 2006 02:18

The problem with additional fl
 
The problem with additional flags to mpiexec was solved when using Mattijs argList.C file in the previous entry:
http://www.cfd-online.com/cgi-bin/Op...=9626#POST9626

Håkan

kumar2 June 2, 2007 04:08

Dear Friends I am using OF1
 
Dear Friends

I am using OF1.3. want to change from LAM to MVAPICH. i did the following

1) export WM_MPLIB=MPICH in bashrc
2) set all path to the MVAPICH library in .bashrc

But i get this error when i do wmake libso in /src/Pstream/


/usr/bin/ld: /usr/local/topspin/mpi/mvapich/gnu/lib/libmpich.a(probe.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC
/usr/local/topspin/mpi/mvapich/gnu/lib/libmpich.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [/home/kri_kuma/OpenFOAM/OpenFOAM-1.3/lib/linuxAMD64Gcc4DPOpt/mpich/libPstream.s o] Error 1

///////////////////////////////////

There is only a libmpich.a in /mvapich/lib

what should i do ?

Thanks a lot in advance

kumar

kumar2 June 6, 2007 18:26

Dear all, Any comments on
 
Dear all,

Any comments on the previous post which is once again repeated here.

//////////////////////////////////////////////
Dear Friends

I am using OF1.3. want to change from LAM to MVAPICH. i did the following

1) export WM_MPLIB=MPICH in bashrc
2) set all path to the MVAPICH library in .bashrc

But i get this error when i do wmake libso in /src/Pstream/


/usr/bin/ld: /usr/local/topspin/mpi/mvapich/gnu/lib/libmpich.a(probe.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC
/usr/local/topspin/mpi/mvapich/gnu/lib/libmpich.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [/home/kri_kuma/OpenFOAM/OpenFOAM-1.3/lib/linuxAMD64Gcc4DPOpt/mpich/libPstream.s o] Error 1

///////////////////////////////////

There is only a libmpich.a in /mvapich/lib

what should i do ?

Thanks a lot in advance

kumar

msrinath80 June 6, 2007 18:46

These are just guesses: 1.
 
These are just guesses:

1. Try to link against a shared library (file extension *.so) as opposed to a static on (*.a). In other words build the MPI library with --with-shared or whatever is the appropriate argument. If you're linking against a pre-built library, try to see if you can find whether it was build with 'shared library' suppprt.
2. Make sure that WM_64 is set to on (assuming you are in 64 bit mode).

kumar2 June 6, 2007 19:08

Hi pUI Thanks a lot for the
 
Hi pUI

Thanks a lot for the prompt reply. I only vaguly understand what you say and hopefully my sysad will help me from here.

kumar


All times are GMT -4. The time now is 04:47.