CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Installation

Grid Engine OpenFOAM15dev and OpenMPI124

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   February 17, 2009, 10:41
Default Hi, I have a cluster with O
  #1
Member
 
Tian
Join Date: Mar 2009
Posts: 97
Rep Power: 7
tian is on a distinguished road
Hi,

I have a cluster with OpenMPI-1.2.4 and Grid Engine. Did somebody know how I can tell OpenFOAM that it use the nativ OpenMPI and not the OpenMPI-1.2.6 in the ThirdParty? Thanks

Bye
Thomas
__________________
BIM HVACTool
tian is offline   Reply With Quote

Old   February 17, 2009, 10:59
Default I don't know what the sourcefo
  #2
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 774
Rep Power: 17
olesen will become famous soon enough
I don't know what the sourceforge "dev" version has, but in the normal distribution you should take a look at the etc/settings.sh file to find the OPENMPI settings. You could try adding your own definitions. Eg, you could define WM_MPLIB=SYSTEM_OPENMPI in the etc/bashrc and handle this case in etc/settings.sh

Keep in mind that you should also examine src/Pstream/Allwmake to make sure the Pstreams get re-made correctly. With any luck, you won't run into any linker errors.

However, there shouldn't be any particular reason that you can't just use the version in ThirdParty, it should coexist fine with any number of other versions of openmpit (provided that your LD_LIBRARY_PATH etc aren't cross-contaminated).
olesen is offline   Reply With Quote

Old   February 17, 2009, 11:54
Default I agree with Mark. We do it th
  #3
Senior Member
 
BastiL
Join Date: Mar 2009
Posts: 462
Rep Power: 10
bastil is on a distinguished road
I agree with Mark. We do it the same way. Building against other MPIs is hard. I tried it once but finally went back to "default" MPI.

Regards
bastil is offline   Reply With Quote

Old   February 17, 2009, 15:24
Default Hi, thank for the answer. F
  #4
Member
 
Tian
Join Date: Mar 2009
Posts: 97
Rep Power: 7
tian is on a distinguished road
Hi,

thank for the answer. For my understanding:

in the cluster there is OpenMPI-1.2.4:
./apps/openmpi/openmpi-1.2.4-gcc/

In the etc/basrh:
export WM_MLIB=OPENMPI

In the settings.sh:
OPENMPI)
# mpi_version=openmpi-1.2.6
mpi_version=openmpi-1.2.4-gcc

# export MPI_HOME=$WM_THIRD_PARTY_DIR/$mpi_version
export MPI_HOME=/apps/openmpi/$mpi_version

# export MPI_ARCH_PATH=$MPI_HOME/platforms/$WM_OPTIONS
export MPI_ARCH_PATH=/apps/openmpi/$mpi_version

# Tell OpenMPI where to find its install directory
export OPAL_PREFIX=$MPI_ARCH_PATH

_foamAddPath $MPI_ARCH_PATH/bin
_foamAddLib $MPI_ARCH_PATH/lib

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/$mpi_version
unset mpi_version
;;

Than I have to re-made the Pstream, right? Thanks for help

Bye
Thomas
__________________
BIM HVACTool
tian is offline   Reply With Quote

Old   February 18, 2009, 13:05
Default Hi, I tried without success
  #5
Member
 
Tian
Join Date: Mar 2009
Posts: 97
Rep Power: 7
tian is on a distinguished road
Hi,

I tried without success. I try to re-made the Pstream and this happen:

tian@poseidon051:~/OpenFOAM/OpenFOAM-1.5-dev/src/Pstream> ./Allwmake
+ wmake libso dummy
/home/tian/OpenFOAM/ThirdParty/gcc-4.3.1/platforms/linux64/bin/../lib/gcc/x86_64 -unknown-linux-gnu/4.3.1/../../../../lib64/libstdc++.so: file not recognized: File format not recognized
collect2: ld returned 1 exit status
make: *** [/home/tian/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/dummy/libPstream.so] Fehler 1
+ case "$WM_MPLIB" in
+ export WM_OPTIONS=linux64GccDPOptOPENMPI
+ WM_OPTIONS=linux64GccDPOptOPENMPI
+ set +x

Note: ignore spurious warnings about missing mpicxx.h headers
+ wmake libso mpi
/home/tian/OpenFOAM/ThirdParty/gcc-4.3.1/platforms/linux64/bin/../lib/gcc/x86_64 -unknown-linux-gnu/4.3.1/../../../../lib64/libstdc++.so: file not recognized: File format not recognized
collect2: ld returned 1 exit status
make: *** [/home/tian/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt//libPstream.so] Fehler 1
tian@poseidon051:~/OpenFOAM/OpenFOAM-1.5-dev/src/Pstream>

Some idea about it?

Thanks

Bye
Thomas
__________________
BIM HVACTool
tian is offline   Reply With Quote

Old   February 19, 2009, 01:30
Default Is there a problem running gri
  #6
Member
 
a a saha
Join Date: Mar 2009
Posts: 67
Rep Power: 7
asaha is on a distinguished road
Is there a problem running grid engine in cluster with OpenFOAM version of openmpi.

I think if the openmpi version in the cluster supports grid engine then you may be able to use the openmpi provided by OpenFOAM without going through the above.

In fact I had a problem in my cluster where the cluster openmpi did not have support for grid engine.

You can check the support for grid engine in the cluster version of openmpi as below:

ompi_info | grep gridengine

The result should be as below.

MCA ras: gridengine (MCA v1.0, API v1.3, Component v1.2.4)
MCA pls: gridengine (MCA v1.0, API v1.3, Component v1.2.4)
asaha is offline   Reply With Quote

Old   February 19, 2009, 01:48
Default Is there a problem running gri
  #7
Member
 
a a saha
Join Date: Mar 2009
Posts: 67
Rep Power: 7
asaha is on a distinguished road
Is there a problem running grid engine in cluster with OpenFOAM version of openmpi.

I think if the openmpi version in the cluster supports grid engine then you may be able to use the openmpi provided by OpenFOAM without going through the above.

In fact I had a problem in my cluster where the cluster openmpi did not have support for grid engine.

You can check the support for grid engine in the cluster version of openmpi as below:

ompi_info | grep gridengine

The result should be as below.

MCA ras: gridengine (MCA v1.0, API v1.3, Component v1.2.4)
MCA pls: gridengine (MCA v1.0, API v1.3, Component v1.2.4)
asaha is offline   Reply With Quote

Old   February 25, 2009, 07:58
Default Hi, I tried all things and
  #8
Member
 
Tian
Join Date: Mar 2009
Posts: 97
Rep Power: 7
tian is on a distinguished road
Hi,

I tried all things and it is not working. It seems that something is wrong with OpenMPI and Grid Engine together.

ompi_info gave me the same info:
MCA ras: gridengine (MCA v1.0, API v1.3, Component v1.2.4)
MCA pls: gridengine (MCA v1.0, API v1.3, Component v1.2.4)


My startscript.sh: "qsub startscript.sh"

#!/bin/bash
#$ -S /bin/bash
#$ -pe mpi 12
#$ -cwd

export PATH="/apps/openmpi/1.2.4-gcc/bin"
export LD_LIBRARY_PATH="/apps/openmpi/1.2.4-gcc/lib"

mpirun -mca pls_gridengine_verbose 1 -np $NSLOTS -machinefile ~/mpichhosts.$JOB_ID echo "hallo"

Everytime I got this error:
error: executing task of job 1464184 failed:
[poseidon058:07894] ERROR: A daemon on node poseidon024 failed to start as expected.
[poseidon058:07894] ERROR: There may be more information available from
[poseidon058:07894] ERROR: the 'qstat -t' command on the Grid Engine tasks.
[poseidon058:07894] ERROR: If the problem persists, please restart the
[poseidon058:07894] ERROR: Grid Engine PE job
[poseidon058:07894] ERROR: The daemon exited unexpectedly with status 1.

qsat -t gave not information about this error. Has somebody experience with it? Thanks!

Bye
Thomas
__________________
BIM HVACTool
tian is offline   Reply With Quote

Old   February 25, 2009, 09:08
Default Hi, I found it out: I ha
  #9
Member
 
Tian
Join Date: Mar 2009
Posts: 97
Rep Power: 7
tian is on a distinguished road
Hi,

I found it out:

I had to set: "control_slaves TRUE" in the PE Grid Engine

Thanks
Bye
Thomas
__________________
BIM HVACTool
tian is offline   Reply With Quote

Old   February 25, 2009, 09:59
Default Hi, now I have some questio
  #10
Member
 
Tian
Join Date: Mar 2009
Posts: 97
Rep Power: 7
tian is on a distinguished road
Hi,

now I have some question about DecomposeParcDict.

How can I set up this file automaticly from the grid engine? I mean, I changed:

numberOfSubdomains 4;
to
numberOfSubdomains slots;

If I use method metis than I have to write the number "1" depend of the slots that I got in this file everytime???

Can somebody show me a good solution , thanks!

Bye
Thomas
__________________
BIM HVACTool
tian is offline   Reply With Quote

Old   February 25, 2009, 10:21
Default In general, I would turn the p
  #11
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 774
Rep Power: 17
olesen will become famous soon enough
In general, I would turn the problem around the other way. Extract the numberOfSubdomains and use it to form your -pe request. This corresponds better at least to our usage pattern: eg, restarting an existing parallel calculation without decomposing yet again. If the domain is already decomposed, you can just pick up the number of processor* directories.


However, if you want it the otherway around, you'll need to add the corresponding logic to you GridEngine job script.
if [ "$NSLOTS" -gt 1 ]
then
# create/edit dictionary, do decompose
# run applicatioon with -parallel
else
# run applicatioon without -parallel
fi

Use your favourite tool (sed, perl, shell) to create/edit the dictionary.

You should note that decomposePar in OpenFOAM-1.5 also has -force and -ifRequired options that prove useful within this type of scripting.
olesen is offline   Reply With Quote

Old   February 26, 2009, 10:43
Default Hi Alex! I know I'm a bit a
  #12
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,747
Rep Power: 36
gschaider will become famous soon enoughgschaider will become famous soon enough
Hi Alex!

I know I'm a bit annoying with this PyFoam-stuff, but if good old Metis is alright for decomposing you might consider this:

http://openfoamwiki.net/index.php/Co...luster_support

But be warned: It works nicely for our place, but I don't know if anybody else uses that part of PyFoam

Bernhard
__________________
Note: I don't use "Friend"-feature on this forum out of principle. Ah. And by the way: I'm not on Facebook either. So don't be offended if I don't accept your invitation/friend request
gschaider is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Sun Grid Engine grtabor OpenFOAM Running, Solving & CFD 28 August 22, 2012 09:27
About OpenFOAM15dev waynezw0618 OpenFOAM Running, Solving & CFD 5 January 21, 2009 17:55
Running parallel job using qsub on sun grid engine nishant_hull OpenFOAM Running, Solving & CFD 5 February 7, 2008 14:52
CFX integration with Sun Grid Engine mausmi CFX 0 September 5, 2005 03:08
CFX and Sun Grid Engine David Hargreaves CFX 1 August 25, 2005 23:50


All times are GMT -4. The time now is 18:33.