CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

running OpenFoam in parallel

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree4Likes

Reply
 
LinkBack Thread Tools Display Modes
Old   September 28, 2009, 23:37
Default running OpenFoam in parallel
  #1
Member
 
vishwanath somashekar
Join Date: Apr 2009
Posts: 41
Rep Power: 8
vishwa is on a distinguished road
Hi,
I am pretty to new to OpenFoam seeing from the number of posts that i have posted in the last few days.. I had a lot of problems running LES of my case. And finally I got it running today.

out high performane cluster has 16 processors per node and i was successful in running the case on 1 node with 16 processors. However I couldn't run it on 2 nodes with 16 processors per node.

here are the details of the decomposeParDict
Code:
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.6                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 32;

method          metis;

simpleCoeffs
{
    n               ( 2 1 1 );
    delta           0.001;
}

hierarchicalCoeffs
{
    n               ( 2 2 1 );
    delta           0.001;
    order           xyz;
}

metisCoeffs
{
}

manualCoeffs
{
    dataFile        "";
}

distributed    no;

roots  ();


// ************************************************************************* //
here is the script that i have written
Code:
#!/bin/tcsh

#   1) Save this script as a file named myscript on hpc4
#   2) On hpc4, Issue                   
#       qsub myscript    to submit the job 
#        Use qstat -a to see job status, 
#         Use qdel jobname to delete one of your jobs
#         jobnames are of the form 1234.hpc4 

###########################################
# Output goes to file BATCH_OUTPUT.
# Error output goes to file BATCH_ERRORS.
# If you want the output to go to another file, change BATCH_OUTPUT 
# or BATCH_ERRORS in the following lines to the full path of that file. 


#PBS  -o BATCH_OUTPUT
#PBS  -e BATCH_ERRORS

#PBS -lmem=64Gb,nodes=2:ppn=16,cput=768:00:00,walltime=24:00:00

cd $PBS_O_WORKDIR


mpirun --hostfile $PBS_NODEFILE -np 32 pisoFoam -case /working004b/mech_eng1/vishwa/OpenFOAM/vishwa-1.6/CIJR/ -parallel > log
here is the error that i get

Code:
[0] 
[0] 
[0] number of processor directories = 16 is not equal to the number of processors = 32
[0] 
FOAM parallel run exiting
[0] 
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 23160 on
node node161 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
-
-------------------------------------------------------------------------

when i ran decomposePar, it did produce 32 processor directories..I am not sure why i get this error..

Any help would be greatly appreciated..

Thanks a tonne in advance..

Vishwa
vishwa is offline   Reply With Quote

Old   September 29, 2009, 12:07
Default
  #2
Member
 
Eelco Gehring
Join Date: Mar 2009
Posts: 70
Rep Power: 8
feijooos is on a distinguished road
Vishwa,

You need to specify the metis coefficients in the decomposeParDict file, so there need to 32 coefficients. See below:

/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 1.6 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 32;

method metis;

simpleCoeffs
{
n ( 2 1 1 );
delta 0.001;
}

hierarchicalCoeffs
{
n ( 2 2 1 );
delta 0.001;
order xyz;
}

metisCoeffs
{
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
}

manualCoeffs
{
dataFile "";
}

distributed no;

roots ();
feijooos is offline   Reply With Quote

Old   November 24, 2009, 14:46
Default
  #3
Senior Member
 
xinguang cui
Join Date: Mar 2009
Posts: 116
Rep Power: 8
flying is on a distinguished road
vishwa and feijooos:

I think both of you have some experience to set up openfoam in cluster.

Please, really please give me some advice on it.

I am trying to set up openfoam in the cluster, but it is really sad that I do not know how to do it.

There is gcc4.4.1 and openmpi 1.3.3 in the cluster.


I should use all of the complier in the cluster. I am not allowed to use any complier in the thirdparty when the code is run in multi-nodes.


So, I need to compile the openfoam with the compile in cluster.


First question, Are gcc 4.4.1 and openmpi 1.3.3 enough for compile openfoam 1.5?
Second question, if it is enough , how do I do it?


I have changed the complier option and mpi setting up option. As follows


# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# WM_COMPILER_INST = OpenFOAM | System
#
WM_COMPILER_INST=OpenFOAM (Orginal)
WM_COMPILER_INST=System (changed)

case "$WM_MPLIB" in
OPENMPI)
mpi_version=openmpi-1.2.6
# export MPI_HOME=$WM_THIRD_PARTY_DIR/$mpi_version
#
export MPI_HOME=$WM_THIRD_PARTY_DIR/$mpi_version (orginal)
# export MPI_ARCH_PATH=$MPI_HOME/platforms/$WM_OPTIONS (orginal

export MPI_ARCH_PATH=/opt/openmpi/1.3.3/gcc-4.4.1 (changed)


It could compile. But there is some error information as follows:



Note: ignore spurious warnings about missing mpicxx.h headers
+ wmake libso mpi
Making dependency list for source file OPwrite.C
could not open file ompi/mpi/cxx/pmpicxx.h for source file OPwrite.C
could not open file ompi/mpi/cxx/constants.h for source file OPwrite.C
could not open file ompi/mpi/cxx/functions.h for source file OPwrite.C
could not open file ompi/mpi/cxx/datatype.h for source file OPwrite.C
could not open file ompi/mpi/cxx/exception.h for source file Opwrite.C



Who would like to give me some idea about it?
flying is offline   Reply With Quote

Old   November 24, 2009, 15:51
Default
  #4
Member
 
Eelco Gehring
Join Date: Mar 2009
Posts: 70
Rep Power: 8
feijooos is on a distinguished road
sorry, I can't help you. I don't know much about installing OF.
feijooos is offline   Reply With Quote

Old   February 4, 2010, 11:27
Default
  #5
Member
 
Matthias Kern
Join Date: Mar 2009
Location: Karlsruhe, Germany
Posts: 36
Rep Power: 8
make is on a distinguished road
dear flying,

it seems like you haven't set the path to the openmpi includes on your cluster correctly. The dependency builder is not able to find the header files. MPI_ARCH_PATH should point to the path where the include directory of openmpi is located. And don't forget to set WM_MPLIB in etc/bashrc within the OpenFOAM directory correctly.

regards,
Matthias
make is offline   Reply With Quote

Old   February 5, 2010, 10:21
Default
  #6
Senior Member
 
xinguang cui
Join Date: Mar 2009
Posts: 116
Rep Power: 8
flying is on a distinguished road
Thank you for your reply!

I have changed the path and set the WM_MPLIB in etc/bashrc. But it still doesn't work.

Best wishes!
flying is offline   Reply With Quote

Old   February 5, 2010, 11:05
Default
  #7
Member
 
Matthias Kern
Join Date: Mar 2009
Location: Karlsruhe, Germany
Posts: 36
Rep Power: 8
make is on a distinguished road
Dear flying,

this is what my settings.sh look like for OpenMPI:

OPENMPI)
mpi_version=1.3.3
export MPI_HOME=/software/all/openmpi/$mpi_version
export MPI_ARCH_PATH=$MPI_HOME

# Tell OpenMPI where to find its install directory
export OPAL_PREFIX=$MPI_ARCH_PATH

_foamAddPath $MPI_ARCH_PATH/bin
_foamAddLib $MPI_ARCH_PATH/lib

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/$mpi_version
unset mpi_version
;;


Within the /software/all/openmpi/1.3.3 directory is the include folder of openmpi /software/all/openmpi/1.3.3/include/openmpi. Your dependency builder is looking for the folder ompi in this directory.
E.g. one line in In your post looks like:
could not open file ompi/mpi/cxx/pmpicxx.h for source file OPwrite.C

So it seems like you still haven't set the path correctly. Check that these directories are within the path you specified for
MPI_ARCH_PATH.
According to your specification the file

/opt/openmpi/1.3.3/gcc-4.4.1/include/openmpi/ompi/mpi/cxx/pmpicxx.h

should exist. I sligthly wonder about the gcc-4.4.1 folder in your path. Maybe try
MPI_ARCH_PATH=/opt/openmpi/1.3.3.

Regards,
Matthias
make is offline   Reply With Quote

Old   April 7, 2010, 18:32
Default Pararllel Running
  #8
Member
 
Join Date: Nov 2009
Posts: 48
Rep Power: 7
farhagim is on a distinguished road
Hi there,

I am a beginner user of the OpenFoam...I really need to learn how to run openfoam with parallel processing. I read the all the available guideline in this site www.OpenFoam.com)...but I have basic problem with that. First of all I can not find "etc/hosts " in OpenFoam Directories where I should define the host names of the machines. Since I am not familiar with that kind of file
, I would be so grateful if anybody provide me one of these kind of files.

2. I read somewhere that I have to create a "file that contains the
host names of the machines. The file can be given any name and located at any path. " I could not understand what does it means ...or what kind of the files do i have to create....can anybody give a hint or example about that.

3. In decompose Dict, the roots entry is a list of root paths, <root0>, <root1>, . . . , for each node:
roots
<nRoots>
(
"<root0>"
"<root1>"
...
);

I need somebody help me throough this too..
AS you can see , I do not know anything about the configuration of running the case in parallel. I would be so grateful if you can help me

Thanks alot
Farhangi
Quote:
Originally Posted by make View Post
Dear flying,

this is what my settings.sh look like for OpenMPI:

OPENMPI)
mpi_version=1.3.3
export MPI_HOME=/software/all/openmpi/$mpi_version
export MPI_ARCH_PATH=$MPI_HOME

# Tell OpenMPI where to find its install directory
export OPAL_PREFIX=$MPI_ARCH_PATH

_foamAddPath $MPI_ARCH_PATH/bin
_foamAddLib $MPI_ARCH_PATH/lib

export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/$mpi_version
unset mpi_version
;;


Within the /software/all/openmpi/1.3.3 directory is the include folder of openmpi /software/all/openmpi/1.3.3/include/openmpi. Your dependency builder is looking for the folder ompi in this directory.
E.g. one line in In your post looks like:
could not open file ompi/mpi/cxx/pmpicxx.h for source file OPwrite.C

So it seems like you still haven't set the path correctly. Check that these directories are within the path you specified for
MPI_ARCH_PATH.
According to your specification the file

/opt/openmpi/1.3.3/gcc-4.4.1/include/openmpi/ompi/mpi/cxx/pmpicxx.h

should exist. I sligthly wonder about the gcc-4.4.1 folder in your path. Maybe try
MPI_ARCH_PATH=/opt/openmpi/1.3.3.

Regards,
Matthias
farhagim is offline   Reply With Quote

Old   February 17, 2011, 07:17
Default same problem
  #9
Senior Member
 
maysmech's Avatar
 
Join Date: Jan 2010
Posts: 347
Blog Entries: 2
Rep Power: 8
maysmech is on a distinguished road
Quote:
Originally Posted by farhagim View Post
Hi there,

I am a beginner user of the OpenFoam...I really need to learn how to run openfoam with parallel processing. I read the all the available guideline in this site www.OpenFoam.com)...but I have basic problem with that. First of all I can not find "etc/hosts " in OpenFoam Directories where I should define the host names of the machines. Since I am not familiar with that kind of file
, I would be so grateful if anybody provide me one of these kind of files.

2. I read somewhere that I have to create a "file that contains the
host names of the machines. The file can be given any name and located at any path. " I could not understand what does it means ...or what kind of the files do i have to create....can anybody give a hint or example about that.

3. In decompose Dict, the roots entry is a list of root paths, <root0>, <root1>, . . . , for each node:
roots
<nRoots>
(
"<root0>"
"<root1>"
...
);

I need somebody help me throough this too..
AS you can see , I do not know anything about the configuration of running the case in parallel. I would be so grateful if you can help me

Thanks alot
Farhangi
I have same problems.
Any help will be appreciated.
maysmech is offline   Reply With Quote

Old   February 17, 2011, 13:22
Default openfoam in cluster
  #10
New Member
 
Join Date: Mar 2009
Posts: 18
Rep Power: 8
fsaltara is on a distinguished road
I had a similar problem, and solved it simply running the Allwmake script in the thirdparty folder.
fsaltara is offline   Reply With Quote

Old   February 17, 2011, 14:02
Default
  #11
Senior Member
 
maysmech's Avatar
 
Join Date: Jan 2010
Posts: 347
Blog Entries: 2
Rep Power: 8
maysmech is on a distinguished road
I have used two machines (laptop and desktop) which are connected via cross cable.
Quote:
maysam-desktop cpu = 6
maysam-laptop cpu = 2

I added to /etc/hosts of both machines:
192.168.1.1 maysam-desktop
192.168.1.2 maysam-laptop
Also set ip same as above for both of them.

After trying to run, this error is seen:

Quote:
maysam@maysam-desktop:~/OpenFOAM/maysam-1.7.0/run/icotest$ mpirun --hostfile host.txt -np 8 icoFoam -parallel
maysam@maysam-laptop's password:
--------------------------------------------------------------------------
mpirun was unable to launch the specified application as it could not find an executable:

Executable: icoFoam
Node: maysam-laptop

while attempting to start process rank 6.
--------------------------------------------------------------------------
maysam@maysam-desktop:~/OpenFOAM/maysam-1.7.0/run/icotest$
Any help will be appreciated.
maysmech is offline   Reply With Quote

Old   March 29, 2011, 13:17
Default
  #12
Member
 
longamon's Avatar
 
David Aljure
Join Date: Mar 2011
Location: CTTC Universidad Politécnica de Catalunya. Spain
Posts: 38
Rep Power: 6
longamon is on a distinguished road
I don't know if this comes too late, but heres a sample of my decomposeParDict:


numberOfSubdomains 6; //number of cpus you'll use

method scotch; //method of decomposition

simpleCoeffs
{
n ( 2 2 2 );
delta 0.001;
}

hierarchicalCoeffs
{
n ( 1 1 1 );
delta 0.001;
order xyz;
}

// in my case I use scotch decomposition, so I specify the processsor weight for each cpu

scotchCoeffs
{
processorWeights
(
1
1
1
1
1
1
);
}

manualCoeffs
{
dataFile "";
}

distributed yes; //distribute the data on diferent harddrives

roots 5
(
"/home/davidal"
"/home/davidal"
"/home/davidal"
"/home/davidal"
"/home/davidal"
);

since there are 6 cpus, there is one master and 5 slaves. the master will be processor 0, so there's no need to specify a root to it, the other 5 cpus are slave. note that one machine can have more than one cpu. in my case, the case file is located under /home/davidal for all my cpus.

see
Cluster OpenFOAM [Solved]
for more info

In my case i will have 6 folders of info in each cpu, all named /processor0, /processor1, /processor2, /processor3, /processor4 and /processor5. Since I have 2 cpu's per machines i will be saving data on only 2 of these folders on each machine.
machine1 will save data on processor0 and processor1
machine2 will save data on processor2 and processor3
machine3 will save data on processor4 and processor5

hope this helps, I'm quite new to OF and struggled with this for a bit, but now its working fine
longamon is offline   Reply With Quote

Old   March 29, 2011, 15:41
Default
  #13
Senior Member
 
maysmech's Avatar
 
Join Date: Jan 2010
Posts: 347
Blog Entries: 2
Rep Power: 8
maysmech is on a distinguished road
Quote:
Originally Posted by longamon View Post
I don't know if this comes too late, but heres a sample of my decomposeParDict:


numberOfSubdomains 6; //number of cpus you'll use

method scotch; //method of decomposition

simpleCoeffs
{
n ( 2 2 2 );
delta 0.001;
}

hierarchicalCoeffs
{
n ( 1 1 1 );
delta 0.001;
order xyz;
}

// in my case I use scotch decomposition, so I specify the processsor weight for each cpu

scotchCoeffs
{
processorWeights
(
1
1
1
1
1
1
);
}

manualCoeffs
{
dataFile "";
}

distributed yes; //distribute the data on diferent harddrives

roots 5
(
"/home/davidal"
"/home/davidal"
"/home/davidal"
"/home/davidal"
"/home/davidal"
);

since there are 6 cpus, there is one master and 5 slaves. the master will be processor 0, so there's no need to specify a root to it, the other 5 cpus are slave. note that one machine can have more than one cpu. in my case, the case file is located under /home/davidal for all my cpus.

see
Cluster OpenFOAM [Solved]
for more info

In my case i will have 6 folders of info in each cpu, all named /processor0, /processor1, /processor2, /processor3, /processor4 and /processor5. Since I have 2 cpu's per machines i will be saving data on only 2 of these folders on each machine.
machine1 will save data on processor0 and processor1
machine2 will save data on processor2 and processor3
machine3 will save data on processor4 and processor5

hope this helps, I'm quite new to OF and struggled with this for a bit, but now its working fine
Dear David,
Thanks for your post.
My problem has been solved. you can see its solution at:
_Parallel Run with two PCs problems_

Regards,
Maysam
maysmech is offline   Reply With Quote

Old   July 21, 2012, 00:19
Default
  #14
New Member
 
pengfei han
Join Date: Jul 2011
Location: China
Posts: 6
Blog Entries: 3
Rep Power: 6
pfhan is on a distinguished road
Thanks, David, it is of great help to me.
pfhan is offline   Reply With Quote

Old   August 3, 2012, 09:49
Default
  #15
Senior Member
 
Samuele Z
Join Date: Oct 2009
Location: Mozzate - Co - Italy
Posts: 490
Rep Power: 9
samiam1000 is on a distinguished road
Hi All,

I would run a parallel case on a server.

I read that I should give this command:

mpirun -np 4 -hostname FILE buoyantPimpleFoam -parallel

where FILE is a text file where I write the name of the machine I am running my case.

Does this file need any header or it is enough a single line with the name of the machine?

Thanks,
Samuele
samiam1000 is offline   Reply With Quote

Old   August 3, 2012, 10:13
Default
  #16
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,488
Blog Entries: 34
Rep Power: 86
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Samuele,

I'm going to quote (minor changes to make it more legible):
Quote:
Originally Posted by Lexx View Post
2- In file machines (which can be created in the tutorial) put the machine name followed by how many processors are used on this machine.

Example:
Code:
ubuntu1
 ubuntu2 cpu = 2
In this case, the FILE is named "machines".

Best regards,
Bruno

Last edited by wyldckat; August 3, 2012 at 10:14. Reason: "the text FILE" didn't look good, so I removed "text"...
wyldckat is offline   Reply With Quote

Old   August 3, 2012, 10:15
Default
  #17
Senior Member
 
Samuele Z
Join Date: Oct 2009
Location: Mozzate - Co - Italy
Posts: 490
Rep Power: 9
samiam1000 is on a distinguished road
Thanks a lot Bruno.

I solved it.

I opened another thread (maybe you can check it or delete it and continue here) to understand how to put my process in background, when running on the server.

Thanks a lot,
Samuele
samiam1000 is offline   Reply With Quote

Old   August 3, 2012, 10:19
Default
  #18
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,488
Blog Entries: 34
Rep Power: 86
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Samuele,

OK... I guess the thread you speak of is this: http://www.cfd-online.com/Forums/ope...el-server.html

I'll answer it there...

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   August 3, 2012, 10:21
Default
  #19
Senior Member
 
Samuele Z
Join Date: Oct 2009
Location: Mozzate - Co - Italy
Posts: 490
Rep Power: 9
samiam1000 is on a distinguished road
Perfect.

Thanks a lot!
samiam1000 is offline   Reply With Quote

Old   November 7, 2012, 12:44
Default Need help with parallel run
  #20
New Member
 
Arun S
Join Date: Feb 2012
Location: Chennai, India
Posts: 23
Rep Power: 5
arunsmec is on a distinguished road
Send a message via Skype™ to arunsmec
I am trying to do an LES of roundjet using parallel computing. I used the decomposePar utility to decompose the domain. My workstation is equipped with 2 quadcore processors with multithreading (2 threads/core). Does decomposition into more than 8 subdomains make any difference?

Last edited by arunsmec; November 7, 2012 at 13:14.
arunsmec is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Superlinear speedup in OpenFOAM 13 msrinath80 OpenFOAM Running, Solving & CFD 18 March 3, 2015 06:36
Large test case for running OpenFoam in parallel fhy OpenFOAM Running, Solving & CFD 22 September 22, 2009 11:13
Statically Compiling OpenFOAM Issues herzfeldd OpenFOAM Installation 21 January 6, 2009 10:38
Kubuntu uses dash breaks All scripts in tutorials platopus OpenFOAM Bugs 8 April 15, 2008 07:52
Performance of interFoam running in parallel hsieh OpenFOAM Running, Solving & CFD 8 September 14, 2006 09:15


All times are GMT -4. The time now is 07:32.