CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

OpenFOAM with Inifiband & parallel efficiency

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   June 11, 2011, 12:53
Question OpenFOAM with Inifiband & parallel efficiency
  #1
New Member
 
Join Date: Oct 2010
Posts: 24
Rep Power: 6
LijieNPIC is on a distinguished road
Hi foamers,

I feel mad about my extremely slow parallel computing efficiency. I have modified channelFoam to perform DNS of pipe flow. The mesh is around 4 million. Each node has 8cores, but I only use 4 cores in each node. Because I found when I use 6cores or 8cores each node, the computing time does not less than 4 cores each node. What is more, when I use 6 nodes, that is 24 cores, the computing time almost the same as the 20 cores. I don't know why. The interconnect of my cluster is infiniband. I know from other thread that the parallel efficiency and scallability of IB are better than GigE and 10GigE, but the OpenFOAM shouled be recompiled. Unfortunately, I don't know wheather my recompiling procedure is right:
• Modify OpenFOAM-1.6/etc/bashrc to use MPI entry rather OPENMPI (WM_MPLIB:=MPI).
• Change MPI entry within settings.sh to system OpenMPI (export MPI_HOME=/usr/mpi/gcc/openmpi-1.4.2).
• Add the following to wmake/rules/linux64Gcc/mplib
PFLAGS= -DOMPI_SKIP_MPICXX
PINC = -I$(MPI_ARCH_PATH)/include
PLIBS = -L$(MPI_ARCH_PATH)/lib64 –lmpi
Then ./Allwmake in ThirdParty directory and OpenFOAM-1.7.1

After that, I performed parallel simulation, but error occured:
bash: orted: Command not found

Any one can help me?
Thank you very much in advance.
LijieNPIC is offline   Reply With Quote

Old   June 14, 2011, 03:38
Default
  #2
Member
 
Gregor Olenik
Join Date: Jun 2009
Location: http://greole.github.io/
Posts: 78
Rep Power: 8
gregor is on a distinguished road
Hi,
you have changed from openMPI to MPI, but do you have MPI libraries installed correctly ? Since it cant find orted it is either not there or the path variable is not set properly.

gregor
gregor is offline   Reply With Quote

Old   June 15, 2011, 01:52
Default
  #3
New Member
 
Join Date: Oct 2010
Posts: 24
Rep Power: 6
LijieNPIC is on a distinguished road
Hi, gregor. Somenone told me another method to use infiniband. If it works, I will let you know
LijieNPIC is offline   Reply With Quote

Old   June 16, 2011, 02:27
Default
  #4
Member
 
Yong Wang
Join Date: Apr 2009
Posts: 32
Rep Power: 8
ywang is on a distinguished road
Quote:
Originally Posted by LijieNPIC View Post
Hi, gregor. Somenone told me another method to use infiniband. If it works, I will let you know
Try to reduce the times of file operating.

I am doing DNS with a code based on OF. For a mesh with 128*128*128 cells, the speedup of 4096 processors is about 1000. I am also testing with a finer mesh and wish to obtain a better scalability.
ywang is offline   Reply With Quote

Old   June 16, 2011, 04:59
Default
  #5
New Member
 
Join Date: Oct 2010
Posts: 24
Rep Power: 6
LijieNPIC is on a distinguished road
Quote:
Originally Posted by ywang View Post
Try to reduce the times of file operating.

I am doing DNS with a code based on OF. For a mesh with 128*128*128 cells, the speedup of 4096 processors is about 1000. I am also testing with a finer mesh and wish to obtain a better scalability.
Hi, ywang. Can you post you code in the forum? Now, I don't know whether my mpi works with infiniband? One friend suggests me that recompile OpenFOAM, and use system's openmpi rather than OF's. But when I recompile, error occured. Later, I will post the error information in the forum
LijieNPIC is offline   Reply With Quote

Old   June 16, 2011, 05:11
Default
  #6
Member
 
Yong Wang
Join Date: Apr 2009
Posts: 32
Rep Power: 8
ywang is on a distinguished road
Quote:
Originally Posted by LijieNPIC View Post
Hi, ywang. Can you post you code in the forum? Now, I don't know whether my mpi works with infiniband? One friend suggests me that recompile OpenFOAM, and use system's openmpi rather than OF's. But when I recompile, error occured. Later, I will post the error information in the forum
Sorry,,,,the code i am using has tens of files,,,

the solvers you used before are the re-compiled solvers from other place? Anyway, I suggest you use the local mpi and recompile the OF.
ywang is offline   Reply With Quote

Old   June 16, 2011, 18:27
Default
  #7
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 368
Rep Power: 10
arjun is on a distinguished road
Quote:
Originally Posted by ywang View Post
Try to reduce the times of file operating.

I am doing DNS with a code based on OF. For a mesh with 128*128*128 cells, the speedup of 4096 processors is about 1000. I am also testing with a finer mesh and wish to obtain a better scalability.

If you are using 128x128x128 mesh, it is very likely that you are using cartesian mesh. Why are you using openFOAM for it?? You can write yourself a 100 times faster solver than openFOAM for it since direct solution to poisson problem in uniform cartesian mesh is available.

Or I am mistaken and your mesh is not cartesian??
arjun is offline   Reply With Quote

Old   June 16, 2011, 18:44
Default
  #8
Member
 
Yong Wang
Join Date: Apr 2009
Posts: 32
Rep Power: 8
ywang is on a distinguished road
Quote:
Originally Posted by arjun View Post
If you are using 128x128x128 mesh, it is very likely that you are using cartesian mesh. Why are you using openFOAM for it?? You can write yourself a 100 times faster solver than openFOAM for it since direct solution to poisson problem in uniform cartesian mesh is available.

Or I am mistaken and your mesh is not cartesian??

I am using unstructured mesh for complex geometry. The 128x128x128 mesh is non-uniform and used for the benchmark channel flow.
ywang is offline   Reply With Quote

Old   June 23, 2011, 01:16
Default
  #9
New Member
 
Join Date: Oct 2010
Posts: 24
Rep Power: 6
LijieNPIC is on a distinguished road
Quote:
Originally Posted by ywang View Post
Sorry,,,,the code i am using has tens of files,,,

the solvers you used before are the re-compiled solvers from other place? Anyway, I suggest you use the local mpi and recompile the OF.
The solver I use is dnsPisoFoam, which is origin from channelFoam. Now I use the local openmpi, because using the system openmpi does not help to improve scalability. I tried the following three means to run parallel
1 mpirun -np 16 --hostfile machine --mca btl self,openib dnsPisoFoam -parallel
2 mpirun -np 16 --hostfile machine --mca btl self,tcp dnsPisoFoam -parallel
3 mpirun -np 16 --hostfile machine dnsPisoFoam -parallel
The first and the third means cost almost the same time, while the second mean costs more time

But still the scalability cannot be improved.
LijieNPIC is offline   Reply With Quote

Old   June 23, 2011, 01:19
Default
  #10
New Member
 
Join Date: Oct 2010
Posts: 24
Rep Power: 6
LijieNPIC is on a distinguished road
Quote:
Originally Posted by ywang View Post
Try to reduce the times of file operating.

I am doing DNS with a code based on OF. For a mesh with 128*128*128 cells, the speedup of 4096 processors is about 1000. I am also testing with a finer mesh and wish to obtain a better scalability.
The mesh of my case is 130 million. Running 50000 timesteps, it costs almost a week. Too long time.
LijieNPIC is offline   Reply With Quote

Old   June 23, 2011, 02:01
Default
  #11
Member
 
Yong Wang
Join Date: Apr 2009
Posts: 32
Rep Power: 8
ywang is on a distinguished road
Quote:
Originally Posted by LijieNPIC View Post
The solver I use is dnsPisoFoam, which is origin from channelFoam. Now I use the local openmpi, because using the system openmpi does not help to improve scalability. I tried the following three means to run parallel
1 mpirun -np 16 --hostfile machine --mca btl self,openib dnsPisoFoam -parallel
2 mpirun -np 16 --hostfile machine --mca btl self,tcp dnsPisoFoam -parallel
3 mpirun -np 16 --hostfile machine dnsPisoFoam -parallel
The first and the third means cost almost the same time, while the second mean costs more time

But still the scalability cannot be improved.
I am also working on DNS, but with my own solver using LBM. I am a fresh man in the OpenFOAM community. The only thing I can tell you now is to reduce the file operating and message communication as much as possible. For example, I found there will be a much improvement on the scalability if runTimeModifiable is changed from Yes to No.
ywang is offline   Reply With Quote

Old   June 23, 2011, 03:48
Default
  #12
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 368
Rep Power: 10
arjun is on a distinguished road
Quote:
Originally Posted by LijieNPIC View Post
The mesh of my case is 130 million. Running 50000 timesteps, it costs almost a week. Too long time.

how many cores you are using and what is your time per iteration??

One week for 130million cells and 50000 time steps should be acceptable, is it not??
arjun is offline   Reply With Quote

Old   June 23, 2011, 03:55
Default
  #13
Member
 
Yong Wang
Join Date: Apr 2009
Posts: 32
Rep Power: 8
ywang is on a distinguished road
Quote:
Originally Posted by arjun View Post
how many cores you are using and what is your time per iteration??

One week for 130million cells and 50000 time steps should be acceptable, is it not??
It seems he is running 130 million cell with only 16 processors,,,
ywang is offline   Reply With Quote

Old   June 23, 2011, 04:30
Default
  #14
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 368
Rep Power: 10
arjun is on a distinguished road
Quote:
Originally Posted by ywang View Post
It seems he is running 130 million cell with only 16 processors,,,

I really doubt it.

I tried running a 130 million case with 32 processors (with my software NOT openFOAM) and if I could manage 5000 steps in 1 week I would gladly take it. I really doubt openFOAM is faster than my solver.
arjun is offline   Reply With Quote

Old   June 23, 2011, 05:05
Default
  #15
New Member
 
Join Date: Oct 2010
Posts: 24
Rep Power: 6
LijieNPIC is on a distinguished road
Quote:
Originally Posted by arjun View Post
how many cores you are using and what is your time per iteration??

One week for 130million cells and 50000 time steps should be acceptable, is it not??
I am sorry for my fault. My mesh is 1.3 million, and per time step costs 8s. I use 24 cores ,because after 24cores, the computing time cost doesnot decrease with the increase of cores
LijieNPIC is offline   Reply With Quote

Old   June 23, 2011, 05:10
Default
  #16
New Member
 
Join Date: Oct 2010
Posts: 24
Rep Power: 6
LijieNPIC is on a distinguished road
Quote:
Originally Posted by ywang View Post
I am also working on DNS, but with my own solver using LBM. I am a fresh man in the OpenFOAM community. The only thing I can tell you now is to reduce the file operating and message communication as much as possible. For example, I found there will be a much improvement on the scalability if runTimeModifiable is changed from Yes to No.
Thanks for your reply. I will have a try as what you say. And another question. When I finished the computation, how to get the average value? What is the time interval to sample the field?
LijieNPIC is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Parallel run of OpenFOAM in linux and windows side by side m2montazari OpenFOAM Running, Solving & CFD 5 June 24, 2011 03:26
Large test case for running OpenFoam in parallel fhy OpenFOAM Running, Solving & CFD 22 September 22, 2009 11:13
Parallel solution in OpenFOAM makaveli_lcf OpenFOAM Running, Solving & CFD 0 September 21, 2009 08:07
OpenFoam parallel crashes at random prapanj OpenFOAM Running, Solving & CFD 3 April 22, 2009 07:49
OpenFOAM Training and Workshop Zagreb 2628Jan2006 hjasak OpenFOAM 1 February 2, 2006 22:07


All times are GMT -4. The time now is 20:35.