CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM (http://www.cfd-online.com/Forums/openfoam/)
-   -   OpenFOAM with Inifiband & parallel efficiency (http://www.cfd-online.com/Forums/openfoam/89401-openfoam-inifiband-parallel-efficiency.html)

LijieNPIC June 11, 2011 12:53

OpenFOAM with Inifiband & parallel efficiency
 
Hi foamers,

I feel mad about my extremely slow parallel computing efficiency. I have modified channelFoam to perform DNS of pipe flow. The mesh is around 4 million. Each node has 8cores, but I only use 4 cores in each node. Because I found when I use 6cores or 8cores each node, the computing time does not less than 4 cores each node. What is more, when I use 6 nodes, that is 24 cores, the computing time almost the same as the 20 cores. I don't know why. The interconnect of my cluster is infiniband. I know from other thread that the parallel efficiency and scallability of IB are better than GigE and 10GigE, but the OpenFOAM shouled be recompiled. Unfortunately, I don't know wheather my recompiling procedure is right:
• Modify OpenFOAM-1.6/etc/bashrc to use MPI entry rather OPENMPI (WM_MPLIB:=MPI).
• Change MPI entry within settings.sh to system OpenMPI (export MPI_HOME=/usr/mpi/gcc/openmpi-1.4.2).
• Add the following to wmake/rules/linux64Gcc/mplib
PFLAGS= -DOMPI_SKIP_MPICXX
PINC = -I$(MPI_ARCH_PATH)/include
PLIBS = -L$(MPI_ARCH_PATH)/lib64 –lmpi
Then ./Allwmake in ThirdParty directory and OpenFOAM-1.7.1

After that, I performed parallel simulation, but error occured:
bash: orted: Command not found

Any one can help me?
Thank you very much in advance.

gregor June 14, 2011 03:38

Hi,
you have changed from openMPI to MPI, but do you have MPI libraries installed correctly ? Since it cant find orted it is either not there or the path variable is not set properly.

gregor

LijieNPIC June 15, 2011 01:52

Hi, gregor. Somenone told me another method to use infiniband. If it works, I will let you know

ywang June 16, 2011 02:27

Quote:

Originally Posted by LijieNPIC (Post 312050)
Hi, gregor. Somenone told me another method to use infiniband. If it works, I will let you know

Try to reduce the times of file operating.

I am doing DNS with a code based on OF. For a mesh with 128*128*128 cells, the speedup of 4096 processors is about 1000. I am also testing with a finer mesh and wish to obtain a better scalability.

LijieNPIC June 16, 2011 04:59

Quote:

Originally Posted by ywang (Post 312227)
Try to reduce the times of file operating.

I am doing DNS with a code based on OF. For a mesh with 128*128*128 cells, the speedup of 4096 processors is about 1000. I am also testing with a finer mesh and wish to obtain a better scalability.

Hi, ywang. Can you post you code in the forum? Now, I don't know whether my mpi works with infiniband? One friend suggests me that recompile OpenFOAM, and use system's openmpi rather than OF's. But when I recompile, error occured. Later, I will post the error information in the forum

ywang June 16, 2011 05:11

Quote:

Originally Posted by LijieNPIC (Post 312240)
Hi, ywang. Can you post you code in the forum? Now, I don't know whether my mpi works with infiniband? One friend suggests me that recompile OpenFOAM, and use system's openmpi rather than OF's. But when I recompile, error occured. Later, I will post the error information in the forum

Sorry,,,,the code i am using has tens of files,,,

the solvers you used before are the re-compiled solvers from other place? Anyway, I suggest you use the local mpi and recompile the OF.

arjun June 16, 2011 18:27

Quote:

Originally Posted by ywang (Post 312227)
Try to reduce the times of file operating.

I am doing DNS with a code based on OF. For a mesh with 128*128*128 cells, the speedup of 4096 processors is about 1000. I am also testing with a finer mesh and wish to obtain a better scalability.


If you are using 128x128x128 mesh, it is very likely that you are using cartesian mesh. Why are you using openFOAM for it?? You can write yourself a 100 times faster solver than openFOAM for it since direct solution to poisson problem in uniform cartesian mesh is available.

Or I am mistaken and your mesh is not cartesian??

ywang June 16, 2011 18:44

Quote:

Originally Posted by arjun (Post 312351)
If you are using 128x128x128 mesh, it is very likely that you are using cartesian mesh. Why are you using openFOAM for it?? You can write yourself a 100 times faster solver than openFOAM for it since direct solution to poisson problem in uniform cartesian mesh is available.

Or I am mistaken and your mesh is not cartesian??


I am using unstructured mesh for complex geometry. The 128x128x128 mesh is non-uniform and used for the benchmark channel flow.

LijieNPIC June 23, 2011 01:16

Quote:

Originally Posted by ywang (Post 312242)
Sorry,,,,the code i am using has tens of files,,,

the solvers you used before are the re-compiled solvers from other place? Anyway, I suggest you use the local mpi and recompile the OF.

The solver I use is dnsPisoFoam, which is origin from channelFoam. Now I use the local openmpi, because using the system openmpi does not help to improve scalability. I tried the following three means to run parallel
1 mpirun -np 16 --hostfile machine --mca btl self,openib dnsPisoFoam -parallel
2 mpirun -np 16 --hostfile machine --mca btl self,tcp dnsPisoFoam -parallel
3 mpirun -np 16 --hostfile machine dnsPisoFoam -parallel
The first and the third means cost almost the same time, while the second mean costs more time

But still the scalability cannot be improved.

LijieNPIC June 23, 2011 01:19

Quote:

Originally Posted by ywang (Post 312227)
Try to reduce the times of file operating.

I am doing DNS with a code based on OF. For a mesh with 128*128*128 cells, the speedup of 4096 processors is about 1000. I am also testing with a finer mesh and wish to obtain a better scalability.

The mesh of my case is 130 million. Running 50000 timesteps, it costs almost a week. Too long time.

ywang June 23, 2011 02:01

Quote:

Originally Posted by LijieNPIC (Post 313191)
The solver I use is dnsPisoFoam, which is origin from channelFoam. Now I use the local openmpi, because using the system openmpi does not help to improve scalability. I tried the following three means to run parallel
1 mpirun -np 16 --hostfile machine --mca btl self,openib dnsPisoFoam -parallel
2 mpirun -np 16 --hostfile machine --mca btl self,tcp dnsPisoFoam -parallel
3 mpirun -np 16 --hostfile machine dnsPisoFoam -parallel
The first and the third means cost almost the same time, while the second mean costs more time

But still the scalability cannot be improved.

I am also working on DNS, but with my own solver using LBM. I am a fresh man in the OpenFOAM community. The only thing I can tell you now is to reduce the file operating and message communication as much as possible. For example, I found there will be a much improvement on the scalability if runTimeModifiable is changed from Yes to No.

arjun June 23, 2011 03:48

Quote:

Originally Posted by LijieNPIC (Post 313192)
The mesh of my case is 130 million. Running 50000 timesteps, it costs almost a week. Too long time.


how many cores you are using and what is your time per iteration??

One week for 130million cells and 50000 time steps should be acceptable, is it not??

ywang June 23, 2011 03:55

Quote:

Originally Posted by arjun (Post 313207)
how many cores you are using and what is your time per iteration??

One week for 130million cells and 50000 time steps should be acceptable, is it not??

It seems he is running 130 million cell with only 16 processors,,,

arjun June 23, 2011 04:30

Quote:

Originally Posted by ywang (Post 313210)
It seems he is running 130 million cell with only 16 processors,,,


I really doubt it.

I tried running a 130 million case with 32 processors (with my software NOT openFOAM) and if I could manage 5000 steps in 1 week I would gladly take it. I really doubt openFOAM is faster than my solver.

LijieNPIC June 23, 2011 05:05

Quote:

Originally Posted by arjun (Post 313207)
how many cores you are using and what is your time per iteration??

One week for 130million cells and 50000 time steps should be acceptable, is it not??

I am sorry for my fault. My mesh is 1.3 million, and per time step costs 8s. I use 24 cores ,because after 24cores, the computing time cost doesnot decrease with the increase of cores

LijieNPIC June 23, 2011 05:10

Quote:

Originally Posted by ywang (Post 313195)
I am also working on DNS, but with my own solver using LBM. I am a fresh man in the OpenFOAM community. The only thing I can tell you now is to reduce the file operating and message communication as much as possible. For example, I found there will be a much improvement on the scalability if runTimeModifiable is changed from Yes to No.

Thanks for your reply. I will have a try as what you say. And another question. When I finished the computation, how to get the average value? What is the time interval to sample the field?


All times are GMT -4. The time now is 22:50.