OpenFOAM with Inifiband & parallel efficiency
 

Hi foamers,

I feel mad about my extremely slow parallel computing efficiency. I have modified channelFoam to perform DNS of pipe flow. The mesh is around 4 million. Each node has 8cores, but I only use 4 cores in each node. Because I found when I use 6cores or 8cores each node, the computing time does not less than 4 cores each node. What is more, when I use 6 nodes, that is 24 cores, the computing time almost the same as the 20 cores. I don't know why. The interconnect of my cluster is infiniband. I know from other thread that the parallel efficiency and scallability of IB are better than GigE and 10GigE, but the OpenFOAM shouled be recompiled. Unfortunately, I don't know wheather my recompiling procedure is right:
• Modify OpenFOAM-1.6/etc/bashrc to use MPI entry rather OPENMPI (WM_MPLIB:=MPI).
• Change MPI entry within settings.sh to system OpenMPI (export MPI_HOME=/usr/mpi/gcc/openmpi-1.4.2).
• Add the following to wmake/rules/linux64Gcc/mplib
PFLAGS= -DOMPI_SKIP_MPICXX
PINC = -I$(MPI_ARCH_PATH)/include
PLIBS = -L$(MPI_ARCH_PATH)/lib64 –lmpi
Then ./Allwmake in ThirdParty directory and OpenFOAM-1.7.1

After that, I performed parallel simulation, but error occured:
bash: orted: Command not found

Any one can help me?
Thank you very much in advance.

Hi,
you have changed from openMPI to MPI, but do you have MPI libraries installed correctly ? Since it cant find orted it is either not there or the path variable is not set properly.

gregor

Hi, gregor. Somenone told me another method to use infiniband. If it works, I will let you know

Try to reduce the times of file operating.

I am doing DNS with a code based on OF. For a mesh with 128*128*128 cells, the speedup of 4096 processors is about 1000. I am also testing with a finer mesh and wish to obtain a better scalability.

Hi, ywang. Can you post you code in the forum? Now, I don't know whether my mpi works with infiniband? One friend suggests me that recompile OpenFOAM, and use system's openmpi rather than OF's. But when I recompile, error occured. Later, I will post the error information in the forum

Sorry,,,,the code i am using has tens of files,,,

the solvers you used before are the re-compiled solvers from other place? Anyway, I suggest you use the local mpi and recompile the OF.

If you are using 128x128x128 mesh, it is very likely that you are using cartesian mesh. Why are you using openFOAM for it?? You can write yourself a 100 times faster solver than openFOAM for it since direct solution to poisson problem in uniform cartesian mesh is available.

Or I am mistaken and your mesh is not cartesian??

I am using unstructured mesh for complex geometry. The 128x128x128 mesh is non-uniform and used for the benchmark channel flow.

The solver I use is dnsPisoFoam, which is origin from channelFoam. Now I use the local openmpi, because using the system openmpi does not help to improve scalability. I tried the following three means to run parallel
1 mpirun -np 16 --hostfile machine --mca btl self,openib dnsPisoFoam -parallel
2 mpirun -np 16 --hostfile machine --mca btl self,tcp dnsPisoFoam -parallel
3 mpirun -np 16 --hostfile machine dnsPisoFoam -parallel
The first and the third means cost almost the same time, while the second mean costs more time

But still the scalability cannot be improved.

The mesh of my case is 130 million. Running 50000 timesteps, it costs almost a week. Too long time.

I am also working on DNS, but with my own solver using LBM. I am a fresh man in the OpenFOAM community. The only thing I can tell you now is to reduce the file operating and message communication as much as possible. For example, I found there will be a much improvement on the scalability if runTimeModifiable is changed from Yes to No.

how many cores you are using and what is your time per iteration??

One week for 130million cells and 50000 time steps should be acceptable, is it not??

It seems he is running 130 million cell with only 16 processors,,,

I really doubt it.

I tried running a 130 million case with 32 processors (with my software NOT openFOAM) and if I could manage 5000 steps in 1 week I would gladly take it. I really doubt openFOAM is faster than my solver.

I am sorry for my fault. My mesh is 1.3 million, and per time step costs 8s. I use 24 cores ,because after 24cores, the computing time cost doesnot decrease with the increase of cores

Thanks for your reply. I will have a try as what you say. And another question. When I finished the computation, how to get the average value? What is the time interval to sample the field?