[arjun]

Originally Posted by flavio_galeazzo

I would like to share my newest experience with the scalability of the linear solvers in OpenFoam. I used to run simulations using compressible solvers, and used the PCG linear solver for pressure and PBiCG for the other variables. The scalability was very good up to 256 processors, and I could get one second computational time per time step for a 14 million node grid.
Recently I moved to a incompressible solver, and in that case the GAMG linear solver was far superior than the PCG for the pressure, the other variables stayed with the PBiCG. However, to my surprise, the scalability was very poor this time. I got good results up to 32 processors, with about 10 seconds computational time per time step, and increasing the number of processors do not improved the computational time.
Reading now the ppt from Dr. Jasak, from the post of lakeat, it become clear that the problem is actually the GAMG linear solver. It is very unfortunate, since the GAMG linear solver is indeed very helpfull for the incompressible solver.

Here is the reason why compressible case converges faster than incompressible case.
In case of incompressible navier stokes, the pressure correction is elliptic type.

http://mathworld.wolfram.com/Ellipti...lEquation.html

In numerical methods terms diagonal element = sum of off diagonal elements. (absolute values). This makes this linear system very difficult to converge. This is why a very efficient method for this is critical to efficiency of CFD solver.

In case of compressible solver the linear system has diagonal dominance. It could be easily converged now.

About your scaling issue as to why compressible case scales better. It is very hard to pin point the reason. But my guess is that, (remember I said my guess), since there are less iterations involved the result of poor scaling is not that pronounced as in case of incompressible solver. Less iterations required == less time lost in inefficiency of linear solver.

In case of compressible solver, I might completely give up AMG because BiCGStab etc have better scaling than AMG and system is now easy to converge due to diagonal dominance.

[flavio_galeazzo]

Thanks for the explanation, arjun. It makes more sense to me now.

I think I was too vague in my last post. Sorry lakeat. When I said that it worked well, I wanted to say that when solving an incompressible problem using a compressible solver, the results were very close, however the scalability of the solver was much better.

[kumar]

Hello Foamers,
Our university upgraded our cluster and I have access now to a cluster with infiniband. My home directory was synchronized with the new cluster. SO I can run my jobs. But I have no idea if I am really making use of the infiniband.

I would prefer not to compile OF again, just for making use of infiniband.

1)I need help on "linking Pstream against the system compiled OpenMPI"

How do I do that?
And @Daniel:
1 PFLAGS = -DOMPI_SKIP_MPICXX
2 PINC = -I$(MPI_ARCH_PATH)/include
3 PLIBS = -L$(MPI_ARCH_PATH)/lib -lmpi

2)Could you please let me know where can I find these settings.

3)Also how to make sure If I am using system OpenMpi or openfoam's openMPI while I am submitting my job?

I use the following script to submit the job:
mpirun -hostfile $OAR_NODEFILE -mca btl ^openib -mca plm_rsh_agent "oarsh" \
-np `wc -l $OAR_NODEFILE | cut -d " " -f 1` interFoam -parallel > log

Any help or explanations is really useful for me.
Thanks in Advance
regards
K.Suresh kumar

[lakeat]

Hi Kumar,

Here are my experience,
1. You dont need to take special care about how to link openmpi with infiniband, it is automatically handled by openmpi during openmpi's compilation.
2. By using `which mpirun`, you will see which mpi package you are using.
3. Other settings are in wmake/ folder, reading the etc/bashrc is also helpful. Just use the find command if you dont know where they are.

Anyway, just test on a simple case, and plot the speedup. mpi with infiniband is much faster and sppedup curve is much better.

[lakeat]

<Deleted><Deleted>

[lakeat]

Another vital tip:

When using afs, dont output too frequently if you dont need it. (Including the logs file). Too many io operations will slow down the entire process.

[kumar]

Hello Daniel,
Thanks for the prompt reply. I will setup a simple case and plot the efficiency and then let you know.

regards
K.Suresh kumar

[1/153]

Originally Posted by arjun

Here is the reason why compressible case converges faster than incompressible case.
In case of incompressible navier stokes, the pressure correction is elliptic type.

http://mathworld.wolfram.com/Ellipti...lEquation.html

In numerical methods terms diagonal element = sum of off diagonal elements. (absolute values). This makes this linear system very difficult to converge. This is why a very efficient method for this is critical to efficiency of CFD solver.

In case of compressible solver the linear system has diagonal dominance. It could be easily converged now.

About your scaling issue as to why compressible case scales better. It is very hard to pin point the reason. But my guess is that, (remember I said my guess), since there are less iterations involved the result of poor scaling is not that pronounced as in case of incompressible solver. Less iterations required == less time lost in inefficiency of linear solver.

In case of compressible solver, I might completely give up AMG because BiCGStab etc have better scaling than AMG and system is now easy to converge due to diagonal dominance.

When you are talking about convergence. I am wondering are you referring to steady RANS simulations only? Is your observation applicable to unsteady simulations? Thanks

[1/153]

I am also struggling with mpi implementations.

With a inifniband cluster, will mvapich be substantially better than openmpi?

Does anyone has any real experience?

Thanks a million

[arjun]

Originally Posted by 1/153

When you are talking about convergence. I am wondering are you referring to steady RANS simulations only? Is your observation applicable to unsteady simulations? Thanks

yes, the situation do not change even if calculation is unsteady. I was taking about solution of pressure correction equation at any iteration of SIMPLE.