CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Pre-Processing (http://www.cfd-online.com/Forums/openfoam-pre-processing/)
-   -   interFoam parallel (http://www.cfd-online.com/Forums/openfoam-pre-processing/124224-interfoam-parallel.html)

idefix September 30, 2013 06:25

interFoam parallel
 
Hello together,

I am using interFoam and itīs getting really slow when I use it in parallel.
Iīve got a 8,1 Mio cell-grid and I used 100, 2000 and 4000 processors.
With 100 processors I need 10 hours for one interval to be written out. The intervall is 10^(-4) seconds. When I use 2000 or 4000 processors I need about 1 min for one iteration.

Has anyone some experiences with interFoam in parallel?

Thanks a lot

nimasam September 30, 2013 07:57

it may relate to your matrix solver, would you please post your fvSolution here

Bernhard September 30, 2013 08:01

4000 processors seems a bit excessive for 8M cells. I would assume you did not gain anything by increasing from 2000 to 4000 processors.

How are you solving the pressure equation?

idefix September 30, 2013 08:45

Hello,

thanks for your ideas

Here is my fvSolution-file:

solvers
{
pcorr
{
solver PCG;
preconditioner DIC;
tolerance 1e-10;
relTol 0;
}

p_rgh
{
solver PCG;
preconditioner DIC;
tolerance 1e-07;
relTol 0.05;
}

p_rghFinal
{
solver PCG;
preconditioner DIC;
tolerance 1e-07;
relTol 0;
}

"(U|k|epsilon)"
{
solver PBiCG;
preconditioner DILU;
tolerance 1e-06;
relTol 0;
}

"(U|k|epsilon)Final"
{
solver PBiCG;
preconditioner DILU;
tolerance 1e-08;
relTol 0;
}
}

PIMPLE
{
momentumPredictor no;
nCorrectors 3;
nNonOrthogonalCorrectors 0;
nAlphaCorr 1;
nAlphaSubCycles 4;
cAlpha
}

Thanks a lot

michielm October 4, 2013 04:01

I think using GAMG for the pressure can help speed up the solution process quite a bit.

And I agree with Bernhard that the amount of processors you use is really excessive. For an 8M grid using 2000 processors results in only 4000 cells per processors, which might sound nice but all of these processors have to talk to their neighbours so chances are (big) that the communication between processors becomes the limiting factor if you use that many processors and it might even make you lose speed.

You can easily test the speed up more systematically by running relatively short simulation runs (e.g. only a few hundred timesteps) with different amounts of processors. And see how much speed you gain from adding more processors. A typical way of doing this is doubling the amount of processors every time and look at the speed up. So start e.g. at 50 then 100, 200, 400, etc

Bernhard October 4, 2013 04:43

Be careful, GAMG does not necessary outperform PCG on large parallel case, see: https://www.hpc.ntnu.no/display/hpc/...lywithpisoFoam

santiagomarquezd October 11, 2013 13:32

Hi, it was suggested in this forum to use ~50Kcells/processor, which gives you ~160 processors. I think beyond this value the speed-up will start to decrease due to communication times.

Regards.

idefix October 23, 2013 02:35

Hello,

I tried a lot but in my case I need 320 processors to get a "fast" simulation.
If I use 160 processors I need more than twice the time as I need with 320 processor.
But still I need 5-6 sec per complete iteration step ( I count here the time between the appearing from one "Courant Number mean..." to the next in my output-file)
Is it normal that interFoam needs so much time?

Thanks a lot


All times are GMT -4. The time now is 16:15.