[McClane]

Hello,

I'm running a biphasic flow simulation through porous media with the impesFoam solver, which is a part of a toolbox for OpenFOAM (here is the link in case you are curious or you think it may be useful for you too https://openfoamwiki.net/index.php/C..._3.0_and_newer). To accelerate the process I decided to run it in parallel with mpirun and then the problem arises: some processors get ahead of others in time and "decouple" (I don't know if there is an specific term to call this phenomena, hence the quotation marks).

For example, for a simulation with time limit 20000s:
prossesor0: 14000s
prossesor1: 16400s
prossesor2: 19600s
prossesor3: 17400s
prossesor4: 14200s
prossesor5: 20000s
prossesor6: 15800s
prossesor7: 14400s

The weird part of all this is that I don't receive any kind of error message. The simulation just goes all the way until a processor hits the time threshold to stop, then all the others enter a limbo and don't update their time folders anymore. I've tried with different ways of decomposing the domain, such as scotch, hierarchy and simple, but they all have the same problem.

This is my run txt:

#!/bin/sh

# Source tutorial run functions
. $WM_PROJECT_DIR/bin/tools/RunFunctions

# case 2 - with gravity

runApplication blockMesh
cp 0/Sb.org 0/Sb
cp constant/K.org constant/K
runApplication setFields
cp constant/K 0/K
decomposePar > log.decomposePar
cp processor0/0/K processor0/constant/K
cp processor1/0/K processor1/constant/K
cp processor2/0/K processor2/constant/K
cp processor3/0/K processor3/constant/K
cp processor4/0/K processor4/constant/K
cp processor5/0/K processor5/constant/K
cp processor6/0/K processor6/constant/K
cp processor7/0/K processor7/constant/K
mpirun -np 8 impesFoam -parallel > log.impesFoam
reconstructPar > log.reconstructPar
runApplication postProcess -func sampleDict

And the decomposeParDict:

/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 5 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 8;

method scotch;

simpleCoeffs
{
n (2 1 1);
delta 0.001;
}

hierarchicalCoeffs
{
n (4 2 1);
delta 0.001;
order xyz;
}

metisCoeffs
{
processorWeights
(
1
1
1
1
1
1
1
1
)
}

manualCoeffs
{
dataFile "";
}

distributed no;

roots ( );


// ************************************************** *********************** //

I had read mpirun made sure each domain exchanged information with the neighbors to fulfill their boundary conditions, and so they must be synchronized. So it really does not make sense to me that some processors finish at a completely different time. I know the toolbox works properly, because I have tested and validated with several cases using one core. So I really don't know what is happening. Has anyone ever encountered this problem before?

If anyone thinks the problem lays in other part of the process I'm ready to provide all data you may think is relevant.

Thanks in advance