CFD Online Discussion Forums - AMI + decomposePar (ESI 2012 and 2106)

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM Pre-Processing (https://www.cfd-online.com/Forums/openfoam-pre-processing/)

- - AMI + decomposePar (ESI 2012 and 2106) (https://www.cfd-online.com/Forums/openfoam-pre-processing/239546-ami-decomposepar-esi-2012-2106-a.html)

AMI + decomposePar (ESI 2012 and 2106)

Hi,

I have this particular issue which seems somewhat random.
I have an AMI case (video link), setup for HPC: thus 2048 processors.
Runs well, runs well on 1, 16, or 2048 processors, but will give an error when decomposed for 256, 1024 or 4096 processors right after starting the first PIMPLE loop:

"Unable to set target face for source face" (I'm not starting at time zero):

Code:

No finite volume options present

Constructing face velocity Uf



Courant Number mean: 0.6672754140783199 max: 171.1290384143847



Starting time loop



writeDictionary writeDictionary1: monitoring dictionaries:

    fvSolution

    fvSchemes



Courant Number mean: 0.6672754140783199 max: 171.1290384143847

Time = 5.7216323978745969



PIMPLE: iteration 1

AMI: Creating addressing and weights between 27296 source faces and 20600 target faces

[90] 

[90] 

[90] --> FOAM FATAL ERROR: (openfoam-2106)

[90] Unable to set target face for source face 80

[90] 

[90]     From virtual bool Foam::faceAreaWeightAMI::setNextFaces(Foam::label&, Foam::label&, Foam::label&, const Foam::bitSet&, Foam::labelList&, const Foam::DynamicList<int>&, bool) const

[90]     in file AMIInterpolation/AMIInterpolation/faceAreaWeightAMI/faceAreaWeightAMI.C at line 361.

[90] 

FOAM parallel run aborting

[90] 

[90] #0  Foam::error::printStack(Foam::Ostream&) at ??:?

[90] #1  Foam::error::exitOrAbort(int, bool)[92]

Anyone already experienced this and can give me an hint?

The logs from the decomposePar don't show any notable difference.

I am curious as well to know why does that happens.

Have you run checkMesh for those cases that fail? are there any differences?
What is the decomposition method, is it scotch?
You're not starting the simulation from t=0. Have you used recontructPar on a reordered mesh?

from the log messages, the max courant number is 171!

Hi,

the case is always the same, only with different number of processors and checkMesh (also after decomposition) doesn't give particular warnings.

Yes, scotch (also tried multiLevel scotch) but it doesn't seem to affect anything.

I'm not starting from time zero because I do this for a scaling study, but the problem also appeared for time zero...

Courant is very high and that's the way I want it, that's why I use the pimple algorithm ;-)

I don't remember exactly when I experienced a similar issue getting a similar error message while trying to post-process some decomposed cases with 192 and 384 processors. But the last time, I got it to work: I took a backup of the constant directory and replaced it from a similar case with the same mesh.

Code:

mv constant bak.constant

cp -r ../similar_case_with_the_same_mesh/constant .

# run the simulation (or in my case opening paraview)

...

Thanks, you pointed me in the right direction.
Ironically, using "older" constant or polyMesh files or starting from time zero did not solve my issue, but re-exporting the mesh, using the 0.orig (also without any mesh-linked fields) and cleaning as much as possible did do the trick!
Something got somehow corrupt along the way :eek:

Glad you've solved it!

Some comments,

A] For such a high number of CPUs, try to use something else than pure scotch, maybe ptscotch or Metis. scotch method sometimes creates not a good distribution. check with decomposePar -dry-run -verbose

B] It could be linked with a trouble described here: https://develop.openfoam.com/Develop.../-/issues/1143 , but it should be fixed.

C] remedies:

C1] try to avoid decomposition of AMI using either volumetric restriction or patch restriction. see e.g. $FOAM_ETC/caseDicts/annotated/decomposeParDict

C2] turn your AMI to ACMI - if there is not a pair found, the face is blocked.

happy foaming,
Mat

Quote:

Originally Posted by louisgag (Post 816300)

Code:

No finite volume options present

Constructing face velocity Uf



Courant Number mean: 0.6672754140783199 max: 171.1290384143847



Starting time loop



writeDictionary writeDictionary1: monitoring dictionaries:

    fvSolution

    fvSchemes



Courant Number mean: 0.6672754140783199 max: 171.1290384143847

Time = 5.7216323978745969



PIMPLE: iteration 1

AMI: Creating addressing and weights between 27296 source faces and 20600 target faces

[90] 

[90] 

[90] --> FOAM FATAL ERROR: (openfoam-2106)

[90] Unable to set target face for source face 80

[90] 

[90]     From virtual bool Foam::faceAreaWeightAMI::setNextFaces(Foam::label&, Foam::label&, Foam::label&, const Foam::bitSet&, Foam::labelList&, const Foam::DynamicList<int>&, bool) const

[90]     in file AMIInterpolation/AMIInterpolation/faceAreaWeightAMI/faceAreaWeightAMI.C at line 361.

[90] 

FOAM parallel run aborting

[90] 

[90] #0  Foam::error::printStack(Foam::Ostream&) at ??:?

[90] #1  Foam::error::exitOrAbort(int, bool)[92]

Anyone already experienced this and can give me an hint?

The logs from the decomposePar don't show any notable difference.