AMI + decomposePar (ESI 2012 and 2106)

louisgag · November 11, 2021, 05:21

Hi,

I have this particular issue which seems somewhat random.
I have an AMI case (video link), setup for HPC: thus 2048 processors.
Runs well, runs well on 1, 16, or 2048 processors, but will give an error when decomposed for 256, 1024 or 4096 processors right after starting the first PIMPLE loop:

"Unable to set target face for source face" (I'm not starting at time zero):

Code:

No finite volume options present
Constructing face velocity Uf

Courant Number mean: 0.6672754140783199 max: 171.1290384143847

Starting time loop

writeDictionary writeDictionary1: monitoring dictionaries:
    fvSolution
    fvSchemes

Courant Number mean: 0.6672754140783199 max: 171.1290384143847
Time = 5.7216323978745969

PIMPLE: iteration 1
AMI: Creating addressing and weights between 27296 source faces and 20600 target faces
[90] 
[90] 
[90] --> FOAM FATAL ERROR: (openfoam-2106)
[90] Unable to set target face for source face 80
[90] 
[90]     From virtual bool Foam::faceAreaWeightAMI::setNextFaces(Foam::label&, Foam::label&, Foam::label&, const Foam::bitSet&, Foam::labelList&, const Foam::DynamicList<int>&, bool) const
[90]     in file AMIInterpolation/AMIInterpolation/faceAreaWeightAMI/faceAreaWeightAMI.C at line 361.
[90] 
FOAM parallel run aborting
[90] 
[90] #0  Foam::error::printStack(Foam::Ostream&) at ??:?
[90] #1  Foam::error::exitOrAbort(int, bool)[92]

Anyone already experienced this and can give me an hint?

The logs from the decomposePar don't show any notable difference.

s1291 · November 11, 2021, 09:35

I am curious as well to know why does that happens.

Have you run checkMesh for those cases that fail? are there any differences?
What is the decomposition method, is it scotch?
You're not starting the simulation from t=0. Have you used recontructPar on a reordered mesh?

from the log messages, the max courant number is 171!

louisgag · November 11, 2021, 10:36

Hi,

the case is always the same, only with different number of processors and checkMesh (also after decomposition) doesn't give particular warnings.

Yes, scotch (also tried multiLevel scotch) but it doesn't seem to affect anything.

I'm not starting from time zero because I do this for a scaling study, but the problem also appeared for time zero...

Courant is very high and that's the way I want it, that's why I use the pimple algorithm ;-)

s1291 · November 11, 2021, 12:53

I don't remember exactly when I experienced a similar issue getting a similar error message while trying to post-process some decomposed cases with 192 and 384 processors. But the last time, I got it to work: I took a backup of the constant directory and replaced it from a similar case with the same mesh.

Code:

mv constant bak.constant
cp -r ../similar_case_with_the_same_mesh/constant .
# run the simulation (or in my case opening paraview)
...

louisgag · November 12, 2021, 04:57

Thanks, you pointed me in the right direction.
Ironically, using "older" constant or polyMesh files or starting from time zero did not solve my issue, but re-exporting the mesh, using the 0.orig (also without any mesh-linked fields) and cleaning as much as possible did do the trick!
Something got somehow corrupt along the way

s1291 · November 12, 2021, 05:26

Glad you've solved it!

matejfor · November 19, 2021, 10:26

Some comments,

A] For such a high number of CPUs, try to use something else than pure scotch, maybe ptscotch or Metis. scotch method sometimes creates not a good distribution. check with decomposePar -dry-run -verbose

B] It could be linked with a trouble described here: https://develop.openfoam.com/Develop.../-/issues/1143 , but it should be fixed.

C] remedies:

C1] try to avoid decomposition of AMI using either volumetric restriction or patch restriction. see e.g. $FOAM_ETC/caseDicts/annotated/decomposeParDict

C2] turn your AMI to ACMI - if there is not a pair found, the face is blocked.

happy foaming,
Mat

Quote:

Originally Posted by louisgag

Hi,

I have this particular issue which seems somewhat random.
I have an AMI case (video link), setup for HPC: thus 2048 processors.
Runs well, runs well on 1, 16, or 2048 processors, but will give an error when decomposed for 256, 1024 or 4096 processors right after starting the first PIMPLE loop:

"Unable to set target face for source face" (I'm not starting at time zero):

Code:

No finite volume options present
Constructing face velocity Uf

Courant Number mean: 0.6672754140783199 max: 171.1290384143847

Starting time loop

writeDictionary writeDictionary1: monitoring dictionaries:
    fvSolution
    fvSchemes

Courant Number mean: 0.6672754140783199 max: 171.1290384143847
Time = 5.7216323978745969

PIMPLE: iteration 1
AMI: Creating addressing and weights between 27296 source faces and 20600 target faces
[90] 
[90] 
[90] --> FOAM FATAL ERROR: (openfoam-2106)
[90] Unable to set target face for source face 80
[90] 
[90]     From virtual bool Foam::faceAreaWeightAMI::setNextFaces(Foam::label&, Foam::label&, Foam::label&, const Foam::bitSet&, Foam::labelList&, const Foam::DynamicList<int>&, bool) const
[90]     in file AMIInterpolation/AMIInterpolation/faceAreaWeightAMI/faceAreaWeightAMI.C at line 361.
[90] 
FOAM parallel run aborting
[90] 
[90] #0  Foam::error::printStack(Foam::Ostream&) at ??:?
[90] #1  Foam::error::exitOrAbort(int, bool)[92]

Anyone already experienced this and can give me an hint?

The logs from the decomposePar don't show any notable difference.

November 11, 2021, 09:35		#2
s1291 Member Join Date: Aug 2017 Location: Algeria Posts: 98 Rep Power: 8	I am curious as well to know why does that happens. Have you run checkMesh for those cases that fail? are there any differences? What is the decomposition method, is it scotch? You're not starting the simulation from t=0. Have you used recontructPar on a reordered mesh? from the log messages, the max courant number is 171!

November 11, 2021, 12:53		#4
s1291 Member Join Date: Aug 2017 Location: Algeria Posts: 98 Rep Power: 8	I don't remember exactly when I experienced a similar issue getting a similar error message while trying to post-process some decomposed cases with 192 and 384 processors. But the last time, I got it to work: I took a backup of the constant directory and replaced it from a similar case with the same mesh. Code: mv constant bak.constant cp -r ../similar_case_with_the_same_mesh/constant . # run the simulation (or in my case opening paraview) ... louisgag likes this. Last edited by s1291; November 11, 2021 at 13:55.

November 12, 2021, 04:57		#5
louisgag Senior Member Louis Gagnon Join Date: Mar 2009 Location: Stuttgart, Germany Posts: 338 Rep Power: 18	Thanks, you pointed me in the right direction. Ironically, using "older" constant or polyMesh files or starting from time zero did not solve my issue, but re-exporting the mesh, using the 0.orig (also without any mesh-linked fields) and cleaning as much as possible did do the trick! Something got somehow corrupt along the way s1291 likes this.

November 12, 2021, 05:26		#6
s1291 Member Join Date: Aug 2017 Location: Algeria Posts: 98 Rep Power: 8	Glad you've solved it! __________________ "When in doubt, use brute force." -- Ken Thompson

November 11, 2021, 10:36		#3
louisgag Senior Member Louis Gagnon Join Date: Mar 2009 Location: Stuttgart, Germany Posts: 338 Rep Power: 18	Hi, the case is always the same, only with different number of processors and checkMesh (also after decomposition) doesn't give particular warnings. Yes, scotch (also tried multiLevel scotch) but it doesn't seem to affect anything. I'm not starting from time zero because I do this for a scaling study, but the problem also appeared for time zero... Courant is very high and that's the way I want it, that's why I use the pimple algorithm ;-)