[natty_king]

Hi all,

I've got what seems to be a rather specific problem as I've done a bit of searching and can't find anything that's similar.

I'm running a slightly modified case of the turbine_siting tutorial with a different terrain model. During the snappyHexMesh process it seems to get to the 'Doing final balancing' stage and then freezes with the last output being 'Found 0 zones faces to keep together.'. I've included my sHM log below.

The strange thing is I have had the exact same case file running with the only difference being that I have rotated the terrain to simulate different wind directions. So far I've run the case file for 4 directions each at 10 velocities, N, NW, W and SW but on the S direction I am having this problem.

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.2.2                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.2.2-9240f8b967db
Exec   : snappyHexMesh -overwrite -parallel
Date   : Apr 03 2014
Time   : 16:43:01
Host   : "andrew-pc"
PID    : 3543
Case   : /home/andrew/OpenFOAM/andrew-2.2.2/run/Final/S/v3
nProcs : 6
Slaves : 
5
(
"andrew-pc.3544"
"andrew-pc.3545"
"andrew-pc.3546"
"andrew-pc.3547"
"andrew-pc.3548"
)

Pstream initialized with:
    floatTransfer      : 0
    nProcsSimpleSum    : 0
    commsType          : nonBlocking
    polling iterations : 0
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Disallowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Read mesh in = 0.09 s

Overall mesh bounding box  : (-4000 -4000 -400) (4000 4000 3600)
Relative tolerance         : 1e-06
Absolute matching distance : 0.012

Reading refinement surfaces.
Read refinement surfaces in = 0.28 s

Reading refinement shells.
Read refinement shells in = 0 s

Setting refinement level of surface to be consistent with shells.
Checked shell refinement in = 0 s

Reading features.
Read features in = 0 s


Determining initial surface intersections
-----------------------------------------

Edge intersection testing:
    Number of edges             : 1520000
    Number of edges to retest   : 1520000
    Number of intersected edges : 11058
Calculated surface intersections in = 0.66 s

Initial mesh : cells:500000  faces:1520000  points:520251
Cells per refinement level:
    0    500000

Adding patches for surface regions
----------------------------------

Patch    Type    Region
-----    ----    ------
terrain:

5    wall    terrain_final_total

Added patches in = 0.02 s

Selecting decompositionMethod ptscotch

Refinement phase
----------------

[1] Found point (0 0 200) in cell 13454 on processor 1
[4] Found point (0 0 200) in cell 11566 on processor 4

Surface refinement iteration 0
------------------------------

Marked for refinement due to surface intersection          : 20000 cells.
Determined cells to refine in = 0.22 s
Selected for refinement : 20000 cells (out of 500000)
Edge intersection testing:
    Number of edges             : 1974344
    Number of edges to retest   : 572750
    Number of intersected edges : 44386
Refined mesh in = 0.85 s
After refinement surface refinement iteration 0 : cells:640000  faces:1974344  points:694800
Cells per refinement level:
    0    480000
    1    160000
Skipping balancing since max unbalance 0.00512815705098 is less than allowable 0.1

Surface refinement iteration 1
------------------------------

Marked for refinement due to surface intersection          : 80000 cells.
Determined cells to refine in = 0.06 s
Selected for refinement : 84453 cells (out of 640000)
Edge intersection testing:
    Number of edges             : 3883529
    Number of edges to retest   : 2432004
    Number of intersected edges : 177485
Refined mesh in = 2.12 s
After refinement surface refinement iteration 1 : cells:1231171  faces:3883529  points:1422058
Cells per refinement level:
    0    475547
    1    115624
    2    640000
Skipping balancing since max unbalance 0.0121153049538 is less than allowable 0.1

Surface refinement iteration 2
------------------------------

Marked for refinement due to surface intersection          : 0 cells.
Determined cells to refine in = 0.09 s
Selected for refinement : 0 cells (out of 1231171)
Stopping refining since too few cells selected.


Removing mesh beyond surface intersections
------------------------------------------

Found point (0 0 200) in cell -1 in global region 1 out of 2 regions.
Keeping all cells in region 1 containing point (0 0 200)
Selected for keeping : 1014976 cells.
Edge intersection testing:
    Number of edges             : 3235335
    Number of edges to retest   : 176904
    Number of intersected edges : 177485

Shell refinement iteration 0
----------------------------

Marked for refinement due to distance to explicit features : 0 cells.
Marked for refinement due to refinement shells             : 0 cells.
Determined cells to refine in = 1.83 s
Selected for internal refinement : 0 cells (out of 1014976)
Stopping refining since too few cells selected.


Splitting mesh at surface intersections
---------------------------------------

Introducing baffles for 177485 faces that are intersected by the surface.

Edge intersection testing:
    Number of edges             : 3412794
    Number of edges to retest   : 1316481
    Number of intersected edges : 354944
Created baffles in = 1.36 s


After introducing baffles : cells:1014976  faces:3412794  points:1206378
Cells per refinement level:
    0    466835
    1    58390
    2    489751

Introducing baffles to block off problem cells
----------------------------------------------

markFacesOnProblemCells : marked 354510 additional internal faces to be converted into baffles.
Analyzed problem cells in = 0.47 s


Introducing baffles to delete problem cells.

Edge intersection testing:
    Number of edges             : 3767304
    Number of edges to retest   : 1065461
    Number of intersected edges : 354944
Created baffles in = 1.31 s


After introducing baffles : cells:1014976  faces:3767304  points:1207780
Cells per refinement level:
    0    466835
    1    58390
    2    489751

Remove unreachable sections of mesh
-----------------------------------

Keeping all cells in region 3 containing point (0 0 200)
Selected for keeping : 837846 cells.
Edge intersection testing:
    Number of edges             : 2702095
    Number of edges to retest   : 0
    Number of intersected edges : 177350
Split mesh in = 1.25 s


After subsetting : cells:837846  faces:2702095  points:1027256
Cells per refinement level:
    0    466835
    1    58088
    2    312923

Handling cells with snap problems
---------------------------------

Introducing baffles for 177350 faces that are intersected by the surface.

Edge intersection testing:
    Number of edges             : 2702095
    Number of edges to retest   : 658093
    Number of intersected edges : 177350
Created baffles in = 0.89 s


After introducing baffles : cells:837846  faces:2702095  points:1027256
Cells per refinement level:
    0    466835
    1    58088
    2    312923

Introducing baffles to block off problem cells
----------------------------------------------

markFacesOnProblemCells : marked 0 additional internal faces to be converted into baffles.
Analyzed problem cells in = 0.34 s


Introducing baffles to delete problem cells.

Edge intersection testing:
    Number of edges             : 2702095
    Number of edges to retest   : 0
    Number of intersected edges : 177350
Created baffles in = 0.62 s


After introducing baffles : cells:837846  faces:2702095  points:1027256
Cells per refinement level:
    0    466835
    1    58088
    2    312923

Remove unreachable sections of mesh
-----------------------------------

Keeping all cells in region 0 containing point (0 0 200)
Selected for keeping : 837846 cells.
Edge intersection testing:
    Number of edges             : 2702095
    Number of edges to retest   : 0
    Number of intersected edges : 177350
Split mesh in = 0.98 s


After subsetting : cells:837846  faces:2702095  points:1027256
Cells per refinement level:
    0    466835
    1    58088
    2    312923
dupNonManifoldPoints : Found : 14 non-manifold points (out of 1046310)
Edge intersection testing:
    Number of edges             : 2702095
    Number of edges to retest   : 0
    Number of intersected edges : 177350
getDuplicateFaces : found 0 pairs of duplicate faces.

Detected unsplittable baffles : 0

Merge refined boundary faces
----------------------------

Merging 0 sets of faces.

No faces merged ...

Merging all points on surface that
- are used by only two boundary faces and
- make an angle with a cosine of more than 0.707106781187.

Removing 1 straight edge points ...

Edge intersection testing:
    Number of edges             : 2702095
    Number of edges to retest   : 0
    Number of intersected edges : 177350

Undo iteration 0
----------------
Checking faces in error :
    non-orthogonality >  65 degrees                        : 0
    faces with face pyramid volume < 1e-13                 : 0
    faces with face-decomposition tet quality < 1e-30      : 0
    faces with concavity >  80 degrees                     : 0
    faces with skewness >   4 (internal) or  20 (boundary) : 0
    faces with interpolation weights (0..1)  <  0.05       : 0
    faces with volume ratio of neighbour cells <  0.01     : 0
    faces with face twist <  0.05                          : 0
    faces on cells with determinant < 0.001                : 0

Doing final balancing
---------------------

Found 0 zoned faces to keep together.

Thanks in advance!

[wyldckat]

Greetings Andrew,

I don't know if you've managed to solve this yet, but my suggestion is to try OpenFOAM 2.2.x, or even 2.3.0 or 2.3.x.
I say this because there were several bugs fixed in 2.2.x since 2.2.2 was released and I have a very vague recollection that this particular issue might have already been fixed... let me see if I can find the commit...

OK... there are at least 5 to 10 commits that might be the solution for this problem, all released in 2.2.x after 2.2.2. Listing them all is a bit of a pain, so perhaps by date:

• 2013-10-02
• 2013-10-30
• 2013-12-13
• 2013-12-16
• 2014-01-09
• 2014-01-22

At some dates, there is more than just one commit that might affect the detail you're triggering.

Best regards,
Bruno


PS: I only noticed this thread of yours, because of your second post here on the forum... and when I came to check your posts, I saw this interesting thread!

[natty_king]

Hi Bruno,

It still seemed very temperamental but I managed to get all of the simulations I needed in the end through perseverance!

I was holding off changing the version of openFoam as I was halfway through my dissertation, but I will definitely upgrade now that I have all of the data I need and all my simulations are run.


Thanks very much for the info!

Kind regards,
Andrew

[wildfire230]

Hi All,


Sorry to revise this old thread, but I am facing a similar problem with OpenFOAM-v1812. I am meshing a very large problem, roughly ~1 billion cells, and every time snappyHexMesh seems to get stuck at the final balancing step. This is an example of the final lines of output before the code simply hangs for hours:

Remove unreachable sections of mesh
-----------------------------------

Split mesh in = 419.32 s


After subsetting : cells:1165071507 faces:3851646799 points:1531389731
Cells per refinement level:
0 407633
1 314921
2 5828180
3 57571996
4 46441101
5 80069357
6 166764841
7 257115766
8 550557712

Merge free-standing baffles
---------------------------

freeStandingBaffles : detected 0 free-standing baffles out of 0

Detected free-standing baffles : 0
Merged free-standing baffles in = 5.69 s


dupNonManifoldPoints : Found : 0 non-manifold points (out of 1550575881)
Detected unsplittable baffles : 0

Merge refined boundary faces
----------------------------

Merging 0 sets of faces.

No faces merged ...

Merging all points on surface that
- are used by only two boundary faces and
- make an angle with a cosine of more than 0.7071067811865476.

No straight edges simplified and no points removed ...

Doing final balancing
---------------------

Found 291445830 zoned faces to keep together.
Found 0 separated coupled faces to keep together.

Does anyone have any suggestions for how to fix this problem?



Thanks so much.

[wildfire230]

Has anyone seen this issue? I just tried another job. It's running on 30 nodes, 2 processors per node, 128 GB RAM per node, with roughly 1 billion cells in the domain, and after about 6-8 hours snappyHexMesh successfully makes it through almost all of the castellation step, and then hangs at the "Doing final balancing" step for many hours. Again, I am using v1906.


Thanks

[wildfire230]

Sorry, one more update. After quite a few hours, snappyHexMesh finally threw up this error message:

Doing final balancing
---------------------

Found 249507988 zoned faces to keep together.
Found 0 separated coupled faces to keep together.
--> FOAM Warning :
From function virtual Foam::labelList Foam::hierarchGeomDecomp::decompose(const pointField&, const scalarField&) const
in file hierarchGeomDecomp/hierarchGeomDecomp.C at line 822

Encountered 3 occurrences where the desired decomposition split could not be properly satisfied
new cannot satisfy memory request.
This does not necessarily mean you have run out of virtual memory.
It could be due to a stack violation caused by e.g. bad use of pointers or an out of date shared library

It appears like this final balancing step is failing for some reason and then the simulation crashed with some type of memory error. Does anyone have any ideas??? What does this final balancing step actually do? If it just has to do with redistributing the mesh evenly across the processor cores is there any way to turn it off?


Thanks.

[wyldckat]

Originally Posted by wildfire230

It appears like this final balancing step is failing for some reason and then the simulation crashed with some type of memory error. Does anyone have any ideas??? What does this final balancing step actually do? If it just has to do with redistributing the mesh evenly across the processor cores is there any way to turn it off?

Quick answers:

1. First suspect is the label (integer) type you're using in your build... did you build with 32-bit labels or 64-bit labels? Because you will never be able to reconstruct or redistribute a mesh with 1 billion cells using 32-bit labels.
2. Second suspect is the decomposition method you defined in "decomposeParDict". Did you use "ptscotch", "scotch" or some other? Because:
   • "scotch" will unlikely work properly in parallel;
   • "ptscotch" could potentially have a limitation on this... possibly due to the same issue with using or not 64-bit labels.
   • edit: oops, didn't read the error message properly... you used hierarchical"... try "ptscotch" instead.
3. Third suspect is that this is something that was never tested by OpenFOAM developers... hence the issue you're triggering. Since you're using v1906, you can use their issue tracker to report this: https://develop.openfoam.com/Develop...M-plus/issues/ - but please report only after checking the first two issues I've mentioned.

[wildfire230]

Hi Bruno,


Thanks so much for taking the time to give me some ideas.


Quick responses:
1. I am compiled for 64-bit labels.
2. It looks like "ptscotch" is not a valid decomposition method in v1906? At least I receive this error when decomposing:

--> FOAM FATAL ERROR:
Unknown decompositionMethod ptscotch

Valid decompositionMethods :
10(hierarchical kahip manual metis multiLevel none random scotch simple structured)

If you have any other possible suggestions, they would be greatly appreciated. Could you also please explain what exactly this "doing final balancing" step actually does? I have maxLoadUnbalance set to 1, which means we shouldn't ever do any balancing, or at least that was my impression.


Thanks again

[wyldckat]

Quick answers:

Originally Posted by wildfire230

2. It looks like "ptscotch" is not a valid decomposition method in v1906? At least I receive this error when decomposing:

Oh, sorry, I don't use the versions from OpenFOAM.com on a regular basis, so I wasn't aware of this... or at least I didn't remember about this detail... then that means that the "scotch" option auto-switches from serial to parallel modes automatically.

Originally Posted by wildfire230

If you have any other possible suggestions, they would be greatly appreciated. Could you also please explain what exactly this "doing final balancing" step actually does? I have maxLoadUnbalance set to 1, which means we shouldn't ever do any balancing, or at least that was my impression.

Curious... here's one of the parts of the code that handles balancing:

         scalar nIdealCells =
             mesh_.globalData().nTotalCells()
           / Pstream::nProcs();
  
         scalar unbalance = returnReduce
         (
             mag(1.0-mesh_.nCells()/nIdealCells),
             maxOp<scalar>()
         );
  
         if (unbalance <= maxLoadUnbalance)
         {
             Info<< "Skipping balancing since max unbalance " << unbalance
                 << " is less than allowable " << maxLoadUnbalance
                 << endl;
         }

So the problem is that there were processors that had zero new cells added, which results in the maximum 1.0 being reached... although it's strange... but maybe it's right?

If you believe that all processors were meant to have added new refinement cells, then what you can do is recompile the snappyHexMesh library, after making the following changes:

• Here: https://develop.openfoam.com/Develop...Refine.C#L2526 - you can make a few outputs before the if block, to see what values were achieved, e.g. by using this line before the if block:
  

  Pout << "nNewCells: " << nNewCells << ", vs nIdealNewCells: " << nIdealNewCells << endl;

• Likewise, here: https://develop.openfoam.com/Develop...Refine.C#L2422
  With this line:
  

  Pout << "nNewCells: " << mesh_.nCells() << ", vs  nIdealNewCells: " << nIdealNewCells << endl;

This will help figure out if something is going wrong with the way the calculations are being done. The "Pout" will output the values from all parallel processors.


With this information, it's possible to assess if the issue that is happening is meant to happen or not.

[wildfire230]

Hi Bruno,


Thanks again. I believe the final balancing step comes from here:

if (!dryRun_ && Pstream:: parRun())
{
    Info<< nl
        << "Doing final balancing" << nl
        << "---------------------" << nl
        << endl;

    // Do final balancing. Keep zoned faces on one processor since the
    // snap phase will convert them to baffles and this only works for
    // internal faces.
    meshRefiner_.balance
    (
        true,                           // keepZoneFaces
        false,                          // keepBaffles
        scalarField(mesh.nCells(), 1),  // cellWeights
        decomposer_,
        distributor_
    );
}

I don't think it cares about maxLoadUnbalance, and it runs for any parallel case as far as I can tell.

Could you please answer one quick question? If I manually subdivide my domain and run snappyHexMesh on the different subdomains, will the refined cells line up perfectly with the adjacent subdomains?

Thanks again, I've been struggling with meshing this problem for several months and I'm running out of ideas.

Best regards

[wyldckat]

Quick answers:

Originally Posted by wildfire230

I don't think it cares about maxLoadUnbalance, and it runs for any parallel case as far as I can tell.

I didn't search for the message itself... audible face-palm...

You're right, it doesn't care about the balance limit. So one alternative is that you edit the source code and comment out that block, so that it doesn't re-balance the mesh.

Although this is something that should really be reported to the development team: https://develop.openfoam.com/Develop...M-plus/issues/

Originally Posted by wildfire230

Could you please answer one quick question? If I manually subdivide my domain and run snappyHexMesh on the different subdomains, will the refined cells line up perfectly with the adjacent subdomains?

The cells will not line up properly if they are nearby parts of the mesh if... OK, so:

• If you let it proceed with the snapping phase, then you will have problems, since the mesh will be smoothed over...
• If you only do the castellation, then the refinements should match... as long as there aren't any inconsistencies between meshed regions, for example, a refinement block that had a distance effect that resulted in a refinement toward the next region...

But the trick should be to hack into the code and rebuild the library and move onward. It's a bit of a nasty hack, but at least you're able to mesh it.


If it's a system-wide installation, you can copy the 2 main folders for snappyHexMesh and modify the 2 files inside "Make" accordingly to the usual workflow for custom code.

[egebat7]

Hi there,

I am running my case with about 2000 cores and I am also facing the same issue. Is rebuilding the library really the best way to avoid final balancing? I am working with a cluster and personally this is not the best way to approach this issue.

Best regards,
Ege