[simpomann]

This is already implemented in my workflow but did not solve the issue for me unluckily

[capucsc]

Same here. Additionally, this doesn't explain why meshing on a smaller number of cores is successful. If the surface mesh had problems, all meshing would suffer.

-Chris

Originally Posted by simpomann

This is already implemented in my workflow but did not solve the issue for me unluckily

[wyldckat]

Greetings to all!

@Chris:

Originally Posted by capucsc

Hi Bruno,

The cases can be found here:

https://www.dropbox.com/s/4u4jk2ps6o...es.tar.gz?dl=0

Thanks for taking a look!
-Chris

Sorry for taking so long to look into this. Weird thing is that since it took me so long to look into it, it seems like the bug is already fixed in OpenFOAM 2.3.1 and 2.3.x. I haven't fully confirmed this yet, but my guess is that one or more of the following commits resulted in this issue being fixed:

• https://github.com/OpenFOAM/OpenFOAM...b5eee61b0ad57f
• https://github.com/OpenFOAM/OpenFOAM...6f4055acf95025
• https://github.com/OpenFOAM/OpenFOAM...80c2a6b8a5c208

I tested two of the cases you provided, namely:

• aoa_0.0 - tested with 2.3.0 and 2.3.1, where the first crashed and the second finished generating with success, after around 700 seconds.
• aoa_10.0 - only tested with 2.3.1... and had to break it ahead of time, because it was taking longer than 40 minutes ... but it was not stuck in an infinite loop!

In addition, I took a look at the first STL file in "aoa_0.0" and ran surfaceCheck on it. This utility pointed out a lot of reasons for concern about the STL geometry, which might be making the mesh generation taking longer than necessary. Attached is an image that shows the location I'm most concerned about, namely that concentrated number of small triangles near the centre of the image, because that level of complexity should only be needed if you were trying to simulate in such exact detail on that location of the surface.


Best regards,
Bruno

[GRAUPS]

Bruno,

Weird thing is that since it took me so long to look into it, it seems like the bug is already fixed in OpenFOAM 2.3.1 and 2.3.x.

While Chris' cases may be fixed, I can confirm that the problem still exists for larger models (>12 million cells). I have seen this many times after the github commits that you noted. I can sometimes get around the problem by changing the decomposition type or simply the number or cores I decompose to.

Unfortunately, I haven't been able to reproduce it on a generic model that I can share here, but I just wanted to note that it is still a problem I see.

(12): ERROR: dgraphFold2: out of memory (2)
(15): [18] #0  Foam::error::printStack(Foam::Ostream&)ERROR: dgraphFold2: out of memory (2)
 in "/apps/OpenFOAM/OpenFOAM-2.3.x/platforms/linux64Gcc49DPOpt/lib/libOpenFOAM.so"
[18] #1  Foam::sigSegv::sigHandler(int) in "/apps/OpenFOAM/OpenFOAM-2.3.x/platforms/linux64Gcc49DPOpt/lib/libOpenFOAM.so"
[18] #2   in "/lib64/libc.so.6"
[18] #3  _SCOTCHdgraphFold2 in "/apps/OpenFOAM/ThirdParty-2.3.x/platforms/linux64Gcc49DPOpt/lib/openmpi-1.6.5/libptscotch.so"

[wyldckat]

Greetings Brock,

Interesting... OK, then we might be able to do some trial and error debugging, by having a set of code changes that can give output relevant to debugging the issue. So please make sure you set a few copies of cases aside, where these bugs are reproducible, since the debugging code can then be tested on your side.

But first, a few questions:

1. Are you 100% certain you rebuilt OpenFOAM 2.3.x with success, after a git pull? Did you run Allwmake? And did you confirm if there weren't any build errors?
2. On Chris' geometry, there were some clear issues that gave me doubts how the latest OpenFOAM 2.3.1 and 2.3.x versions were even able to use it in the first place! Therefore, I ask you Brock to please provide us at least the output from using surfaceCheck with your geometry. Because that information can hint as to why there might be problems.
3. In addition, another detail that was considerably suspicious in Chris' cases was how the base mesh was defined, in comparison to the geometry that was being meshed. In other words, the base mesh had some considerably large cells, where only a few of them would intersect the geometry. I was amazed as to how snappyHexMesh was able to iteratively "enhance" the mesh resolution every iteration and gradually pick up new surfaces that could be intersected.
   The request here would be if you could prepare a dummy case that has the same geometrical, base mesh and similar refinement levels. In other words, swapping out your geometry by a bunch of STL cubes/bricks

Best regards,
Bruno

[fabian_roesler]

Hi there

Thanks for this enlightening thread. We do huge simulations with up to 150M cells. Of course, the meshing has to be done in parallel and with the same number of processors used for the simulation to avoid redistribution of the cells.
From time to time we suffer the same error like you do.

(20): ERROR: dgraphFold2: out of memory (2)
(30): ERROR: dgraphFold2: out of memory (2)
(26): ERROR: dgraphFold2: out of memory (2)
(31): ERROR: dgraphFold2: out of memory (2)
[25] #0  Foam::error::printStack(Foam::Ostream&)--------------------------------------------------------------------------
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.  

The process that invoked fork was:

  Local host:          XYZ (PID 24527)
  MPI_COMM_WORLD rank: 25

In our case, we get rid of the errors by optimizing the STL files. SnappyHexMesh has problems with very high aspect shell faces. The former erroneous case meshes perfectly after reduction of high aspect and skew shell faces.
Unfortunately there is not tool in OpenFOAM to optimize shell mesh quality or at least I don't know one. We handle all STL files in ANSA.
Hope this helps a little bit.

Cheers

Fabian

[capucsc]

After spending a considerable time working on how STLs are produced, I can agree with Fabian. I rarely encounter this if surfaceCheck comes back with no errors. I actually got through ~300 4M cell meshes without an error (also OF2.3.X). I would love to see some tools built into OF that help clean up surfaces meshes, but I also understand that is a very difficult task.

I've also tried cfMesh, which seems to have fewer issues with bad surface geometries. If cfMesh works for your application and you're seeing these errors, it might be worth trying.

[GRAUPS]

All,

I've finally been able to reproduce these errors reliably (I've tried on three separate machines) on a somewhat small simple pipe model I can share. I'm hoping you all (and the developers) will be able to reproduce my errors. I created a bug report for it...

http://www.openfoam.org/mantisbt/view.php?id=1792

If anyone with more OF code knowledge would like to take a hack at tracking it down, please go for it! Thanks again for everyone's help!

[smog]

I am facing the same problem with OF240. I can snappyHexmesh with 4 processors, but cannot do the same with 8. It just runs out of memory! Any fix yet?

[GRAUPS]

shashank,

Per the bug report, we think the bug is in the thirdparty scotch decomposition library. Mattijs noted that he's heard that scotch 6.0.4 resolved some of these errors. However, I have been unsuccessful in getting scotch 6.0.4 to compile properly thus far. And judging by the comment in OpenFOAM-dev on commit 38415f8, it looks like OpenFOAM-dev hasn't been upgraded to 6.0.4 for the same reason.

In any case, the workaround for me at the moment is to change the decomposition type to simple. This is annoying because you then have to specify the xyz decomps, but it's a workaround for now.

Brock

[smog]

I worked around it using hierarchical decomposition. In my opinion, it has worked better than scotch for me.

[GRAUPS]

All, as of commit 1da3d4a in 2.4.x and commit e64a892 in dev, I can no longer reproduce this problem on my test models. Shout out to Henry for resolving my bug report and committing the code! Hopefully it's resolved for good.

[clktp]

Hello,

I have just got this error.

Ubuntu 14.04 LTS
Openfoam 2.4.0
x86_64

Base mesh number of cells 187k
decomposition model scotch
number of processors 8

Funny thing is I have been using this stl file for a couple of weeks and I didn't have any problem, later I just made the stl file a bit longer (it is a flat plate with something in the middle of the domain) and change its direction by Blender and received this error. Now I've solved it by using simple method and I thought it may be useful to write here.

[wyldckat]

Originally Posted by clktp

Funny thing is I have been using this stl file for a couple of weeks and I didn't have any problem, later I just made the stl file a bit longer (it is a flat plate with something in the middle of the domain) and change its direction by Blender and received this error. Now I've solved it by using simple method and I thought it may be useful to write here.

Quick answer: Check the values you have for the max local and max global number of cells, namely these entries: https://github.com/OpenFOAM/OpenFOAM...exMeshDict#L81

    // If local number of cells is >= maxLocalCells on any processor
    // switches from from refinement followed by balancing
    // (current method) to (weighted) balancing before refinement.
    maxLocalCells 100000;

    // Overall cell limit (approximately). Refinement will stop immediately
    // upon reaching this number so a refinement level might not complete.
    // Note that this is the number of cells before removing the part which
    // is not 'visible' from the keepPoint. The final number of cells might
    // actually be a lot less.
    maxGlobalCells 2000000;

Expect roughly 1GB of RAM per million cells, which in this example, it would mean that 100MB per subdomain and 2GB of total RAM would be roughly the maximum that snappy was allowed to use.

[oskar]

Been running into this issue a lot with 10-30M cell parallel cases.


After looking into it I'm relatively sure it has something to do with the angles between triangles of the stl file, includedAngle for surfaceFeatureExtract, and resolveFeatureAngle in castellatedMeshControls for snappyHexMesh.


My guess would be that in some cases these angles create a loop or logic error or something that causes the generation to crash.


Most likely to occur if all three of those angles are close to each other. And seems to work better (crash less) the further they are spaced out, like 5-15deg from each other.