snappyhexmesh: Running out of memory (without reason?)
 

Hi,

I'm using SHM for OF 2.1.1 out of box on a VirtualBox machine on Windows 7 with 48 GB RAM and 8 physical cores. 40 GB and all cores are enabled for VirtualBox.

My model runs fine with 2 or 4 cores. When I switch to 8 cores SHM aborts (while not even 10% RAM are used) randomly like this:

I cannot even find this error in the forum. Any hints?

Thanks
Marco

Hi Marco,

A few questions:

1. Which decomposition method are you using while snappyHexMesh is running?
2. Which OpenFOAM version are you using?
3. What Linux distribution and architecture are you using?

Best regards,
Bruno

Hi Bruno,

thanks for your answer.

1. I tried ptscotch. with simple there is no such error. So finally this is an acceptable solution.
2. 2.1.1.
3. the ubuntu version this OF version was built on officially.

Marco

Hi Marco,

Which architecture of Ubuntu? i386/i686 or x86_64/amd64? You can check by running:
If you are using i386/i686, it's only natural that it crashes when it reaches 2GiB of RAM per process.

Best regards,
Bruno

snappyHexMesh - ERROR: dgraphFold2: out of memory (2)
 

Well, now I have encountered the same problem. Trying to get the meshing with snappyHexMesh, I got:

[QUOTE]...
Feature refinement iteration 24
------------------------------

Marked for refinement due to explicit features : 92 cells.
Determined cells to refine in = 13.28 s
Selected for feature refinement : 136 cells (out of 10647165)
Edge intersection testing:
Number of edges : 36562736
Number of edges to retest : 8226
Number of intersected edges : 4230508
Refined mesh in = 2.76 s
After refinement feature refinement iteration 24 : cells:10648117 faces:36562736 points:15512183
Cells per refinement level:
0 1149402
1 228530
2 967512
3 2859801
4 5442872
(14): ERROR: dgraphFold2: out of memory (2)
(13): (16): ERROR: (9): (15): ERROR: dgraphFold2: out of memory (3)
dgraphFold2: out of memory (2)ERROR: [17]
dgraphFold2: out of memory (2)ERROR:
dgraphFold2: out of memory (2)#[11] #
0 0 Foam::error: printStack(Foam::Ostream&)Foam::error: printStack(Foam::Ostream&)--------------------------------------------------------------------------
mpirun has exited due to process rank 9 with PID 23598 on
...
[/QUOTE]

The only available information seems to be related with the scotch library, but not much info really.

In my case, I am running snappyHexMesh in a cluster using 64bit (x86_64), 36 cores, and requesting 4GB per core (way too decent for this type of mesh)... and still... ohh surprise, out of memory problem. Does anyone know how to solve this?

Regards,

Greetings caduqued,

If you ran in parallel, then the library "ptscotch" does have some limitations.

By the way, which OpenFOAM version are you using? Because versions older than OpenFOAM 2.2.0 are using Scotch 5.1.1; while as of OpenFOAM 2.2.0, it uses Scotch 6.0.0, which is allegedly (missing reference ;)) a lot better!

Best regards,
Bruno

Hi Bruno,

Thanks for your quick response. Yes, I am now using OpenFOAM 2.2.0, with (in fact) scotch 2.0.0. The parallelization method I am using is scotch (not ptscotch, as it used to be before), but I assume that is irrelevant, as it is just the name of the method (library being always scotch). With this hindsight, I can now (sadly !) confirm that it seems this new scotch version also exhibits some problems, although I must confess didn't know about them. So, in the meantime, waiting for a fix, I assume it will be safer to just stick to one of the traditional methods, right?

Regards,

Carlos

Hi Carlos,

I forgot that they did rename "ptscotch" to "scotch" when used in parallel, so that the same dictionary could be used for the decomposition.

The only other thing I can remember about is the "multilevel" decomposition strategy... I mentioned it some time ago here: http://www.cfd-online.com/Forums/ope...tml#post367979 - post #8. It's not the silver bullet, but if used correctly, it might help to get the best of both worlds...

Best regards,
Bruno

Hi Bruno,

Thanks again. I did not know about the multilevel strategy, thanks a lot for bring my attention to it!!! (with openFOAM every day is a learning day!!!)

I will try it...

Regards,

Carlos

about dgraphFold2: out of memory
 

Hello, caduqued, about the dgraphFold2: out of memory error, have you solved it? I met this problem and couldn't find a solution.:(

Greetings Slanth,

You'll have to provide more specific details about the case you're meshing, because the error message:
can occur for a lot of reasons.
It is even possible that the problem is:

1. With your base mesh.
2. With the decomposition method being used.
3. With how many cores being used.
4. With which OpenFOAM version you're using.

Best regards,
Bruno

I likewise encounter this error frequently. I'm meshing on a 32 core Opteron machine. I used 'scotch' for decomposition and I work with OF2.3.0. I get messages like:

about 4 out of 10 runs. I've watched this happen with top running, and as other commented have noted, there isn't any issue with available RAM. It would be great to work out a solution, as this is the most unreliable aspect of using OF.

Other details:
uname -a:
ldd /home/someuser/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/bin/snappyHexMesh:

Greetings capucsc,

You're able to reproduce this error a lot more than I was expecting :eek:
Any chance you have one of those examples that have this issue in a small file/case size and be allowed to share it with us, either publicly or privately? This way it would make it a lot easier to try and diagnose the source of this problem!

Best regards,
Bruno

Hi Bruno,

The cases can be found here:

https://www.dropbox.com/s/4u4jk2ps6o...es.tar.gz?dl=0

Thanks for taking a look!
-Chris

Same issue
 

Gentlemen,

I'm also seeing this same issue on a meshing case every once in a while. I'm on an Intel 16 core machine using OF 2.3.x and the scotch decomposition. Unfortunately, I don't have a case I can share at the moment, but please do let me know if a public bug report gets created for this so that I can keep an eye on it. If I come across a case I can share, I will do so!

Thanks!

Brock

I believe, although I have little in way of evidence, that the occurrence of this error is tied to the number of cores used. I never encounter problems when running with 6 cores, but about 70% of runs fail when using 32 cores. If we were to make a bug report, where would it go? OpenFOAM or scotch or somewhere else?

The main place to report bugs for openfoam is here...

http://www.openfoam.org/mantisbt/

But if you report something, make sure you give the developer everything he needs to fix or at least diagnose the problem. It's best if you can isolate the problem down to a small model and then give them that model to debug with. I've had some good luck in reporting, one of the bugs I reported was fixed within 24 hrs!

Brock

Hey,
I encountered the same problem today while also using scotch decomposition with 32 processors on a 64bit cluster headnode. Before, I did a similiar case with the same geometry, but I split up the .stl-files and now I get this error every time.

Case is running if i switch to 27 cores though.

Is there a solution somewhere right now? I looked through the bug tracker but couldnt find a ticket.

I have yet to pinpoint the exact reason this happens, so I haven't submitted a bug report. Debugging this turns out to be rather tricky. I just use fewer processors to mesh.

I have tried many times and finally walk around. At first, you should use surfaceCheck utility to make sure your stl file is closed and does not have any unconneceted parts. Even the total stl file has been decomposed into several parts, it should be closed when combining all of them.

This is already implemented in my workflow but did not solve the issue for me unluckily :(

Same here. Additionally, this doesn't explain why meshing on a smaller number of cores is successful. If the surface mesh had problems, all meshing would suffer.

-Chris

Greetings to all!

@Chris:
Sorry for taking so long to look into this. Weird thing is that since it took me so long to look into it, it seems like the bug is already fixed in OpenFOAM 2.3.1 and 2.3.x. I haven't fully confirmed this yet, but my guess is that one or more of the following commits resulted in this issue being fixed:

• https://github.com/OpenFOAM/OpenFOAM...b5eee61b0ad57f
• https://github.com/OpenFOAM/OpenFOAM...6f4055acf95025
• https://github.com/OpenFOAM/OpenFOAM...80c2a6b8a5c208

I tested two of the cases you provided, namely:

• aoa_0.0 - tested with 2.3.0 and 2.3.1, where the first crashed and the second finished generating with success, after around 700 seconds.
• aoa_10.0 - only tested with 2.3.1... and had to break it ahead of time, because it was taking longer than 40 minutes :( ... but it was not stuck in an infinite loop! :)

In addition, I took a look at the first STL file in "aoa_0.0" and ran surfaceCheck on it. This utility pointed out a lot of reasons for concern about the STL geometry, which might be making the mesh generation taking longer than necessary. Attached is an image that shows the location I'm most concerned about, namely that concentrated number of small triangles near the centre of the image, because that level of complexity should only be needed if you were trying to simulate in such exact detail on that location of the surface.


Best regards,
Bruno

Bruno,

While Chris' cases may be fixed, I can confirm that the problem still exists for larger models (>12 million cells). I have seen this many times after the github commits that you noted. I can sometimes get around the problem by changing the decomposition type or simply the number or cores I decompose to.

Unfortunately, I haven't been able to reproduce it on a generic model that I can share here, but I just wanted to note that it is still a problem I see.

Greetings Brock,

Interesting... OK, then we might be able to do some trial and error debugging, by having a set of code changes that can give output relevant to debugging the issue. So please make sure you set a few copies of cases aside, where these bugs are reproducible, since the debugging code can then be tested on your side.

But first, a few questions:

1. Are you 100% certain you rebuilt OpenFOAM 2.3.x with success, after a git pull? Did you run Allwmake? And did you confirm if there weren't any build errors?
2. On Chris' geometry, there were some clear issues that gave me doubts how the latest OpenFOAM 2.3.1 and 2.3.x versions were even able to use it in the first place! Therefore, I ask you Brock to please provide us at least the output from using surfaceCheck with your geometry. Because that information can hint as to why there might be problems.
3. In addition, another detail that was considerably suspicious in Chris' cases was how the base mesh was defined, in comparison to the geometry that was being meshed. In other words, the base mesh had some considerably large cells, where only a few of them would intersect the geometry. I was amazed as to how snappyHexMesh was able to iteratively "enhance" the mesh resolution every iteration and gradually pick up new surfaces that could be intersected.
   The request here would be if you could prepare a dummy case that has the same geometrical, base mesh and similar refinement levels. In other words, swapping out your geometry by a bunch of STL cubes/bricks :D

Best regards,
Bruno

Shell mesh quality seems to be important for snappy
 

Hi there

Thanks for this enlightening thread. We do huge simulations with up to 150M cells. Of course, the meshing has to be done in parallel and with the same number of processors used for the simulation to avoid redistribution of the cells.
From time to time we suffer the same error like you do.

In our case, we get rid of the errors by optimizing the STL files. SnappyHexMesh has problems with very high aspect shell faces. The former erroneous case meshes perfectly after reduction of high aspect and skew shell faces.
Unfortunately there is not tool in OpenFOAM to optimize shell mesh quality or at least I don't know one. We handle all STL files in ANSA.
Hope this helps a little bit.

Cheers

Fabian

After spending a considerable time working on how STLs are produced, I can agree with Fabian. I rarely encounter this if surfaceCheck comes back with no errors. I actually got through ~300 4M cell meshes without an error (also OF2.3.X). I would love to see some tools built into OF that help clean up surfaces meshes, but I also understand that is a very difficult task.

I've also tried cfMesh, which seems to have fewer issues with bad surface geometries. If cfMesh works for your application and you're seeing these errors, it might be worth trying.

Bug report created
 

All,

I've finally been able to reproduce these errors reliably (I've tried on three separate machines) on a somewhat small simple pipe model I can share. I'm hoping you all (and the developers) will be able to reproduce my errors. I created a bug report for it...

http://www.openfoam.org/mantisbt/view.php?id=1792

If anyone with more OF code knowledge would like to take a hack at tracking it down, please go for it! Thanks again for everyone's help!

Same problem with openFOAM2.4.0
 

I am facing the same problem with OF240. I can snappyHexmesh with 4 processors, but cannot do the same with 8. It just runs out of memory! Any fix yet?

shashank,

Per the bug report, we think the bug is in the thirdparty scotch decomposition library. Mattijs noted that he's heard that scotch 6.0.4 resolved some of these errors. However, I have been unsuccessful in getting scotch 6.0.4 to compile properly thus far. And judging by the comment in OpenFOAM-dev on commit 38415f8, it looks like OpenFOAM-dev hasn't been upgraded to 6.0.4 for the same reason.

In any case, the workaround for me at the moment is to change the decomposition type to simple. This is annoying because you then have to specify the xyz decomps, but it's a workaround for now.

Brock

I worked around it using hierarchical decomposition. In my opinion, it has worked better than scotch for me.

Problem believed to be resolved
 

All, as of commit 1da3d4a in 2.4.x and commit e64a892 in dev, I can no longer reproduce this problem on my test models. Shout out to Henry for resolving my bug report and committing the code! Hopefully it's resolved for good.

Hello,

I have just got this error.

Ubuntu 14.04 LTS
Openfoam 2.4.0
x86_64

Base mesh number of cells 187k
decomposition model scotch
number of processors 8

Funny thing is I have been using this stl file for a couple of weeks and I didn't have any problem, later I just made the stl file a bit longer (it is a flat plate with something in the middle of the domain) and change its direction by Blender and received this error. Now I've solved it by using simple method and I thought it may be useful to write here.

Quick answer: Check the values you have for the max local and max global number of cells, namely these entries: https://github.com/OpenFOAM/OpenFOAM...exMeshDict#L81
Expect roughly 1GB of RAM per million cells, which in this example, it would mean that 100MB per subdomain and 2GB of total RAM would be roughly the maximum that snappy was allowed to use.

Still a issue in 2023!
 

Been running into this issue a lot with 10-30M cell parallel cases.


After looking into it I'm relatively sure it has something to do with the angles between triangles of the stl file, includedAngle for surfaceFeatureExtract, and resolveFeatureAngle in castellatedMeshControls for snappyHexMesh.


My guess would be that in some cases these angles create a loop or logic error or something that causes the generation to crash.


Most likely to occur if all three of those angles are close to each other. And seems to work better (crash less) the further they are spaced out, like 5-15deg from each other.