CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Meshing & Mesh Conversion

bad setSize while building up a too large mesh ?

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree4Likes
  • 1 Post By Orgogozo
  • 1 Post By Orgogozo
  • 1 Post By Orgogozo
  • 1 Post By Orgogozo

Reply
 
LinkBack Thread Tools Display Modes
Old   December 17, 2012, 09:51
Default bad setSize while building up a too large mesh ?
  #1
New Member
 
Laurent Orgogozo
Join Date: Mar 2011
Location: Toulouse
Posts: 18
Rep Power: 6
Orgogozo is on a distinguished road
Dear foamers,

I am trying to build up a mesh of about 600 millions of cells (3396*3396*50) and when I run blockMesh I get the following error :


"
Creating cells
Creating points with scale 1

--> FOAM FATAL ERROR:
bad set size -835122496

From function List<T>::setSize(const label)
in file /usr/local/OpenFoam/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/List.C at line 322.

FOAM aborting
"

The procedure I use work for the building of a four time smaller mesh, of about 150 millions of cells (1698*1698*50), but give the error above for the larger case.

Does anyone have an idea of what should be done to solve this problem ? Thanks by advance for any help.

Laurent
Orgogozo is offline   Reply With Quote

Old   April 12, 2013, 14:26
Default
  #2
New Member
 
Anirban Jana
Join Date: Apr 2010
Location: Pittsburgh, PA, USA
Posts: 19
Rep Power: 7
jans is on a distinguished road
Hi Laurent,
I just encountered the same error. Were you able to overcome this problem?

My guess is that this is a case of overflow in a variable of type int. For example, for large meshes, the variables holding the number of points, faces, cells etc are susceptible to overflow. The solution may be to switch to "long int" when this happens.

@ OpenFOAM developers: Should this be submitted as a bug report?

Thanks
Anirban
jans is offline   Reply With Quote

Old   June 15, 2013, 09:59
Default
  #3
New Member
 
Albert Pinto
Join Date: May 2013
Posts: 16
Rep Power: 4
Abracurcix is on a distinguished road
Hello Laurent,
This questions of yours appears to have not gotten a response from any of the other fellow FOAMers. Did you manage to solve this problem? The problem seems to be a bit more involved than merely promoting "int" to "long int" in the label.H header file in primitives/ints/label (by defining FOAM_LABEL64 or setting FOAM_LABEL_MAX to a value greater than 2,000,000,000).

Cheers,
Albert

Quote:
Originally Posted by Orgogozo View Post
Dear foamers,

I am trying to build up a mesh of about 600 millions of cells (3396*3396*50) and when I run blockMesh I get the following error :


"
Creating cells
Creating points with scale 1

--> FOAM FATAL ERROR:
bad set size -835122496

From function List<T>::setSize(const label)
in file /usr/local/OpenFoam/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/List.C at line 322.

FOAM aborting
"

The procedure I use work for the building of a four time smaller mesh, of about 150 millions of cells (1698*1698*50), but give the error above for the larger case.

Does anyone have an idea of what should be done to solve this problem ? Thanks by advance for any help.

Laurent
Abracurcix is offline   Reply With Quote

Old   June 16, 2013, 08:27
Default
  #4
New Member
 
Laurent Orgogozo
Join Date: Mar 2011
Location: Toulouse
Posts: 18
Rep Power: 6
Orgogozo is on a distinguished road
Dear Jans, Dear Abracurcix,

I didn't manage to slve this problem yet ; instead I dealt with smaller meshes. However I will try again soon so thank you for your threads I'll look cqrefully about this issue of long int. I'll post a threads as soon as (hopefully) i succed in working with lqrger meshes.

Best regards,

Laurent
Orgogozo is offline   Reply With Quote

Old   June 16, 2013, 08:45
Default
  #5
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,253
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all!

If my memory doesn't fail me, the estimate is that 3396*3396*50 cells will require around 575 GB of RAM! A machine to test this is rather difficult to come by for testing such a big case. Let alone the time it takes to run in a single core for generating it...

Nonetheless, if your machines do have that amount of RAM or more, I do strongly suggest that you report this on the bug tracker: http://www.openfoam.org/bugs/

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   June 16, 2013, 09:15
Default
  #6
New Member
 
Laurent Orgogozo
Join Date: Mar 2011
Location: Toulouse
Posts: 18
Rep Power: 6
Orgogozo is on a distinguished road
Dear wyldckat,

I do use a fat node with 2TB RAM. But i encountered this bug with an old version of OF (2.0.1 I think), so I'll try again with 2.2 in the following days and then do a bug reports if I still get the same error.

Best regards,

Laurent
wyldckat likes this.
Orgogozo is offline   Reply With Quote

Old   June 23, 2013, 17:55
Default
  #7
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,253
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all!

I've given this a try and I found out that there are several pieces of source code in OpenFOAM that are not prepared for "FOAM_LABEL64"

The steps I've taken were:
  1. Edit the file "wmake/rules/linux64Gcc/c++" and added:
    Code:
    -DFOAM_LABEL64
    to the end of the line "ptFLAGS".
  2. Run:
    Code:
    foam
    wcleanAll
  3. Then run (for using 6 cores):
    Code:
    export WM_NCOMPPROCS=6
    ./Allwmake > make.log 2>&1
And the library "surfMesh" was only the tip of the iceberg Then came "meshTools", "scotchDecomp", "ptscotchDecomp" and "autoMesh", if I'm not mistaken.

So, once again, since you guys have the machines with tons of RAM and are interested in the solution, please report this on the bug tracker: http://www.openfoam.org/bugs/

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   June 27, 2013, 13:13
Default
  #8
New Member
 
Laurent Orgogozo
Join Date: Mar 2011
Location: Toulouse
Posts: 18
Rep Power: 6
Orgogozo is on a distinguished road
Dear all,

The bug is reported (ID 0000903 on the mantis).

Thank you all for your answers

Cheers

Laurent
wyldckat likes this.
Orgogozo is offline   Reply With Quote

Old   November 3, 2013, 06:52
Default
  #9
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,253
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all!

I haven't had the time and opportunity to look into this myself, but in the following bug report there are some instructions that might help: http://www.openfoam.org/mantisbt/view.php?id=1033

By the way, it's possible that OpenCFD won't fix the problem unless someone sponsors the fix and subsequent testing.

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   September 25, 2014, 00:07
Default
  #10
Member
 
Matthew J. Churchfield
Join Date: Nov 2009
Location: Boulder, Colorado, USA
Posts: 49
Rep Power: 8
mchurchf is on a distinguished road
Laurent,

Were you ever able to build very large meshes, or successfully compile with -DFOAM_LABEL64?
mchurchf is offline   Reply With Quote

Old   October 14, 2014, 06:59
Default
  #11
New Member
 
Laurent Orgogozo
Join Date: Mar 2011
Location: Toulouse
Posts: 18
Rep Power: 6
Orgogozo is on a distinguished road
Matthew,

I checked again with the version 2.0.x of OpenFOAM : it still does not work. With parallel meshing however one can build up relatively large meshes, with 100 millions number of mesh cells or more. See for example the following paper :
http://authors.elsevier.com/a/1PsPx2OInFfaV

Best regards,

Laurent
Orgogozo is offline   Reply With Quote

Old   October 29, 2014, 10:23
Default
  #12
Senior Member
 
Andrea Pasquali
Join Date: Sep 2009
Location: Germany
Posts: 142
Rep Power: 7
andrea.pasquali is on a distinguished road
Dear all,

I found the same problem with the integer size in OpenFOAM.
I recompiled OpenFOAM with the extension "-DFOAM_LABEL64" checking and correcting the pieces of code where the compilation gives me error.
Now it works but I noticed that OpenFOAM in 64 requires 30% memory more than in 32 (running the same application with the same grid size).
Should I check something else during the compilation?
Or are there any other suggestions?

Best regards
Andrea
__________________
Andrea Pasquali
andrea.pasquali is offline   Reply With Quote

Old   November 1, 2014, 10:44
Default
  #13
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,253
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Andrea,

Quote:
Originally Posted by andrea.pasquali View Post
Now it works but I noticed that OpenFOAM in 64 requires 30% memory more than in 32 (running the same application with the same grid size).
I believe you've tripped over the reason why OpenFOAM isn't built by default with "FOAM_LABEL64" turned on and why it's not being very well maintained.
The problem of using 64-bit "labels" is that this means that in fact it uses 64-bit integers, instead of 32-bit integers. This means that all integers used in the code will use up 8 bytes per value, instead of 4 bytes. Therefore, consider yourself lucky it's only consuming more 30% RAM and not 50%

One solution for this would require diagnosing all parts of the code and to isolate the situations where "2^31 = 2147483648" items (both negative and positive) is more than enough, i.e. where more than 2 billion cells/faces/points/hash lists are necessary. Also keep in mind that this can scale up to needing matrices for the equations with roughly as many entries over at least 2 dimensions...
Keep in mind that OpenFOAM relies heavily on indexing points, cells and faces, hence using all labels at the same bit-length.

Another solution would be to rely on hybrid mathematical libraries, such as GMP: https://gmplib.org/ - in theory, it could be able to handle 48-bit mathematics, which would equate to "2^47 = 1.40737488410⁴" items. problem is that storage and access to 48-bit data isn't very "standard", which means that it would possibly require some strong efforts in having an intermediate interpretation layer between OpenFOAM's way of doing things and GMP's way of doing things. There was some discussion about this sometime ago... let me check... here you go: Mixing single and double precision - although that was more focused on the floating precision and not integer precision...

The bottom line is: if you need to go into the 64-bit realm, be sure to have enough RAM to pay the price
The other solution would be to handle your mesh in parallel, so that each partition doesn't go over the 32-bit limit.

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   January 1, 2015, 05:45
Default
  #14
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,253
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all and a Happy New Year!

For those who aren't keeping track of the bug report, it has been fixed yesterday!
This change will unlikely make it's way to OpenFOAM 2.3, since this is a pretty major new feature... and fix... completion of a feature

Using this repository should roughly be used similarly to the git repository 2.3.x.

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   March 15, 2015, 12:59
Default bug solved ; test on CALMIP ok
  #15
New Member
 
Laurent Orgogozo
Join Date: Mar 2011
Location: Toulouse
Posts: 18
Rep Power: 6
Orgogozo is on a distinguished road
Dear all,

I have just tested the building of a mesh of ~600 millions cells with the Dev version of OpenFOAM on the fat node of CALMIP* (UVPROD, 2TB RAM) and it works fine. I will make a more detail return on this issue later on this year.

Greetings,

*http://www.calmip.univ-toulouse.fr
wyldckat likes this.
Orgogozo is offline   Reply With Quote

Old   May 21, 2015, 05:53
Default Scaling curve on a billion cells mesh
  #16
New Member
 
Laurent Orgogozo
Join Date: Mar 2011
Location: Toulouse
Posts: 18
Rep Power: 6
Orgogozo is on a distinguished road
Dear all,

With the development version of OpenFoam*, we have managed to make a scaling curve from 400 to 3200 cores on a case with a 1.2 billion cells mesh. We experienced super-linearity until 3200 cores.

The runs have been done on EOS**, the new cluster of CALMIP. The building and the decomposition of the mesh were performed on an UVPROD cluster, with ~3TB RAM avalaible (the requests were using maximum 1.5 TB requests, i.e. half of the machine).

This result has been presented in the beginning of may 2015 at the Maison de la Simulation in Saclay, France. You may find at the url below the pdf of this seminar (billion mesh cells scaling results : slides 20-22).

http://www.maisondelasimulation.fr/en/semmod.php?a
(scroll to may 2015 to find the presentation)

Cheers,

Laurent


*https://github.com/OpenFOAM/OpenFOAM-dev

**http://www.calmip.univ-toulouse.fr/s...php?article388
wyldckat likes this.
Orgogozo is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
SnappyHexMesh for internal Flow vishwa OpenFOAM Native Meshers: snappyHexMesh and Others 23 August 6, 2014 03:50
Error compiling OpenFOAM-1.6-ext Canesin OpenFOAM Installation 136 April 25, 2012 11:22
[ICEM] Problem making structured mesh on a surface froztbear ANSYS Meshing & Geometry 4 November 10, 2011 09:52
LES on a scaled geomentry (Mesh and Large scales) comb Main CFD Forum 0 February 8, 2011 07:21
Compilation error OF1.5-dev on Suse10.3 darenyang OpenFOAM Installation 0 April 29, 2009 04:55


All times are GMT -4. The time now is 05:15.