CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Meshing & Mesh Conversion (http://www.cfd-online.com/Forums/openfoam-meshing/)
-   -   bad setSize while building up a too large mesh ? (http://www.cfd-online.com/Forums/openfoam-meshing/110655-bad-setsize-while-building-up-too-large-mesh.html)

Orgogozo December 17, 2012 09:51

bad setSize while building up a too large mesh ?
 
Dear foamers,

I am trying to build up a mesh of about 600 millions of cells (3396*3396*50) and when I run blockMesh I get the following error :


"
Creating cells
Creating points with scale 1

--> FOAM FATAL ERROR:
bad set size -835122496

From function List<T>::setSize(const label)
in file /usr/local/OpenFoam/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/List.C at line 322.

FOAM aborting
"

The procedure I use work for the building of a four time smaller mesh, of about 150 millions of cells (1698*1698*50), but give the error above for the larger case.

Does anyone have an idea of what should be done to solve this problem ? Thanks by advance for any help.

Laurent

jans April 12, 2013 14:26

Hi Laurent,
I just encountered the same error. Were you able to overcome this problem?

My guess is that this is a case of overflow in a variable of type int. For example, for large meshes, the variables holding the number of points, faces, cells etc are susceptible to overflow. The solution may be to switch to "long int" when this happens.

@ OpenFOAM developers: Should this be submitted as a bug report?

Thanks
Anirban

Abracurcix June 15, 2013 09:59

Hello Laurent,
This questions of yours appears to have not gotten a response from any of the other fellow FOAMers. Did you manage to solve this problem? The problem seems to be a bit more involved than merely promoting "int" to "long int" in the label.H header file in primitives/ints/label (by defining FOAM_LABEL64 or setting FOAM_LABEL_MAX to a value greater than 2,000,000,000).

Cheers,
Albert

Quote:

Originally Posted by Orgogozo (Post 397913)
Dear foamers,

I am trying to build up a mesh of about 600 millions of cells (3396*3396*50) and when I run blockMesh I get the following error :


"
Creating cells
Creating points with scale 1

--> FOAM FATAL ERROR:
bad set size -835122496

From function List<T>::setSize(const label)
in file /usr/local/OpenFoam/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/List.C at line 322.

FOAM aborting
"

The procedure I use work for the building of a four time smaller mesh, of about 150 millions of cells (1698*1698*50), but give the error above for the larger case.

Does anyone have an idea of what should be done to solve this problem ? Thanks by advance for any help.

Laurent


Orgogozo June 16, 2013 08:27

Dear Jans, Dear Abracurcix,

I didn't manage to slve this problem yet ; instead I dealt with smaller meshes. However I will try again soon so thank you for your threads I'll look cqrefully about this issue of long int. I'll post a threads as soon as (hopefully) i succed in working with lqrger meshes.

Best regards,

Laurent

wyldckat June 16, 2013 08:45

Greetings to all!

If my memory doesn't fail me, the estimate is that 3396*3396*50 cells will require around 575 GB of RAM! A machine to test this is rather difficult to come by for testing such a big case. Let alone the time it takes to run in a single core for generating it...

Nonetheless, if your machines do have that amount of RAM or more, I do strongly suggest that you report this on the bug tracker: http://www.openfoam.org/bugs/

Best regards,
Bruno

Orgogozo June 16, 2013 09:15

Dear wyldckat,

I do use a fat node with 2TB RAM. But i encountered this bug with an old version of OF (2.0.1 I think), so I'll try again with 2.2 in the following days and then do a bug reports if I still get the same error.

Best regards,

Laurent

wyldckat June 23, 2013 17:55

Greetings to all!

I've given this a try and I found out that there are several pieces of source code in OpenFOAM that are not prepared for "FOAM_LABEL64" :(

The steps I've taken were:
  1. Edit the file "wmake/rules/linux64Gcc/c++" and added:
    Code:

    -DFOAM_LABEL64
    to the end of the line "ptFLAGS".
  2. Run:
    Code:

    foam
    wcleanAll

  3. Then run (for using 6 cores):
    Code:

    export WM_NCOMPPROCS=6
    ./Allwmake > make.log 2>&1

And the library "surfMesh" was only the tip of the iceberg :( Then came "meshTools", "scotchDecomp", "ptscotchDecomp" and "autoMesh", if I'm not mistaken.

So, once again, since you guys have the machines with tons of RAM and are interested in the solution, please report this on the bug tracker: http://www.openfoam.org/bugs/

Best regards,
Bruno

Orgogozo June 27, 2013 13:13

Dear all,

The bug is reported (ID 0000903 on the mantis).

Thank you all for your answers

Cheers

Laurent

wyldckat November 3, 2013 06:52

Greetings to all!

I haven't had the time and opportunity to look into this myself, but in the following bug report there are some instructions that might help: http://www.openfoam.org/mantisbt/view.php?id=1033

By the way, it's possible that OpenCFD won't fix the problem unless someone sponsors the fix and subsequent testing.

Best regards,
Bruno

mchurchf September 25, 2014 00:07

Laurent,

Were you ever able to build very large meshes, or successfully compile with -DFOAM_LABEL64?

Orgogozo October 14, 2014 06:59

Matthew,

I checked again with the version 2.0.x of OpenFOAM : it still does not work. With parallel meshing however one can build up relatively large meshes, with 100 millions number of mesh cells or more. See for example the following paper :
http://authors.elsevier.com/a/1PsPx2OInFfaV

Best regards,

Laurent

andrea.pasquali October 29, 2014 10:23

Dear all,

I found the same problem with the integer size in OpenFOAM.
I recompiled OpenFOAM with the extension "-DFOAM_LABEL64" checking and correcting the pieces of code where the compilation gives me error.
Now it works but I noticed that OpenFOAM in 64 requires 30% memory more than in 32 (running the same application with the same grid size).
Should I check something else during the compilation?
Or are there any other suggestions?

Best regards
Andrea

wyldckat November 1, 2014 10:44

Greetings Andrea,

Quote:

Originally Posted by andrea.pasquali (Post 516491)
Now it works but I noticed that OpenFOAM in 64 requires 30% memory more than in 32 (running the same application with the same grid size).

I believe you've tripped over the reason why OpenFOAM isn't built by default with "FOAM_LABEL64" turned on and why it's not being very well maintained.
The problem of using 64-bit "labels" is that this means that in fact it uses 64-bit integers, instead of 32-bit integers. This means that all integers used in the code will use up 8 bytes per value, instead of 4 bytes. Therefore, consider yourself lucky it's only consuming more 30% RAM and not 50% ;)

One solution for this would require diagnosing all parts of the code and to isolate the situations where "2^31 = 2147483648" items (both negative and positive) is more than enough, i.e. where more than 2 billion cells/faces/points/hash lists are necessary. Also keep in mind that this can scale up to needing matrices for the equations with roughly as many entries over at least 2 dimensions...
Keep in mind that OpenFOAM relies heavily on indexing points, cells and faces, hence using all labels at the same bit-length.

Another solution would be to rely on hybrid mathematical libraries, such as GMP: https://gmplib.org/ - in theory, it could be able to handle 48-bit mathematics, which would equate to "2^47 = 1.40737488410⁴" items. problem is that storage and access to 48-bit data isn't very "standard", which means that it would possibly require some strong efforts in having an intermediate interpretation layer between OpenFOAM's way of doing things and GMP's way of doing things. There was some discussion about this sometime ago... let me check... here you go: http://www.cfd-online.com/Forums/ope...precision.html - although that was more focused on the floating precision and not integer precision...

The bottom line is: if you need to go into the 64-bit realm, be sure to have enough RAM to pay the price ;)
The other solution would be to handle your mesh in parallel, so that each partition doesn't go over the 32-bit limit.

Best regards,
Bruno

wyldckat January 1, 2015 05:45

Greetings to all and a Happy New Year!

For those who aren't keeping track of the bug report, it has been fixed yesterday!
This change will unlikely make it's way to OpenFOAM 2.3, since this is a pretty major new feature... and fix... completion of a feature :)

Using this repository should roughly be used similarly to the git repository 2.3.x.

Best regards,
Bruno

Orgogozo March 15, 2015 12:59

bug solved ; test on CALMIP ok
 
Dear all,

I have just tested the building of a mesh of ~600 millions cells with the Dev version of OpenFOAM on the fat node of CALMIP* (UVPROD, 2TB RAM) and it works fine. I will make a more detail return on this issue later on this year.

Greetings,

*http://www.calmip.univ-toulouse.fr

Orgogozo May 21, 2015 05:53

Scaling curve on a billion cells mesh
 
Dear all,

With the development version of OpenFoam*, we have managed to make a scaling curve from 400 to 3200 cores on a case with a 1.2 billion cells mesh. We experienced super-linearity until 3200 cores.

The runs have been done on EOS**, the new cluster of CALMIP. The building and the decomposition of the mesh were performed on an UVPROD cluster, with ~3TB RAM avalaible (the requests were using maximum 1.5 TB requests, i.e. half of the machine).

This result has been presented in the beginning of may 2015 at the Maison de la Simulation in Saclay, France. You may find at the url below the pdf of this seminar (billion mesh cells scaling results : slides 20-22).

http://www.maisondelasimulation.fr/en/semmod.php?a
(scroll to may 2015 to find the presentation)

Cheers,

Laurent


*https://github.com/OpenFOAM/OpenFOAM-dev

**http://www.calmip.univ-toulouse.fr/s...php?article388


All times are GMT -4. The time now is 08:43.