|
[Sponsors] |
[OpenFOAM.org] utilities with 32 bit labels works but 64 does not |
|
LinkBack | Thread Tools | Search this Thread | Display Modes |
October 16, 2018, 17:52 |
utilities with 32 bit labels works but 64 does not
|
#1 |
Senior Member
Join Date: Nov 2010
Location: USA
Posts: 1,232
Rep Power: 24 |
I'm working with some larger meshes and I need 64 bit integers. I have an existing installation with labelsize 32. When I set it to 64 and recompile, everything seems to compile fine.
However, it seems none of the mesh utilities work. For example, if I run mergeMeshes with the 32 bit labels it executes mergeMeshes, but for 64 I just get command not found. Can anyone help me decipher why this would be? |
|
October 20, 2018, 15:16 |
|
#2 |
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,975
Blog Entries: 45
Rep Power: 128 |
Quick questions:
__________________
|
|
October 20, 2018, 22:45 |
|
#3 |
Senior Member
Join Date: Nov 2010
Location: USA
Posts: 1,232
Rep Power: 24 |
1. I just swapped the $WM_LABEL_SIZE in the /etc/bashrc file to be 64.
2. Nothing special, just the ./Allwmake -j 8 3. I will have to do this tomorrow and look at the logs, thanks for your help. |
|
October 24, 2018, 17:19 |
|
#4 | |
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,975
Blog Entries: 45
Rep Power: 128 |
Quote:
Mmmm... I forgot to ask, but: Which exact version/variant/fork of OpenFOAM are you using? |
||
October 31, 2018, 10:01 |
|
#5 |
Senior Member
Join Date: Nov 2010
Location: USA
Posts: 1,232
Rep Power: 24 |
This is vanilla OF 5 with just 1 or two minor changes.
I seem to have gotten to the point now where both builds will run the same applications (at least, recognize they exist) just fine. I'm not quite what difference I made, after going through the install guide several times and reconciling package differences with the cluster I have something is different and it worked? I know that's not helpful. Perhaps there's new light to shed though. Now I have a new problem - the build with 64-bit integers will give a failure reading binary block error when opening a case that the 32-bit build has no problem with. Though after reading the mesh for some time the 32-bit build will crash giving a std::bad_array_new_length error, as the mesh is too large to open in serial, but it can at least read the mesh, apparently the 64 cannot. Am I correct in thinking if I run Allwmake twice, and the second gives no errors, then the build was successful? There are a few errors like: cp: cannot stat '../bin/d[agm]*': no such file or directory in the stderr for scotch. I am just doing a clean build and storing the stderr and stdout separately so I can review them more carefully this time. |
|
October 31, 2018, 22:58 |
|
#6 |
Senior Member
Join Date: Nov 2010
Location: USA
Posts: 1,232
Rep Power: 24 |
Since the 32-bit version can read the binary mesh and the 64 cannot, could it just be because the mesh is written with 32-bit integers that the 64 doesn't know how to read? So in theory if I used the 32 to convert to ascii, then use the 64 to convert to binary, everything would be OK?
|
|
November 1, 2018, 11:26 |
|
#7 | ||||
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,975
Blog Entries: 45
Rep Power: 128 |
Quick answers:
Quote:
Quote:
Quote:
Quote:
However, there are a few limitations:
|
|||||
November 1, 2018, 13:22 |
|
#8 |
Senior Member
Join Date: Nov 2010
Location: USA
Posts: 1,232
Rep Power: 24 |
Thanks Bruno. Yes, foamFormatConvert was what I was using (though it doesn't work on the faces file for some reason, I had to hack it to get it to work, see Trouble using foamFormatConvert)
Minor correction. The mesh was built with 32-bit labels from a 3rd party mesher, and will read in to OpenFOAM with 32-bit labels but only in parallel - in any serial operation (besides decomposition), it will crash, giving the error: std::bad_array_new_length Now in my potentially naive interpretation, this is because some data structure when the grid is loaded (but not the grid itself) becomes too large for a 32-bit integer. When loaded in parallel, this structure is split into multiple pieces among different processes and so it never hits the limit. But in serial, all those pieces are in 1 machine's memory, causing it to crash. So that got me thinking I should try the 64-bit build. I've encountered that sort of problem with other codes on very large meshes. I will shortly try the ASCII conversion, but from your words it seems trying to natively read the binary files between them will never work, and that makes sense to me. I would have liked the later process more, but it is what it is. I suppose I could just make everything I do 64-bit, though I wonder what the performance impact will be. I don't understand what data structure it is that causes the serial session to crash but not the parallel one. In this case I was trying to use mergeMeshes, but simply running checkMesh or anything of that nature will cause a serial process to crash. Interestingly enough there aren't any problems with decomposePar (and I think reconstructPar works too), and I';m guessing there will be no problem with foamFormatConvert... |
|
November 1, 2018, 16:30 |
|
#9 | |||||
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,975
Blog Entries: 45
Rep Power: 128 |
Quote:
Quote:
Quote:
Quote:
Quote:
|
||||||
November 1, 2018, 16:57 |
|
#10 |
Senior Member
Join Date: Nov 2010
Location: USA
Posts: 1,232
Rep Power: 24 |
This is just on OpenFOAM 5.0. I made a few modifications to a source term, but nothing more, so all the utilities (besides foamFormatConvert, as stated before) should be plain vanilla.
From running checkMesh, it appears the numbers are around: points: 200M faces: 600M cells: 200M though I expect to be able to run cases much larger than that. These seem under the 32-bit limit, which combined with the fact that everything is fine when I do something that just reads the grid (decomposePar), makes me suspect the problem only occurs when some utilities (mergeMeshes) make some temporary thing that's way too big. |
|
November 3, 2018, 09:56 |
|
#11 |
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,975
Blog Entries: 45
Rep Power: 128 |
It may be that there is some critical flaw in the mesh exported by the 3rd party mesher, which is only exposed by some applications. For example, an incomplete cell definition or excess number of points or faces...
The other possibility is that since you're merging two meshes, one of them has a critical flaw that does not allow the mergeMeshes utility to properly walk along all faces on each mesh. For example, for whatever reason, the walking algorithm may crash due to an index staying set to -1, because it didn't find the correct correspondence... The 64-bit build may be able to hide any particular flaws, for example, the need for extra memory. For example, accessing to an index -1 may be possible with the 64-bit, just because it had more arrays allocated from a contiguous array in RAM... Either way, I would have to be able to look into the meshes myself and check with a debugger or some old school outputs in the right places to pinpoint the breaking point and the reason for it. |
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
OpenFOAM vs Ubuntu 10.10 64 bit | vkrastev | OpenFOAM Installation | 12 | April 23, 2011 10:14 |
Fluent version 6.3.26 (64 bit) on Window 7 Professional (64 bit) | wlt_1985 | FLUENT | 7 | April 18, 2011 03:22 |
[Technical] Negative labels in faceProcAddressing | ngj | OpenFOAM Meshing & Mesh Conversion | 6 | March 29, 2011 15:54 |
Issue with running in parallel on multiple nodes | daveatstyacht | OpenFOAM | 7 | August 31, 2010 17:16 |
UDF problem/32 bit vs. 64 bit | RE13 | FLUENT | 2 | February 25, 2008 11:31 |