[SOLVED] SigFpe when running ANY application in parallel
Hi everybody.
I have a very simple case made of a box-shaped volume created as a single block in blockMesh. The mesh is made of cubic cells, without any non-ortogonality. Everything runs fine until I try to run the solver or any other application using mpirun. Then the polymesh loading fails and create a sigFpe error.
As a simple demonstration of the error i run checkMesh (single core), decomposePar, checkmesh (decomposed).
When i run checkMesh this is the output:
Code:
/*---------------------------------------------------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 2.2.1 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
Build : 2.2.1-57f3c3617a2d
Exec : checkMesh
Date : Apr 23 2015
Time : 10:06:40
Host : "node166"
PID : 30400
Case : /gpfs/scratch/userexternal/lamerio0/Rete/M3
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Disallowing user-supplied system call operations
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time
Create polyMesh for time = 0
Time = 0
Mesh stats
points: 6031169
faces: 17826816
internal faces: 17562624
cells: 5898240
faces per cell: 6
boundary patches: 7
point zones: 0
face zones: 0
cell zones: 0
Overall number of cells of each type:
hexahedra: 5898240
prisms: 0
wedges: 0
pyramids: 0
tet wedges: 0
tetrahedra: 0
polyhedra: 0
Checking topology...
Boundary definition OK.
Cell to face addressing OK.
Point usage OK.
Upper triangular ordering OK.
Face vertices OK.
Number of regions: 1 (OK).
Checking patch topology for multiply connected surfaces...
Patch Faces Points Surface topology
cyclic_bottom 61440 62177 ok (non-closed singly connected)
cyclic_top 61440 62177 ok (non-closed singly connected)
cyclic_left 61440 62177 ok (non-closed singly connected)
cyclic_right 61440 62177 ok (non-closed singly connected)
in 5040 5328 ok (non-closed singly connected)
out 9216 9409 ok (non-closed singly connected)
net 4176 4649 ok (non-closed singly connected)
Checking geometry...
Overall domain bounding box (0 -0.04 -0.04) (0.8 0.04 0.04)
Mesh (non-empty, non-wedge) directions (1 1 1)
Mesh (non-empty) directions (1 1 1)
Boundary openness (1.7218198e-16 3.407543e-16 -4.6831815e-17) OK.
Max cell openness = 3.5101045e-16 OK.
Max aspect ratio = 1.5 OK.
Minimum face area = 6.9444444e-07. Maximum face area = 1.0416667e-06. Face area magnitudes OK.
Min volume = 8.6805556e-10. Max volume = 8.6805556e-10. Total volume = 0.00512. Cell volumes OK.
Mesh non-orthogonality Max: 0 average: 0
Non-orthogonality check OK.
Face pyramids OK.
Max skewness = 2.8799967e-06 OK.
Coupled point location match (average 1.232301e-17) OK.
Mesh OK.
End
Sounds great!
Now i run decomposePar with this decomposeParDict:
Code:
FoamFile
{
version 2.0;
format ascii;
class dictionary;
object decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
numberOfSubdomains 10;
method scotch;
distributed false;
roots
(
);
It returns:
Code:
/*---------------------------------------------------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 2.2.1 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
Build : 2.2.1-57f3c3617a2d
Exec : decomposePar
Date : Apr 23 2015
Time : 10:00:12
Host : "node166"
PID : 23719
Case : /gpfs/scratch/userexternal/lamerio0/Rete/M3
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Disallowing user-supplied system call operations
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time
Decomposing mesh region0
Create mesh
Calculating distribution of cells
Selecting decompositionMethod scotch
Finished decomposition in 17.57 s
Calculating original mesh data
Distributing cells to processors
Distributing faces to processors
Distributing points to processors
Constructing processor meshes
Processor 0
Number of cells = 591664
Number of faces shared with processor 1 = 12561
Number of faces shared with processor 1 = 15
Number of faces shared with processor 1 = 24
Number of faces shared with processor 3 = 12095
Number of faces shared with processor 3 = 13
Number of faces shared with processor 3 = 4
Number of faces shared with processor 3 = 8
Number of processor patches = 7
Number of processor faces = 24720
Number of boundary faces = 24004
Processor 1
Number of cells = 589712
Number of faces shared with processor 0 = 12561
Number of faces shared with processor 0 = 15
Number of faces shared with processor 0 = 24
Number of faces shared with processor 9 = 12199
Number of faces shared with processor 9 = 2
Number of faces shared with processor 9 = 2
Number of faces shared with processor 9 = 10
Number of faces shared with processor 9 = 3
Number of processor patches = 8
Number of processor faces = 24816
Number of boundary faces = 25086
Processor 2
Number of cells = 590427
Number of faces shared with processor 4 = 11695
Number of faces shared with processor 4 = 4
Number of faces shared with processor 4 = 6
Number of faces shared with processor 4 = 5
Number of processor patches = 4
Number of processor faces = 11710
Number of boundary faces = 33342
Processor 3
Number of cells = 591917
Number of faces shared with processor 0 = 12095
Number of faces shared with processor 0 = 4
Number of faces shared with processor 0 = 13
Number of faces shared with processor 0 = 8
Number of faces shared with processor 4 = 11647
Number of faces shared with processor 4 = 6
Number of faces shared with processor 4 = 2
Number of faces shared with processor 4 = 1
Number of faces shared with processor 4 = 14
Number of processor patches = 9
Number of processor faces = 23790
Number of boundary faces = 24428
Processor 4
Number of cells = 589664
Number of faces shared with processor 2 = 11695
Number of faces shared with processor 2 = 4
Number of faces shared with processor 2 = 5
Number of faces shared with processor 2 = 6
Number of faces shared with processor 3 = 11647
Number of faces shared with processor 3 = 2
Number of faces shared with processor 3 = 6
Number of faces shared with processor 3 = 14
Number of faces shared with processor 3 = 1
Number of processor patches = 9
Number of processor faces = 23380
Number of boundary faces = 24842
Processor 5
Number of cells = 590639
Number of faces shared with processor 6 = 12086
Number of faces shared with processor 6 = 1
Number of faces shared with processor 6 = 7
Number of faces shared with processor 6 = 1
Number of faces shared with processor 6 = 13
Number of processor patches = 5
Number of processor faces = 12108
Number of boundary faces = 33578
Processor 6
Number of cells = 589995
Number of faces shared with processor 5 = 12086
Number of faces shared with processor 5 = 7
Number of faces shared with processor 5 = 1
Number of faces shared with processor 5 = 13
Number of faces shared with processor 5 = 1
Number of faces shared with processor 7 = 11834
Number of faces shared with processor 7 = 10
Number of faces shared with processor 7 = 1
Number of faces shared with processor 7 = 19
Number of processor patches = 9
Number of processor faces = 23972
Number of boundary faces = 24414
Processor 7
Number of cells = 589221
Number of faces shared with processor 6 = 11834
Number of faces shared with processor 6 = 1
Number of faces shared with processor 6 = 10
Number of faces shared with processor 6 = 19
Number of faces shared with processor 8 = 11479
Number of faces shared with processor 8 = 12
Number of faces shared with processor 8 = 5
Number of faces shared with processor 8 = 8
Number of processor patches = 8
Number of processor faces = 23368
Number of boundary faces = 25196
Processor 8
Number of cells = 588220
Number of faces shared with processor 7 = 11479
Number of faces shared with processor 7 = 5
Number of faces shared with processor 7 = 12
Number of faces shared with processor 7 = 8
Number of faces shared with processor 9 = 11772
Number of faces shared with processor 9 = 20
Number of faces shared with processor 9 = 10
Number of faces shared with processor 9 = 2
Number of processor patches = 8
Number of processor faces = 23308
Number of boundary faces = 23794
Processor 9
Number of cells = 586781
Number of faces shared with processor 1 = 12199
Number of faces shared with processor 1 = 2
Number of faces shared with processor 1 = 2
Number of faces shared with processor 1 = 3
Number of faces shared with processor 1 = 10
Number of faces shared with processor 8 = 11772
Number of faces shared with processor 8 = 20
Number of faces shared with processor 8 = 2
Number of faces shared with processor 8 = 10
Number of processor patches = 9
Number of processor faces = 24020
Number of boundary faces = 25052
Number of processor faces = 107596
Max number of cells = 591917 (0.35485162% above average 589824)
Max number of processor patches = 9 (18.421053% above average 7.6)
Max number of faces between processors = 24816 (15.320272% above average 21519.2)
Time = 0
Processor 0: field transfer
Processor 1: field transfer
Processor 2: field transfer
Processor 3: field transfer
Processor 4: field transfer
Processor 5: field transfer
Processor 6: field transfer
Processor 7: field transfer
Processor 8: field transfer
Processor 9: field transfer
End.
But, if I run any application using mpirun on the decomposed case I receive a sigFpe error after the "Create polyMesh for time = 0" phase.
E.g. if I run " mpirun -np 10 checkMesh -parallel" I receive:
[error moved in the next post due to characters limit]
I tried to change OF version from 2.3.0 to 2.2.1, I also changed cluster, nothing worked.
I al so changed the number of processors from 10 to 9 and to 11, but neither this worked.
How can I solve the problem?
|