# parallel calculation issues

 Register Blogs Members List Search Today's Posts Mark Forums Read

 December 20, 2018, 08:51 parallel calculation issues #1 New Member   Join Date: Oct 2018 Posts: 9 Rep Power: 7 Hello everyone, I'm very new to linux and OF, and currently I'm going with a very frustrating problem. I've successfully set a case and run it to 5000 timesteps in serial without any mistake. So I'm now trying to run it in parallel. But it aborted at first timestep with the famous "float point exception". The log is as follows: Code: ```Time = 1 BiCGStab: Solving for Ux, Initial residual = 1, Final residual = 0.00262781, No Iterations 1 BiCGStab: Solving for Uy, Initial residual = 1, Final residual = 0.000321715, No Iterations 1 BiCGStab: Solving for Uz, Initial residual = 1, Final residual = 0.00339929, No Iterations 2 BiCGStab: Solving for p, Initial residual = 1, Final residual = 0.0440403, No Iterations 1 time step continuity errors : sum local = 0.0192303, global = -0.007747, cumulative = -0.007747 BiCGStab: Solving for p, Initial residual = 0.861166, Final residual = 0.0152853, No Iterations 2 time step continuity errors : sum local = 0.068281, global = -0.0116883, cumulative = -0.0194353 BiCGStab: Solving for i, Initial residual = 1, Final residual = 0.000584795, No Iterations 2 ExecutionTime = 3.99 s ClockTime = 4 s Cyclic GGI pair (perBlade1, perBlade2) Area: 0.00523772 0.00523772 Diff = -5.96635e-13 or 1.13911e-08 % Flux: 155599 581.131 Diff = -155996 or 100.255 % Cyclic GGI pair (perUp1, perUp2) Area: 0.00967552 0.00967552 Diff = 3.23594e-13 or 3.34446e-09 % Flux: 46.6035 0.00235069 Diff = -46.6009 or 99.9944 % Cyclic GGI pair (perDown1, perDown2) Area: 0.00364872 0.00364872 Diff = -2.67976e-13 or 7.34439e-09 % Flux: 48.5907 124200 Diff = -123929 or 255047 % Flux divergence min = 1.29895e-07 max = 8.40826e+06 average: 4.40663e+08 [admin:24845] *** Process received signal *** [admin:24847] *** Process received signal *** [admin:24847] Signal: Floating point exception (8) [admin:24847] Signal code: (-6) [admin:24847] Failing at address: 0x4240000610f [admin:24849] *** Process received signal *** [admin:24849] Signal: Floating point exception (8) [admin:24849] Signal code: (-6) [admin:24849] Failing at address: 0x42400006111 [admin:24845] Signal: Floating point exception (8) [admin:24845] Signal code: (-6) [admin:24845] Failing at address: 0x4240000610d [admin:24849] [ 0] [admin:24847] /lib64/libc.so.6(+0x35670)[0x2acb53b6a670] [admin:24849] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2acb53b6a5f7] [admin:24849] [ 2] /lib64/libc.so.6(+0x35670)[0x2acb53b6a670] [admin:24849] [ 3] [admin:24850] *** Process received signal *** [ 0] /lib64/libc.so.6(+0x35670)[0x2b0a72aea670] [admin:24847] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b0a72aea5f7] [admin:24847] [ 2] [admin:24850] Signal: Floating point exception (8) [admin:24850] Signal code: (-6) [admin:24850] Failing at address: 0x42400006112 [admin:24846] *** Process received signal *** [admin:24845] [ 0] /lib64/libc.so.6(+0x35670)[0x2b0a72aea670] [admin:24847] [ 3] [admin:24846] Signal: Floating point exception (8) [admin:24846] Signal code: (-6) [admin:24846] Failing at address: 0x4240000610e /lib64/libc.so.6(+0x35670)[0x2ac434a32670] [admin:24845] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2ac434a325f7] [admin:24845] [ 2] /lib64/libc.so.6(+0x35670)[0x2ac434a32670] [admin:24845] [ 3] [admin:24846] [ 0] [admin:24851] *** Process received signal *** /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2acb527d4d48] [admin:24849] [ 4] /lib64/libc.so.6(+0x35670)[0x2b9a99132670] [admin:24846] [ 1] [admin:24850] [ 0] /lib64/libc.so.6(gsignal+0x37)[0x2b9a991325f7] [admin:24846] [ 2] /lib64/libc.so.6(+0x35670)[0x2ad07bffc670] [admin:24850] [ 1] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2b0a71754d48] [admin:24847] [ 4] /lib64/libc.so.6(+0x35670)[0x2b9a99132670] [admin:24846] [ 3] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2acb51c73a98] [admin:24849] [ 5] /lib64/libc.so.6(gsignal+0x37)[0x2ad07bffc5f7] [admin:24850] [ 2] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2acb6fb4ca93] [admin:24849] [ 6] /lib64/libc.so.6(+0x35670)[0x2ad07bffc670] [admin:24850] [ 3] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2b0a70bf3a98] [admin:24847] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2b0a8ea70a93] [admin:24847] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2ac43369cd48] [admin:24845] [ 4] [admin:24851] Signal: Floating point exception (8) [admin:24851] Signal code: (-6) [admin:24851] Failing at address: 0x42400006113 [admin:24851] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2acb52640f89] [admin:24849] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2b0a715c0f89] [admin:24847] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2b9a97d9cd48] [admin:24846] [ 4] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2ad07ac66d48] [admin:24850] [ 4] [ 0] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2b9a9723ba98] [admin:24846] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2b9ab5030a93] [admin:24846] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2ad07a105a98] /lib64/libc.so.6(+0x35670)[0x2b0d70a5d670] [admin:24851] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b0d70a5d5f7] [admin:24851] [ 2] /lib64/libc.so.6(+0x35670)[0x2b0d70a5d670] [admin:24851] [ 3] [admin:24850] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2b0a715cdbda] [admin:24847] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2acb5264dbda] [admin:24849] [ 8] [admin:24848] *** Process received signal *** [admin:24848] Signal: Floating point exception (8) [admin:24848] Signal code: (-6) [admin:24848] Failing at address: 0x42400006110 /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2b9a97c08f89] [admin:24846] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2ad097f22a93] [admin:24850] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2b0d6f6c7d48] [admin:24848] [ 0] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2b0a715c5c64] [admin:24847] [ 9] /lib64/libc.so.6(+0x35670)[0x2b221dbe8670] [admin:24848] [ 1] steadyCompressibleMRFFoam[0x421670] [admin:24847] [10] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2acb52645c64] [admin:24849] [ 9] /lib64/libc.so.6(gsignal+0x37)[0x2b221dbe85f7] [admin:24848] [ 2] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b0a72ad6b15] [admin:24847] [11] steadyCompressibleMRFFoam[0x4241f9] [admin:24847] *** End of error message *** /lib64/libc.so.6(+0x35670)[0x2b221dbe8670] [admin:24848] [ 3] steadyCompressibleMRFFoam[0x421670] [admin:24849] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2acb53b56b15] [admin:24849] [11] steadyCompressibleMRFFoam[0x4241f9] [admin:24849] *** End of error message *** [admin:24851] [ 4] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2b0d6eb66a98] [admin:24851] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2b0d8c9aca93] [admin:24851] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2ad07aad2f89] [admin:24850] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2b9a97c15bda] [admin:24846] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2b221c852d48] [admin:24848] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2b0d6f533f89] [admin:24851] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2ad07aadfbda] [admin:24850] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2b9a97c0dc64] [admin:24846] [ 9] steadyCompressibleMRFFoam[0x421670] [admin:24846] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b9a9911eb15] [admin:24846] [11] steadyCompressibleMRFFoam[0x4241f9] [admin:24846] *** End of error message *** [ 4] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0x144)[0x2b221bcf1ae4] [admin:24848] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2b2239b45a93] [admin:24848] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2b0d6f540bda] [admin:24851] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2ad07aad7c64] [admin:24850] [ 9] steadyCompressibleMRFFoam[0x421670] [admin:24850] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ad07bfe8b15] [admin:24850] [11] steadyCompressibleMRFFoam[0x4241f9] [admin:24850] *** End of error message *** /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2b221c6bef89] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2b0d6f538c64] [admin:24851] [ 9] steadyCompressibleMRFFoam[0x421670] [admin:24851] [10] [admin:24848] [ 7] Mach number min = 0.472644 max = 0.522382 /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b0d70a49b15] [admin:24851] [11] steadyCompressibleMRFFoam[0x4241f9] [admin:24851] *** End of error message *** /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2ac432b3ba98] [admin:24845] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2ac450999a93] [admin:24845] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2b221c6cbbda] [admin:24848] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2ac433508f89] [admin:24845] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2b221c6c3c64] [admin:24848] [ 9] steadyCompressibleMRFFoam[0x421670] [admin:24848] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b221dbd4b15] [admin:24848] [11] steadyCompressibleMRFFoam[0x4241f9] [admin:24848] *** End of error message *** /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2ac433515bda] [admin:24845] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2ac43350dc64] [admin:24845] [ 9] steadyCompressibleMRFFoam[0x421670] [admin:24845] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ac434a1eb15] [admin:24845] [11] steadyCompressibleMRFFoam[0x4241f9] [admin:24845] *** End of error message *** -------------------------------------------------------------------------- mpirun noticed that process rank 2 with PID 0 on node admin exited on signal 8 (Floating point exception). --------------------------------------------------------------------------``` And it is my decomposeParDict: Code: ```numberOfSubdomains 8; method patchConstrained; // All GGI interfaces should be listed here: globalFaceZones ( perBlade1Zone perBlade2Zone perUp1Zone perUp2Zone perDown1Zone perDown2Zone ); patchConstrainedCoeffs { method scotch; numberOfSubdomains 8; // You can force the faces of a patch to be on the same processor // this is currently needed for mixingPlane patchConstraints ( ); } distributed no;``` I considered the incompatibility of foam-extend-4.0 with system openmpi2.0.1. So I try to compile openmpi1.8.8 in Thirdparty. But when I try mpirun, the terminal told me there is something wrong with MCA. The log is as follows: Code: ```[admin:21477] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_posix" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored [admin:21475] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_posix" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored [admin:21477] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_mmap" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored [admin:21475] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_mmap" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored [admin:21477] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_sysv" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored [admin:21475] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_sysv" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored -------------------------------------------------------------------------- It looks like opal_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during opal_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): opal_shmem_base_select failed --> Returned value -1 instead of OPAL_SUCCESS``` I'm very confused and don't how to solve it. Could anybody help me?

 Tags foam-extend 4.0, parallel calculation