parallel calculation issues

tjuAeron · December 20, 2018, 09:51

Hello everyone,

I'm very new to linux and OF, and currently I'm going with a very frustrating problem.

I've successfully set a case and run it to 5000 timesteps in serial without any mistake. So I'm now trying to run it in parallel. But it aborted at first timestep with the famous "float point exception". The log is as follows:

Code:

Time = 1

BiCGStab:  Solving for Ux, Initial residual = 1, Final residual = 0.00262781, No Iterations 1
BiCGStab:  Solving for Uy, Initial residual = 1, Final residual = 0.000321715, No Iterations 1
BiCGStab:  Solving for Uz, Initial residual = 1, Final residual = 0.00339929, No Iterations 2
BiCGStab:  Solving for p, Initial residual = 1, Final residual = 0.0440403, No Iterations 1
time step continuity errors : sum local = 0.0192303, global = -0.007747, cumulative = -0.007747
BiCGStab:  Solving for p, Initial residual = 0.861166, Final residual = 0.0152853, No Iterations 2
time step continuity errors : sum local = 0.068281, global = -0.0116883, cumulative = -0.0194353
BiCGStab:  Solving for i, Initial residual = 1, Final residual = 0.000584795, No Iterations 2
ExecutionTime = 3.99 s  ClockTime = 4 s

Cyclic GGI pair (perBlade1, perBlade2)
Area: 0.00523772 0.00523772 Diff = -5.96635e-13 or 1.13911e-08 % 
Flux: 155599 581.131 Diff = -155996 or 100.255 %
Cyclic GGI pair (perUp1, perUp2)
Area: 0.00967552 0.00967552 Diff = 3.23594e-13 or 3.34446e-09 % 
Flux: 46.6035 0.00235069 Diff = -46.6009 or 99.9944 %
Cyclic GGI pair (perDown1, perDown2)
Area: 0.00364872 0.00364872 Diff = -2.67976e-13 or 7.34439e-09 % 
Flux: 48.5907 124200 Diff = -123929 or 255047 %
Flux divergence min = 1.29895e-07 max = 8.40826e+06 average: 4.40663e+08
[admin:24845] *** Process received signal ***
[admin:24847] *** Process received signal ***
[admin:24847] Signal: Floating point exception (8)
[admin:24847] Signal code:  (-6)
[admin:24847] Failing at address: 0x4240000610f
[admin:24849] *** Process received signal ***
[admin:24849] Signal: Floating point exception (8)
[admin:24849] Signal code:  (-6)
[admin:24849] Failing at address: 0x42400006111
[admin:24845] Signal: Floating point exception (8)
[admin:24845] Signal code:  (-6)
[admin:24845] Failing at address: 0x4240000610d
[admin:24849] [ 0] [admin:24847] /lib64/libc.so.6(+0x35670)[0x2acb53b6a670]
[admin:24849] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2acb53b6a5f7]
[admin:24849] [ 2] /lib64/libc.so.6(+0x35670)[0x2acb53b6a670]
[admin:24849] [ 3] [admin:24850] *** Process received signal ***
[ 0] /lib64/libc.so.6(+0x35670)[0x2b0a72aea670]
[admin:24847] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b0a72aea5f7]
[admin:24847] [ 2] [admin:24850] Signal: Floating point exception (8)
[admin:24850] Signal code:  (-6)
[admin:24850] Failing at address: 0x42400006112
[admin:24846] *** Process received signal ***
[admin:24845] [ 0] /lib64/libc.so.6(+0x35670)[0x2b0a72aea670]
[admin:24847] [ 3] [admin:24846] Signal: Floating point exception (8)
[admin:24846] Signal code:  (-6)
[admin:24846] Failing at address: 0x4240000610e
/lib64/libc.so.6(+0x35670)[0x2ac434a32670]
[admin:24845] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2ac434a325f7]
[admin:24845] [ 2] /lib64/libc.so.6(+0x35670)[0x2ac434a32670]
[admin:24845] [ 3] [admin:24846] [ 0] [admin:24851] *** Process received signal ***
/public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2acb527d4d48]
[admin:24849] [ 4] /lib64/libc.so.6(+0x35670)[0x2b9a99132670]
[admin:24846] [ 1] [admin:24850] [ 0] /lib64/libc.so.6(gsignal+0x37)[0x2b9a991325f7]
[admin:24846] [ 2] /lib64/libc.so.6(+0x35670)[0x2ad07bffc670]
[admin:24850] [ 1] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2b0a71754d48]
[admin:24847] [ 4] /lib64/libc.so.6(+0x35670)[0x2b9a99132670]
[admin:24846] [ 3] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2acb51c73a98]
[admin:24849] [ 5] /lib64/libc.so.6(gsignal+0x37)[0x2ad07bffc5f7]
[admin:24850] [ 2] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2acb6fb4ca93]
[admin:24849] [ 6] /lib64/libc.so.6(+0x35670)[0x2ad07bffc670]
[admin:24850] [ 3] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2b0a70bf3a98]
[admin:24847] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2b0a8ea70a93]
[admin:24847] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2ac43369cd48]
[admin:24845] [ 4] [admin:24851] Signal: Floating point exception (8)
[admin:24851] Signal code:  (-6)
[admin:24851] Failing at address: 0x42400006113
[admin:24851] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2acb52640f89]
[admin:24849] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2b0a715c0f89]
[admin:24847] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2b9a97d9cd48]
[admin:24846] [ 4] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2ad07ac66d48]
[admin:24850] [ 4] [ 0] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2b9a9723ba98]
[admin:24846] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2b9ab5030a93]
[admin:24846] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2ad07a105a98]
/lib64/libc.so.6(+0x35670)[0x2b0d70a5d670]
[admin:24851] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b0d70a5d5f7]
[admin:24851] [ 2] /lib64/libc.so.6(+0x35670)[0x2b0d70a5d670]
[admin:24851] [ 3] [admin:24850] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2b0a715cdbda]
[admin:24847] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2acb5264dbda]
[admin:24849] [ 8] [admin:24848] *** Process received signal ***
[admin:24848] Signal: Floating point exception (8)
[admin:24848] Signal code:  (-6)
[admin:24848] Failing at address: 0x42400006110
/public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2b9a97c08f89]
[admin:24846] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2ad097f22a93]
[admin:24850] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2b0d6f6c7d48]
[admin:24848] [ 0] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2b0a715c5c64]
[admin:24847] [ 9] /lib64/libc.so.6(+0x35670)[0x2b221dbe8670]
[admin:24848] [ 1] steadyCompressibleMRFFoam[0x421670]
[admin:24847] [10] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2acb52645c64]
[admin:24849] [ 9] /lib64/libc.so.6(gsignal+0x37)[0x2b221dbe85f7]
[admin:24848] [ 2] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b0a72ad6b15]
[admin:24847] [11] steadyCompressibleMRFFoam[0x4241f9]
[admin:24847] *** End of error message ***
/lib64/libc.so.6(+0x35670)[0x2b221dbe8670]
[admin:24848] [ 3] steadyCompressibleMRFFoam[0x421670]
[admin:24849] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2acb53b56b15]
[admin:24849] [11] steadyCompressibleMRFFoam[0x4241f9]
[admin:24849] *** End of error message ***
[admin:24851] [ 4] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2b0d6eb66a98]
[admin:24851] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2b0d8c9aca93]
[admin:24851] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2ad07aad2f89]
[admin:24850] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2b9a97c15bda]
[admin:24846] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4sqrtERNS_5FieldIdEERKNS_5UListIdEE+0x28)[0x2b221c852d48]
[admin:24848] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2b0d6f533f89]
[admin:24851] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2ad07aadfbda]
[admin:24850] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2b9a97c0dc64]
[admin:24846] [ 9] steadyCompressibleMRFFoam[0x421670]
[admin:24846] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b9a9911eb15]
[admin:24846] [11] steadyCompressibleMRFFoam[0x4241f9]
[admin:24846] *** End of error message ***
[ 4] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0x144)[0x2b221bcf1ae4]
[admin:24848] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2b2239b45a93]
[admin:24848] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2b0d6f540bda]
[admin:24851] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2ad07aad7c64]
[admin:24850] [ 9] steadyCompressibleMRFFoam[0x421670]
[admin:24850] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ad07bfe8b15]
[admin:24850] [11] steadyCompressibleMRFFoam[0x4241f9]
[admin:24850] *** End of error message ***
/public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2b221c6bef89]
/public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2b0d6f538c64]
[admin:24851] [ 9] steadyCompressibleMRFFoam[0x421670]
[admin:24851] [10] [admin:24848] [ 7] Mach number min = 0.472644 max = 0.522382
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b0d70a49b15]
[admin:24851] [11] steadyCompressibleMRFFoam[0x4241f9]
[admin:24851] *** End of error message ***
/public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libcompressibleRASModels.so(_ZN4Foam4sqrtINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_+0xf8)[0x2ac432b3ba98]
[admin:24845] [ 5] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libutilityFunctionObjects.so(_ZN4Foam10MachNumber7executeEv+0x223)[0x2ac450999a93]
[admin:24845] [ 6] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2b221c6cbbda]
[admin:24848] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam18functionObjectList7executeEv+0x139)[0x2ac433508f89]
[admin:24845] [ 7] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2b221c6c3c64]
[admin:24848] [ 9] steadyCompressibleMRFFoam[0x421670]
[admin:24848] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b221dbd4b15]
[admin:24848] [11] steadyCompressibleMRFFoam[0x4241f9]
[admin:24848] *** End of error message ***
/public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4TimeppEv+0x7a)[0x2ac433515bda]
[admin:24845] [ 8] /public/home/linjw1/foam/foam-extend-4.0/lib/linux64GccDPOpt/libfoam.so(_ZN4Foam4Time4loopEv+0x24)[0x2ac43350dc64]
[admin:24845] [ 9] steadyCompressibleMRFFoam[0x421670]
[admin:24845] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ac434a1eb15]
[admin:24845] [11] steadyCompressibleMRFFoam[0x4241f9]
[admin:24845] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 0 on node admin exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------

And it is my decomposeParDict:

Code:

numberOfSubdomains 8;

method    patchConstrained;

// All GGI interfaces should be listed here:
globalFaceZones
(
    perBlade1Zone
    perBlade2Zone
    perUp1Zone
    perUp2Zone
    perDown1Zone
    perDown2Zone
);

patchConstrainedCoeffs
{
    method            scotch;
    numberOfSubdomains    8;
    // You can force the faces of a patch to be on the same processor
    // this is currently needed for mixingPlane
    patchConstraints
    (

    );
}

distributed     no;

I considered the incompatibility of foam-extend-4.0 with system openmpi2.0.1. So I try to compile openmpi1.8.8 in Thirdparty. But when I try mpirun, the terminal told me there is something wrong with MCA. The log is as follows:

Code:

[admin:21477] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_posix" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored
[admin:21475] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_posix" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored
[admin:21477] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_mmap" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored
[admin:21475] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_mmap" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored
[admin:21477] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_sysv" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored
[admin:21475] mca_base_component_repository_open: shmem "/public/home/linjw1/foam/foam-extend-4.0/ThirdParty/packages/openmpi-1.8.8/platforms/linux64GccDPOpt/lib/openmpi/mca_shmem_sysv" uses an MCA interface that is not recognized (component MCA v2.0.0 != supported MCA v2.1.0) -- ignored
--------------------------------------------------------------------------
It looks like opal_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_shmem_base_select failed
  --> Returned value -1 instead of OPAL_SUCCESS

I'm very confused and don't how to solve it. Could anybody help me?

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
MPI error in parallel application	usv001	OpenFOAM Programming & Development	2	September 14, 2017 12:30
chtMultiRegionSimpleFoam: crash on parallel run	student666	OpenFOAM Running, Solving & CFD	3	April 20, 2017 12:05
simpleFoam parallel	AndrewMortimer	OpenFOAM Running, Solving & CFD	12	August 7, 2015 19:45
Problem with parallel calculation	papteo	ANSYS	1	January 15, 2015 11:45
Coal combustion, big node, slow calculation, parallel	Whitebear	CFX	1	April 8, 2014 04:27