CFD Online Discussion Forums

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)
-   OpenFOAM Running, Solving & CFD (https://www.cfd-online.com/Forums/openfoam-solving/)
-   -   Backtrace errors while running on a HPC with buoyantBoussinesqSimpleFoam (https://www.cfd-online.com/Forums/openfoam-solving/128443-backtrace-errors-while-running-hpc-buoyantboussinesqsimplefoam.html)

Thangam January 12, 2014 21:26

Backtrace errors while running on a HPC with buoyantBoussinesqSimpleFoam
 
I encountered this error while running a job on a HPC. The job was aborted with this error. However, when I rescheduled the same to job to continue from the latestTime, it continued without any error. Any ideas on why this happened?


Code:

buoyantBoussinesqSimpleFoam:10105 terminated with signal 7 at PC=507590 SP=7fffb7654708. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10112 terminated with signal 7 at PC=507590 SP=7fff19623c88. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10113 terminated with signal 7 at PC=507590 SP=7fff360e7b08. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10106 terminated with signal 7 at PC=507590 SP=7fffa422d488. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10108 terminated with signal 7 at PC=507590 SP=7fff43220788. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10109 terminated with signal 7 at PC=507590 SP=7fff7e663388. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10111 terminated with signal 7 at PC=507590 SP=7fffdb30f808. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10107 terminated with signal 7 at PC=507590 SP=7fffb1dfb588. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10110 terminated with signal 7 at PC=2b3ff3a12690 SP=7fff5e99a128. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10104 terminated with signal 7 at PC=507590 SP=7fff9e866e08. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10792 terminated with signal 7 at PC=3557609a43 SP=7fff5c925130. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10791 terminated with signal 7 at PC=2b920bf0a070 SP=7fff87390738. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10399 terminated with signal 7 at PC=2aae1a9a8070 SP=7fffab21e7f8. 
 Backtrace:
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b5dca08dd11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b5dcc652d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b5dcc650a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b5dcc65e79c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b5dca3cd378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 
 buoyantBoussinesqSimpleFoam:9761 terminated with signal 7 at PC=2ad7de11a070 SP=7fff1b9a6f38. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10400 terminated with signal 7 at PC=304f609a43 SP=7fffa6c2e370. 
 Backtrace:
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(+0x16e690)
 [0x2b3ff3a12690]
 
 buoyantBoussinesqSimpleFoam:9762 terminated with signal 7 at PC=3cdc80df45 SP=7fff0509d690. 
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:9763 terminated with signal 7 at PC=2aeeaadb7370 SP=7fff094f61c8. 
 Backtrace:
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfieldFunctionObjects.so(+0x8f370)
 [0x2aeeaadb7370]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ac5c17f7d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ac5c3dbcd69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ac5c3dbaa23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ac5c3dc879c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ac5c1b37378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b4dc0073d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b4dc2638d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b4dc2636a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b4dc264479c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b4dc03b3378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ad0cba0ad11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ad0cdfcfd69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ad0cdfcda23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ad0cdfdb79c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ad0cbd4a378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ae02173dd11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ae023d02d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ae023d00a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ae023d0e79c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ae021a7d378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ae010494d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ae012a59d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ae012a57a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ae012a6579c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ae0107d4378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b8dfcbdfd11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b8dff1a4d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b8dff1a2a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b8dff1b079c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b8dfcf1f378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b3a42c51d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b3a45216d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b3a45214a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b3a4522279c]/group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b3a42f91378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 --------------------------------------------------------------------------
 Sorry!  You were supposed to get help about:
    odls-default:could-not-kill
 But I couldn't open the help file:
    /ivec/devel/intel/12.1.7/openmpi/1.6.5/share/openmpi/help-odls-default.txt: Interrupted
 system call.  Sorry!
 --------------------------------------------------------------------------
 
 buoyantBoussinesqSimpleFoam:10382 terminated with signal 7 at PC=2b6086d8f070 SP=7fffe8f4e138. 
 Backtrace:
 --------------------------------------------------------------------------
 mpirun noticed that process rank 61 with PID 10107 on node e530 exited on signal 7 (Bus
 error).


lakeat July 11, 2014 11:45

I saw a similar error these days. Have you found an answer?

Thangam July 11, 2014 23:39

Hi,

I did not go much into details but I understood (after some googling!) that
signal 7 is a bus error. Which you can see from:

--------------------------------------------------------------------------
mpirun noticed that process rank 61 with PID 10107 on node e530 exited on signal 7 (Bus
error).
------------------------------------------------------------------------------------------------------------------------------------------
I am not sure about the details though. However, some explanations for the bus error:

A "Bus Error" may come, for example, from accessing unaligned data (i.e. like trying to access a 4 byte integer with a pointer to the middle of it). What this means will vary from platform to platform. (I haven't come up with a good example of this one yet.)

A "Bus Error" can also often indicate a memory overwrite, e.g. somebody wrote a number where a pointer is kept. Often caused by going past the end of an array and into the system pointers at the start of the next memory block.


All times are GMT -4. The time now is 07:23.