Backtrace errors while running on a HPC with buoyantBoussinesqSimpleFoam

Thangam · January 12, 2014, 21:26

I encountered this error while running a job on a HPC. The job was aborted with this error. However, when I rescheduled the same to job to continue from the latestTime, it continued without any error. Any ideas on why this happened?

Code:

buoyantBoussinesqSimpleFoam:10105 terminated with signal 7 at PC=507590 SP=7fffb7654708.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10112 terminated with signal 7 at PC=507590 SP=7fff19623c88.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10113 terminated with signal 7 at PC=507590 SP=7fff360e7b08.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10106 terminated with signal 7 at PC=507590 SP=7fffa422d488.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10108 terminated with signal 7 at PC=507590 SP=7fff43220788.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10109 terminated with signal 7 at PC=507590 SP=7fff7e663388.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10111 terminated with signal 7 at PC=507590 SP=7fffdb30f808.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10107 terminated with signal 7 at PC=507590 SP=7fffb1dfb588.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10110 terminated with signal 7 at PC=2b3ff3a12690 SP=7fff5e99a128.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10104 terminated with signal 7 at PC=507590 SP=7fff9e866e08.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10792 terminated with signal 7 at PC=3557609a43 SP=7fff5c925130.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10791 terminated with signal 7 at PC=2b920bf0a070 SP=7fff87390738.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10399 terminated with signal 7 at PC=2aae1a9a8070 SP=7fffab21e7f8.  
 Backtrace:
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b5dca08dd11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b5dcc652d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b5dcc650a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b5dcc65e79c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b5dca3cd378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 
 buoyantBoussinesqSimpleFoam:9761 terminated with signal 7 at PC=2ad7de11a070 SP=7fff1b9a6f38.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10400 terminated with signal 7 at PC=304f609a43 SP=7fffa6c2e370.  
 Backtrace:
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(+0x16e690)
 [0x2b3ff3a12690]
 
 buoyantBoussinesqSimpleFoam:9762 terminated with signal 7 at PC=3cdc80df45 SP=7fff0509d690.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:9763 terminated with signal 7 at PC=2aeeaadb7370 SP=7fff094f61c8.  
 Backtrace:
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfieldFunctionObjects.so(+0x8f370)
 [0x2aeeaadb7370]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ac5c17f7d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ac5c3dbcd69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ac5c3dbaa23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ac5c3dc879c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ac5c1b37378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b4dc0073d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b4dc2638d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b4dc2636a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b4dc264479c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b4dc03b3378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ad0cba0ad11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ad0cdfcfd69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ad0cdfcda23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ad0cdfdb79c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ad0cbd4a378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ae02173dd11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ae023d02d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ae023d00a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ae023d0e79c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ae021a7d378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ae010494d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ae012a59d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ae012a57a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ae012a6579c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ae0107d4378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b8dfcbdfd11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b8dff1a4d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b8dff1a2a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b8dff1b079c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b8dfcf1f378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b3a42c51d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b3a45216d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b3a45214a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b3a4522279c]/group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b3a42f91378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 --------------------------------------------------------------------------
 Sorry!  You were supposed to get help about:
     odls-default:could-not-kill
 But I couldn't open the help file:
     /ivec/devel/intel/12.1.7/openmpi/1.6.5/share/openmpi/help-odls-default.txt: Interrupted 
 system call.  Sorry!
 --------------------------------------------------------------------------
 
 buoyantBoussinesqSimpleFoam:10382 terminated with signal 7 at PC=2b6086d8f070 SP=7fffe8f4e138.  
 Backtrace:
 --------------------------------------------------------------------------
 mpirun noticed that process rank 61 with PID 10107 on node e530 exited on signal 7 (Bus 
 error).

lakeat · July 11, 2014, 11:45

I saw a similar error these days. Have you found an answer?

Thangam · July 11, 2014, 23:39

Hi,

I did not go much into details but I understood (after some googling!) that
signal 7 is a bus error. Which you can see from:

--------------------------------------------------------------------------
mpirun noticed that process rank 61 with PID 10107 on node e530 exited on signal 7 (Bus
error).
------------------------------------------------------------------------------------------------------------------------------------------
I am not sure about the details though. However, some explanations for the bus error:

A "Bus Error" may come, for example, from accessing unaligned data (i.e. like trying to access a 4 byte integer with a pointer to the middle of it). What this means will vary from platform to platform. (I haven't come up with a good example of this one yet.)

A "Bus Error" can also often indicate a memory overwrite, e.g. somebody wrote a number where a pointer is kept. Often caused by going past the end of an array and into the system pointers at the start of the next memory block.

July 11, 2014, 11:45		#2
lakeat Senior Member Daniel WEI (老魏) Join Date: Mar 2009 Location: Beijing, China Posts: 689 Blog Entries: 9 Rep Power: 21	I saw a similar error these days. Have you found an answer? __________________ ~ Daniel WEI ------------- Boeing Research & Technology - China Beijing, China Email

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Errors in running "wmake"	Thamali	OpenFOAM Programming & Development	5	November 11, 2013 07:40
Upgraded from Karmic Koala 9.10 to Lucid Lynx10.04.3	bookie56	OpenFOAM Installation	8	August 13, 2011 04:03
Errors in running a icoFsiFoam case	jin_xu	OpenFOAM Pre-Processing	0	June 9, 2008 06:48
Kubuntu uses dash breaks All scripts in tutorials	platopus	OpenFOAM Bugs	8	April 15, 2008 07:52
IcoFoam parallel woes	msrinath80	OpenFOAM Running, Solving & CFD	9	July 22, 2007 02:58

July 11, 2014, 23:39		#3
Thangam Member Thangam Natarajan Join Date: Dec 2010 Location: Perth Posts: 70 Rep Power: 16	Hi, I did not go much into details but I understood (after some googling!) that signal 7 is a bus error. Which you can see from: -------------------------------------------------------------------------- mpirun noticed that process rank 61 with PID 10107 on node e530 exited on signal 7 (Bus error). ------------------------------------------------------------------------------------------------------------------------------------------ I am not sure about the details though. However, some explanations for the bus error: A "Bus Error" may come, for example, from accessing unaligned data (i.e. like trying to access a 4 byte integer with a pointer to the middle of it). What this means will vary from platform to platform. (I haven't come up with a good example of this one yet.) A "Bus Error" can also often indicate a memory overwrite, e.g. somebody wrote a number where a pointer is kept. Often caused by going past the end of an array and into the system pointers at the start of the next memory block.