CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

Backtrace errors while running on a HPC with buoyantBoussinesqSimpleFoam

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   January 12, 2014, 22:26
Default Backtrace errors while running on a HPC with buoyantBoussinesqSimpleFoam
  #1
Member
 
Thangam Natarajan
Join Date: Dec 2010
Location: Perth
Posts: 57
Rep Power: 7
Thangam is on a distinguished road
I encountered this error while running a job on a HPC. The job was aborted with this error. However, when I rescheduled the same to job to continue from the latestTime, it continued without any error. Any ideas on why this happened?


Code:
buoyantBoussinesqSimpleFoam:10105 terminated with signal 7 at PC=507590 SP=7fffb7654708.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10112 terminated with signal 7 at PC=507590 SP=7fff19623c88.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10113 terminated with signal 7 at PC=507590 SP=7fff360e7b08.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10106 terminated with signal 7 at PC=507590 SP=7fffa422d488.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10108 terminated with signal 7 at PC=507590 SP=7fff43220788.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10109 terminated with signal 7 at PC=507590 SP=7fff7e663388.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10111 terminated with signal 7 at PC=507590 SP=7fffdb30f808.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10107 terminated with signal 7 at PC=507590 SP=7fffb1dfb588.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10110 terminated with signal 7 at PC=2b3ff3a12690 SP=7fff5e99a128.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10104 terminated with signal 7 at PC=507590 SP=7fff9e866e08.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10792 terminated with signal 7 at PC=3557609a43 SP=7fff5c925130.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10791 terminated with signal 7 at PC=2b920bf0a070 SP=7fff87390738.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10399 terminated with signal 7 at PC=2aae1a9a8070 SP=7fffab21e7f8.  
 Backtrace:
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b5dca08dd11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b5dcc652d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b5dcc650a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b5dcc65e79c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b5dca3cd378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 
 buoyantBoussinesqSimpleFoam:9761 terminated with signal 7 at PC=2ad7de11a070 SP=7fff1b9a6f38.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:10400 terminated with signal 7 at PC=304f609a43 SP=7fffa6c2e370.  
 Backtrace:
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(+0x16e690)
 [0x2b3ff3a12690]
 
 buoyantBoussinesqSimpleFoam:9762 terminated with signal 7 at PC=3cdc80df45 SP=7fff0509d690.  
 Backtrace:
 
 buoyantBoussinesqSimpleFoam:9763 terminated with signal 7 at PC=2aeeaadb7370 SP=7fff094f61c8.  
 Backtrace:
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfieldFunctionObjects.so(+0x8f370)
 [0x2aeeaadb7370]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ac5c17f7d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ac5c3dbcd69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ac5c3dbaa23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ac5c3dc879c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ac5c1b37378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b4dc0073d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b4dc2638d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b4dc2636a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b4dc264479c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b4dc03b3378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ad0cba0ad11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ad0cdfcfd69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ad0cdfcda23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ad0cdfdb79c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ad0cbd4a378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ae02173dd11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ae023d02d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ae023d00a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ae023d0e79c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ae021a7d378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2ae010494d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2ae012a59d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2ae012a57a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2ae012a6579c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2ae0107d4378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b8dfcbdfd11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b8dff1a4d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b8dff1a2a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b8dff1b079c]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b8dfcf1f378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 buoyantBoussinesqSimpleFoam(_intel_fast_memcpy+0x0)[0x507590]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZNK4Foam21processorFvPatchFieldIdE21up
 dateInterfaceMatrixERKNS_5FieldIdEERS3_RKNS_9lduMatrixES5_hNS_8UPstream10commsTypesE+0x371)
 [0x2b3a42c51d11]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix22updateMatrixInterfaces
 ERKNS_10FieldFieldINS_5FieldEdEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEERKNS2_IdEERSC_h+0x339
 )[0x2b3a45216d69]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam9lduMatrix4AmulERNS_5FieldIdEERKNS
 _3tmpIS2_EERKNS_10FieldFieldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEh+0x243)
 [0x2b3a45214a23]
 /group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libOpenFOAM.so(_ZNK4Foam3PCG5solveERNS_5FieldIdEERKS2_h+0x
 8bc)[0x2b3a4522279c]/group/fdrg/OpenFOAM-2.1.x/platforms/linux64IccDPOpt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS_10dicti
 onaryE+0x688)[0x2b3a42f91378]
 buoyantBoussinesqSimpleFoam(_ZN4Foam8fvMatrixIdE5solveEv+0x11b)[0x46259b]
 buoyantBoussinesqSimpleFoam[0x423287]
 /lib64/libc.so.6(__libc_start_main+0xfd)[0x30a281ecdd]
 buoyantBoussinesqSimpleFoam[0x4214d9]
 --------------------------------------------------------------------------
 Sorry!  You were supposed to get help about:
     odls-default:could-not-kill
 But I couldn't open the help file:
     /ivec/devel/intel/12.1.7/openmpi/1.6.5/share/openmpi/help-odls-default.txt: Interrupted 
 system call.  Sorry!
 --------------------------------------------------------------------------
 
 buoyantBoussinesqSimpleFoam:10382 terminated with signal 7 at PC=2b6086d8f070 SP=7fffe8f4e138.  
 Backtrace:
 --------------------------------------------------------------------------
 mpirun noticed that process rank 61 with PID 10107 on node e530 exited on signal 7 (Bus 
 error).
Thangam is offline   Reply With Quote

Old   July 11, 2014, 11:45
Default
  #2
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 12
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
I saw a similar error these days. Have you found an answer?
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   July 11, 2014, 23:39
Default
  #3
Member
 
Thangam Natarajan
Join Date: Dec 2010
Location: Perth
Posts: 57
Rep Power: 7
Thangam is on a distinguished road
Hi,

I did not go much into details but I understood (after some googling!) that
signal 7 is a bus error. Which you can see from:

--------------------------------------------------------------------------
mpirun noticed that process rank 61 with PID 10107 on node e530 exited on signal 7 (Bus
error).
------------------------------------------------------------------------------------------------------------------------------------------
I am not sure about the details though. However, some explanations for the bus error:

A "Bus Error" may come, for example, from accessing unaligned data (i.e. like trying to access a 4 byte integer with a pointer to the middle of it). What this means will vary from platform to platform. (I haven't come up with a good example of this one yet.)

A "Bus Error" can also often indicate a memory overwrite, e.g. somebody wrote a number where a pointer is kept. Often caused by going past the end of an array and into the system pointers at the start of the next memory block.
Thangam is offline   Reply With Quote

Reply

Tags
buoyanboussinesqsimple, error, hpc

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Errors in running "wmake" Thamali OpenFOAM Programming & Development 5 November 11, 2013 08:40
Upgraded from Karmic Koala 9.10 to Lucid Lynx10.04.3 bookie56 OpenFOAM Installation 8 August 13, 2011 04:03
Errors in running a icoFsiFoam case jin_xu OpenFOAM Pre-Processing 0 June 9, 2008 06:48
Kubuntu uses dash breaks All scripts in tutorials platopus OpenFOAM Bugs 8 April 15, 2008 07:52
IcoFoam parallel woes msrinath80 OpenFOAM Running, Solving & CFD 9 July 22, 2007 02:58


All times are GMT -4. The time now is 16:02.