[puneet336]

Hi all,
I am trying to run a fluent 3d simulation with 16 processes on a single node having centos 6.5 with fluent 15 and i ran into some runtime error.
time fluent -p -g 3d -t 16 -i jour.jou -mpi=intel
i have tried both intel & openmpi , the error is same in both cases.

What might be the reason behind this(w.r.t fluent)? Anyone faced this issue ? Can this be an issue with .dat/.cas file & is there a way to verify the same? Any help/hint will be very useful.

Please let me know if more information is required from my side.
The log is with openmpi-1.10:

Writing "first-102.cas"...
   38700 hexahedral cells, zone  2, binary.
   ......
   41236 node flags, binary.
Done.



*** glibc detected *** /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7: free(): invalid next size (normal): 0x00000000055cfdb0 ***
*** glibc detected *** /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7: malloc(): memory corruption: 0x000000000551ee40 ***
*** glibc detected *** /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7: free(): invalid next size (normal): 0x00000000056fdac0 ***
*** glibc detected *** /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7: malloc(): memory corruption: 0x000000000551ee40 ***
*** glibc detected *** /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7: malloc(): memory corruption: 0x00000000055d0dc0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x30c6675e66]
/lib64/libc.so.6[0x30c66789b3]
/home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7[0x1eee49c]
/home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7(PRF_Command_Start+0xfb5)[0x7a5135]
/home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7(PRF_Node_repl+0x2e)[0x7a413e]
/home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7(fluent_main+0x387)[0x151be57]
/home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7(main+0x55)[0x151c995]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x30c661ed5d]
/home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7(__gxx_personality_v0+0x479)[0x4bce59]
======= Memory map: ========
00400000-02378000 r-xp 00000000 b5:84fa2 144115322612362348              /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7
02578000-02841000 rw-p 01f78000 b5:84fa2 144115322612362348              /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7
02841000-03770000 rw-p 00000000 00:00 0 
0433a000-05b0e000 rw-p 00000000 00:00 0                                  [heap]
30c5e00000-30c5e20000 r-xp 00000000 08:05 791698                         /lib64/ld-2.12.so
30c601f000-30c6020000 r--p 0001f000 08:05 791698                         /lib64/ld-2.12.so
30c6020000-30c6021000 rw-p 00020000 08:05 791698                         /lib64/ld-2.12.so
30c6021000-30c6022000 rw-p 00000000 00:00 0 
30c6200000-30c6202000 r-xp 00000000 08:05 791915                         /lib64/libdl-2.12.so
30c6202000-30c6402000 ---p 00002000 08:05 791915                         /lib64/libdl-2.12.so
30c6402000-30c6403000 r--p 00002000 08:05 791915                         /lib64/libdl-2.12.so
30c6403000-30c6404000 rw-p 00003000 08:05 791915                         /lib64/libdl-2.12.so
30c6600000-30c678a000 r-xp 00000000 08:05 791719                         /lib64/libc-2.12.so
30c678a000-30c698a000 ---p 0018a000 08:05 791719                         /lib64/libc-2.12.so
30c698a000-30c698e000 r--p 0018a000 08:05 791719                         /lib64/libc-2.12.so
30c698e000-30c698f000 rw-p 0018e000 08:05 791719                         /lib64/libc-2.12.so
30c698f000-30c6994000 rw-p 00000000 00:00 0 
30c6a00000-30c6a17000 r-xp 00000000 08:05 792002                         /lib64/libpthread-2.12.so
30c6a17000-30c6c17000 ---p 00017000 08:05 792002                         /lib64/libpthread-2.12.so
30c6c17000-30c6c18000 r--p 00017000 08:05 792002                         /lib64/libpthread-2.12.so
30c6c18000-30c6c19000 rw-p 00018000 08:05 792002                         /lib64/libpthread-2.12.so
30c6c19000-30c6c1d000 rw-p 00000000 00:00 0 
30c6e00000-30c6e83000 r-xp 00000000 08:05 791817                         /lib64/libm-2.12.so
30c6e83000-30c7082000 ---p 00083000 08:05 791817                         /lib64/libm-2.12.so
30c7082000-30c7083000 r--p 00082000 08:05 791817                         /lib64/libm-2.12.so
30c7083000-30c7084000 rw-p 00083000 08:05 791817                         /lib64/libm-2.12.so
30c7200000-30c7207000 r-xp 00000000 08:05 791998                         /lib64/librt-2.12.so
30c7207000-30c7406000 ---p 00007000 08:05 791998                         /lib64/librt-2.12.so
30c7406000-30c7407000 r--p 00006000 08:05 791998                         /lib64/librt-2.12.so
30c7407000-30c7408000 rw-p 00007000 08:05 791998                         /lib64/librt-2.12.so
30c7a00000-30c7a16000 r-xp 00000000 08:05 791677                         /lib64/libgcc_s-4.4.7-20120601.so.1
30c7a16000-30c7c15000 ---p 00016000 08:05 791677                         /lib64/libgcc_s-4.4.7-20120601.so.1
30c7c15000-30c7c16000 rw-p 00015000 08:05 791677                         /lib64/libgcc_s-4.4.7-20120601.so.1
30c8200000-30c8216000 r-xp 00000000 08:05 791837                         /lib64/libnsl-2.12.so
30c8216000-30c8415000 ---p 00016000 08:05 791837                         /lib64/libnsl-2.12.so
30c8415000-30c8416000 r--p 00015000 08:05 791837                         /lib64/libnsl-2.12.so
30c8416000-30c8417000 rw-p 00016000 08:05 791837                         /lib64/libnsl-2.12.so
30c8417000-30c8419000 rw-p 00000000 00:00 0 
30c8e00000-30c8e02000 r-xp 00000000 08:05 791647                         /lib64/libutil-2.12.so
30c8e02000-30c9001000 ---p 00002000 08:05 791647                         /lib64/libutil-2.12.so
30c9001000-30c9002000 r--p 00001000 08:05 791647                         /lib64/libutil-2.12.so
30c9002000-30c9003000 rw-p 00002000 08:05 791647                         /lib64/libutil-2.12.so
2b542d789000-2b542d78b000 rw-p 00000000 00:00 0 
2b542d78b000-2b542d79a000 r-xp 00000000 b5:84fa2 144115322612361098      /home/apps/ansys_inc/v150/fluent/fluent15.0.7/multiport/mpi_wrapper/lnamd64/openmpi/libmpi_wrapper.so
2b542d79a000-2b542d999000 ---p 0000f000 b5:84fa2 144115322612361098      /home/apps/ansys_inc/v150/fluent/fluent15.0.7/multiport/mpi_wrapper/lnamd64/openmpi/libmpi_wrapper.so
2b542d999000-2b542d99a000 rw-p 0000e000 b5:84fa2 144115322612361098      /home/apps/ansys_inc/v150/fluent/fluent15.0.7/multiport/mpi_wrapper/lnamd64/openmpi/libmpi_wrapper.so
2b542d99a000-2b542d99b000 rw-p 00000000 00:00 0 
2b542d9a9000-2b542dc30000 r-xp 00000000 b5:84fa2 144115322612362321      /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/syslib/libimf.so
2b542dc30000-2b542dd30000 ---p 00287000 b5:84fa2 144115322612362321      /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/syslib/libimf.so
2b542dd30000-2b542dd74000 rw-p 00287000 b5:84fa2 144115322612362321      /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/syslib/libimf.so
2b542dd74000-2b542dd75000 rw-p 00000000 00:00 0 
2b542dd75000-2b542ddbe000 r-xp 00000000 b5:84fa2 144115322612362315      /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/syslib/libintlc.so.5
2b542ddbe000-2b542debd000 ---p 00049000 b5:84fa2 144115322612362315      /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/syslib/libintlc.so.5
==============================================================================

Node 9: Process 29678: Received signal SIGIOT.

==============================================================================

===============Message from the Cortex Process================================

Fatal error in one of the compute processes.

==============================================================================

also with single process run (-t 1) i got:

Writing "first-102.cas"...
........
Done.



*** glibc detected *** /home/apps/ansys_inc/v150/fluent/fluent15.0.7/lnamd64/3d_node/fluent_mpi.15.0.7: malloc(): memory corruption: 0x000000000a88e620 ***

and the fluent hangs

Iattached to the running fluent processes via gdb and i got:

(gdb) backtrace
#0  0x00000030c6a0cd2b in pthread_once () from /lib64/libpthread.so.0
#1  0x00000030c66feac4 in backtrace () from /lib64/libc.so.6
#2  0x00000030c667054b in __libc_message () from /lib64/libc.so.6
#3  0x00000030c6675e66 in malloc_printerr () from /lib64/libc.so.6
#4  0x00000030c6679904 in _int_malloc () from /lib64/libc.so.6
#5  0x00000030c667a6b1 in malloc () from /lib64/libc.so.6
#6  0x00000030c5e0cbfd in _dl_map_object_deps () from /lib64/ld-linux-x86-64.so.2
#7  0x00000030c5e12af1 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#8  0x00000030c5e0e206 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#9  0x00000030c5e1254a in _dl_open () from /lib64/ld-linux-x86-64.so.2
#10 0x00000030c6726ca0 in do_dlopen () from /lib64/libc.so.6
#11 0x00000030c5e0e206 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#12 0x00000030c6726df7 in __libc_dlopen_mode () from /lib64/libc.so.6
#13 0x00000030c66fe995 in init () from /lib64/libc.so.6
#14 0x00000030c6a0cd33 in pthread_once () from /lib64/libpthread.so.0
#15 0x00000030c66feac4 in backtrace () from /lib64/libc.so.6
#16 0x00000030c667054b in __libc_message () from /lib64/libc.so.6
#17 0x00000030c6675e66 in malloc_printerr () from /lib64/libc.so.6
#18 0x00000030c6679904 in _int_malloc () from /lib64/libc.so.6
#19 0x00000030c667a6b1 in malloc () from /lib64/libc.so.6
#20 0x0000000001e85387 in ?? ()
#21 0x0000000001e8346a in vmalloc ()
#22 0x0000000001ed0322 in CX_Malloc_MU_Unsafe ()
#23 0x000000000095c822 in Datashipper_new ()
#24 0x00000000009495ba in Write_Thread_Data_Section ()
#25 0x000000000094a137 in Write_Data_Section ()
#26 0x0000000000726f65 in ?? ()
#27 0x0000000001ee2afa in eval ()
#28 0x00000000007a546a in PRF_Command_Start ()
#29 0x00000000007a413e in PRF_Node_repl ()
#30 0x000000000151be57 in fluent_main ()
#31 0x000000000151c995 in main ()

[puneet336]

Thankyou all ,
it seems bit weird but -mpi=pcmpi solved my problem!!