[Varada]

Hello REEF3D team,

I am currently using REEF3D-23.03/REEF3D-release_candidate/ version and working on a wave structure interaction problem within a three-dimensional numerical wave tank. When I run the simulation on my personal computer, utilizing 12 cores, it runs without any issues. However, upon attempting to run the simulation on the High-Performance Computing Environment (HPCE), I encountered an error message, which I've provided below.

I also attempted to run the simulation on HPCE with 8 cores, and it proceeded smoothly without any errors. I have attached the relevant files for your review. Could you please examine the attached files and help me identify the problem? I have also tried the latest release candidate (RC), but I am still facing the same issue when using a higher number of cores.

Thank you for your assistance.

[Attach relevant files here]

[Varada]

Please review the error I mentioned earlier.

[cn249:17924] *** Process received signal ***
[cn249:17924] Signal: Segmentation fault (11)
[cn249:17924] Signal code: Address not mapped (1)
[cn249:17924] Failing at address: 0x357e6
[cn249:17924] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x2b16af07b5d0]
[cn249:17924] [ 1] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x8ec000]
[cn249:17924] [ 2] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x53f81f]
[cn249:17924] [ 3] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x5982c3]
[cn249:17924] [ 4] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x597ef5]
[cn249:17924] [ 5] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x85ce84]
[cn249:17924] [ 6] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x5b3abf]
[cn249:17924] [ 7] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x5b4711]
[cn249:17924] [ 8] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x41c2dd]
[cn249:17924] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b16af2aa3d5]
[cn249:17924] [10] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x41f55f]
[cn249:17924] *** End of error message ***
[cn249:17933] *** Process received signal ***
[cn249:17933] Signal: Segmentation fault (11)
[cn249:17933] Signal code: (128)
[cn249:17933] Failing at address: (nil)
[cn249:17933] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x2ab5b6f2e5d0]
[cn249:17933] [ 1] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x8ec0ec]
[cn249:17933] [ 2] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x53f691]
[cn249:17933] [ 3] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x5982c3]
[cn249:17933] [ 4] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x597ef5]
[cn249:17933] [ 5] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x85ce84]
[cn249:17933] [ 6] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x5b3abf]
[cn249:17933] [ 7] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x5b4711]
[cn249:17933] [ 8] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x41c2dd]
[cn249:17933] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ab5b715d3d5]
[cn249:17933] [10] /lfs/usrhome/phd/oe19d201/REEF3D-code/REEF3D-master-23.08/bin/REEF3D[0x41f55f]
[cn249:17933] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 54 with PID 0 on node cn249 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

[kamath]

My guess is that this is a memory allocation problem on the HPC. Check how much total RAM you are getting allocated on the HPC for your 80 cores. It probably needs more than that.
Do you get allocated 8 cores and 80 cores on the same node? Or do you get more nodes when you increase the number of cores to 80?
Generally memory is attached to the nodes. Each node generally has may be 16/32 cores depending on the machine.

But you say your are using the RC, while the path shows master. Are you sure about the version you are using?

Also dont need M 20 2 in ctrl.txt

[keshav_20]

We faced the same problem with REEF3D RC and master version. The Job is run on 2 nodes with 40 cores each and 192 GB ram.
no. of nodes = 2, no. of cores = 40 x 2 = 80, Total RAM = 192 x 2 = 384 GB