SU2 process returned error '15' during optimization
 

Hello everyone,

I am using the SU2_7.0.6 version for 3d optimization with High-Performance Computer having 40 processors on each node. My slurm file is as follows.


#!/bin/bash
#SBATCH --job-name=Delta_5
#SBATCH --nodes=3
#SBATCH --ntasks=120
#SBATCH --time=24:00:00
#SBATCH -o slurmjob-%j.out
#SBATCH -e slurmjob-%j.err

module load compiler/openmpi/3.1.0/gnu
module load python/3.6

export SU2_RUN=/home/apps/su2_7.0.6_AD_support/SU2_7.0.6/bin
export SU2_HOME=/home/apps/su2_7.0.6_AD_support/SU2_7.0.6/SU2-7.0.6
export PATH=$PATH:$SU2_RUN
export PYTHONPATH=$PYTHONPATH:$SU2_RUN


export SU2_MPI_COMMAND="mpirun -np %i %s"

shape_optimization.py -n 120 -o SLSQP -f turb_Corr_Del_5_optimisation.cfg


While using more than 1 node for optimization it runs for 2-3 iterations and the optimization stops with the error message
“SU2 process returned error '15'
Abort(811688463) on node 17 (rank 17 in comm 0): Fatal error in PMPI_Finalize: Other MPI error, error stack:
PMPI_Finalize(367)...............: MPI_Finalize failed
PMPI_Finalize(278)...............:
MPID_Finalize(1033)..............:
MPIDI_OFI_mpi_finalize_hook(1579): OFI domain close failed (ofi_init.c:1579:MPIDI_OFI_mpi_finalize_hook:Devic e or resource busy) ”


The process runs for 2-3 more iteration when restarting the project and the same error repeats.


Please suggest what might be causing this error and how I can overcome this issue.

Thank you
Regards
Bikalpa Bomjan Gurung