[Denao]

Hi all,

I have an issue ending openfoamn jobs submitted via a slurm SBatch script.

I have the following submission script:

#!/bin/bash
#SBATCH --export=ALL
#SBATCH -J openFoam-TH-Aft-12kts
#SBATCH --nodes=1-1
#SBATCH --tasks-per-node=11
#SBATCH --time=48:00:00
#SBATCH --signal=B:18@1800

. /opt/openfoam5/etc/bashrc

mpirun interFoam -parallel

and in the controlDict have the follwoing:

OptimisationSwitches
{
    stopAtWriteNowSignal    18;
}

When the jobs reach the end of their allocated times the slurm_XXX.log shows the following:

slurmstepd-CFD2: error: *** JOB 171 ON CFD2 CANCELLED AT 2019-12-08T13:35:14 DUE TO TIME LIMIT ***
mpirun: Forwarding signal 18 to job
sigStopAtWriteNow : setting up write and stop at end of the next iteration

However timesteps are not written and, when i check the cluster, I see orphaned interFoam tasks, some still running on a core at 100% and some taking very low %.

Any assistance in getting these jobs to terminate properly would be appreciated (as I am loosing the results at the end of the job!)

[Denao]

Slight update on the above. It appears that, in at least some cases that the timestep is in fact written. However the presence of orhaned processes makes me wonder how complete/reliable this is.

[jherb]

Do you know, how long SLURM waits, until it finally kills the job after sending the signal?