[clapointe]

Ah, so the point is that you were running them sequentially, but want to farm them out as individual jobs? You want to look into slurm job arrays. I don't have any scripts handy, but someone else might. Otherwise, a quick google reveals a ton of hits.

Caelan

[Sgvaa]

Yeah, I want to submit all job separately to save their outputs as separate files, but not necessary that jobs need to be sequentially submitted..

[Sgvaa]

Originally Posted by clapointe

If they are to be run sequentially, then you can combine them to form one (long) script. Or write a script that calls the others. In pseudo code :

slurm stuff here

load modules

source script 1
source script 2
source script 3
...
source script 50

Caelan

I tried following, but again created as single job only.

#!/bin/bash
#SBATCH --job-name=XXX
#SBATCH --cpus-per-task=1
#SBATCH --output=XXX.txt
#SBATCH --partition=long
#SBATCH --time=00:12:00
#SBATCH --mail-type=BEGIN,END,FAIL,TIME_LIMIT
#SBATCH --signal=XXX

load modules

/folder1/folder2/folder3 sbatch layer30.sbatch
/folder1/folder2/folder3 sbatch layer31.sbatch

[dduque]

I have finally been able to run in parallel within a slurm cluster with this two-file procedure !!

[dduque]

Originally Posted by AshaEgreck

mpirun --oversubscribe -np 4 buoyantBoussinesqPimpleFoam -parallel >



...




blockMesh
setFields
decomposePar

srun -n 1 my_mpi_application

Yes, this two-file procedure has finally done the trick on a slurm cluster !




(Posted twice, but I've been unable to delete previous post ... )

[guanjiang.chen]

Originally Posted by SymplPilot

Hi everyone,

Our cluster uses the SLURM batch system also. However, the "srun" command is recommended to run all jobs. For example, to run a Foam application on 16 cores one has to use something like:

srun -n 16 xxxFoam -parallel

This is a problem, since OpenFOAM expects to see:

mpirun -np 16 xxxFoam -parallel

I wonder if anybody knows how to fix OpenFOAM to run with "srun" and "-n" as opposed to "mpirun" and "-np" ? Thanks for the help.

-SP

Have you solved this problem?
I use "srun --mpi=pmi2 pisoFoam -parallel" or "srun --mpi=pmi2 -n 26 pisoFoam -parallel"

[guanjiang.chen]

#!/bin/bash
## Submission script for Cluster
#SBATCH --job-name=test
#SBATCH --time=0:01:00 # hh:mm:ss
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=28
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1000 # megabytes
#SBATCH --partition=cpu

## system error message output file
## leave %j as it's being replaced by JOB ID number
#SBATCH -e foamjobname.std.err_%j

## system message output file
#SBATCH -o foamjobname.std.out_%j

## send mail after job is finished
#SBATCH --mail-type=end
#SBATCH --mail-user=1224133639@qq.com

# Load modules required for runtime e.g.
source /mnt/storage/scratch/va19337/OpenFOAM/OpenFOAM-6/etc/bashrc WM_LABEL_SIZE=64 WM_MPLIB=OPENMPI FOAMY_HEX_MESH=yes


##### WM_MPLIB = SYSTEMOPENMPI | OPENMPI | SYSTEMMPI | MPICH | MPICH-GM | HPMPI
###### | MPI | FJMPI | QSMPI | SGIMPI | INTELMPI


srun hostname > all_nodes

export I_MPI_PROCESS_MANAGER=mpd


#decomposePar

echo $LD_LIBRARY_PATH > all_nodes1
which mpirun > all_nodes2
echo $I_MPI_PMI_LIBRARY

## run my MPI executable
##mpirun -np 112 pisoFoam -parallel
mpirun --hostfile all_nodes -np 112 icoFoam -parallel