[Bashar]

Hi,
sorry to ask a general question, but I need some advice regarding job submission for oepnFoam. My university have Linux cluster ,and I need to start working on it to run my case in parallel. I have no idea about this kind of work , and I am the first student to use openFoam on this cluster. I managed to know that I need to use "SLURM job submission".I already searched the forums and I couldn't find something that is clear to me.
Any idea about the steps to do that?

Bashar

[Bashar]

I searched a lot and I found that the best way will be to see some old example using PBS then migrating the command to SLURM. I will update you if I managed to get it the right way!
Attached is a cheat sheet for migrating between different systems for anyone interested.

Best wishes
Bashar

[clapointe]

Hi Bashar,

I copied the snippet included below from one of my job submission scripts. You can change requested time, nodes, and cores (depending on the cluster) by editing the --time, -N, and --ntasks-per-node lines, respectively. Place this at the top of your batch script.

#!/bin/bash

##########################
# Set up submit to ....
#SBATCH -J Test
#SBATCH --time=12:00:00
#SBATCH -N 1
#SBATCH --ntasks-per-node 24
#SBATCH -o test-%j.out
##########################

Hope this helps,

-C

[Bashar]

Originally Posted by clapointe

Hi Bashar,



I copied the snippet included below from one of my job submission scripts. You can change requested time, nodes, and cores (depending on the cluster) by editing the --time, -N, and --ntasks-per-node lines, respectively. Place this at the top of your batch script.

#!/bin/bash



##########################

# Set up submit to ....

#SBATCH -J Test

#SBATCH --time=12:00:00

#SBATCH -N 1

#SBATCH --ntasks-per-node 24

#SBATCH -o test-%j.out

##########################

Hope this helps,



-C

Thank you C . Do you have full script file? It will be very helpful .Thank you again


Sent from my iPhone using CFD Online Forum mobile app

[clapointe]

Sure, it's included below. You might have different protocol for loading the openfoam environment, though.

#!/bin/bash

##########################
# Set up submit to ...
#SBATCH -J Test
#SBATCH --time=12:00:00
#SBATCH -N 1
#SBATCH --ntasks-per-node 24
#SBATCH -o test-%j.out
##########################

nodes=$SLURM_JOB_NUM_NODES            # Number of nodes
cores=24                      # number of cores

# LOAD THE OPENFOAM MODULE

ml load gcc openmpi/2.0.1

ml load openfoam/4.1

sed -i "/numberOfSubdomains/c\\numberOfSubdomains    $(($nodes*$cores));" system/decomposeParDict

# Decompose solution (serial)
echo "DECOMPOSE MESH WITH decomposePar..."
decomposePar -cellDist > log.decomposeParDict 2>&1

# Run the solution (parallel)
echo "RUNNING THE SOLVER WITH fireFoam..."
mpirun -np $(($nodes*$cores)) fireFoam -parallel > log.fireFoam 2>&1

-C

[Bashar]

Originally Posted by clapointe

Sure, it's included below. You might have different protocol for loading the openfoam environment, though.

#!/bin/bash



##########################

# Set up submit to ...

#SBATCH -J Test

#SBATCH --time=12:00:00

#SBATCH -N 1

#SBATCH --ntasks-per-node 24

#SBATCH -o test-%j.out

##########################



nodes=$SLURM_JOB_NUM_NODES            # Number of nodes

cores=24                      # number of cores



# LOAD THE OPENFOAM MODULE



ml load gcc openmpi/2.0.1



ml load openfoam/4.1



sed -i "/numberOfSubdomains/c\\numberOfSubdomains    $(($nodes*$cores));" system/decomposeParDict



# Decompose solution (serial)

echo "DECOMPOSE MESH WITH decomposePar..."

decomposePar -cellDist > log.decomposeParDict 2>&1



# Run the solution (parallel)

echo "RUNNING THE SOLVER WITH fireFoam..."

mpirun -np $(($nodes*$cores)) fireFoam -parallel > log.fireFoam 2>&1

-C

Thanks a lot! Yes , I have different protocol , but this is very helpful. I built my own script and I will compare it against yours to see if there is any thing that I didn't take care of.
Thank you .
Bashar


Sent from my iPhone using CFD Online Forum mobile app

[SymplPilot]

Hi everyone,

Our cluster uses the SLURM batch system also. However, the "srun" command is recommended to run all jobs. For example, to run a Foam application on 16 cores one has to use something like:

srun -n 16 xxxFoam -parallel

This is a problem, since OpenFOAM expects to see:

mpirun -np 16 xxxFoam -parallel

I wonder if anybody knows how to fix OpenFOAM to run with "srun" and "-n" as opposed to "mpirun" and "-np" ? Thanks for the help.

-SP

[clapointe]

I think I've used srun for interactive jobs before -- it's a way to ask for resources so that you can then run commands yourself.

Submitting a job on a slurm-managed cluster, though, requires a submission script similar to the one I've posted previously.

Caelan

[SymplPilot]

Hi Caelan,

Thanks, for a quick reply. I understand that all #SBATCH commands to allocate the required resources should be specified properly in a script. My question is about the last command in a script - the run command itself. Note, in the script you posted you used "mpirun" with "-np ".

For interactive jobs, one usually allocates the resources with "salloc" first, then uses "srun" to run a job, like in "srun -n 16 ./jobname". Not sure if "srun" will recognize "-np", it is "-n" only.

-SP

[clapointe]

srun automatically allocates the resources. If you want to work via an interactive job, use this command:

srun -N <insert number of nodes here> --ntasks-per-node=<insert number of cores here> --pty bash

this will get you an interactive job on N*ntasks cores. then, you can use mpirun as normal:

mpirun -np N*ntasks <insert solver here> -parallel

Caelan

[SymplPilot]

Thanks for the suggestions. I agree "srun" can be used to allocate resources (under circumstances). However, its main purpose to launch parallel jobs in SLURM. I wonder if OF would work with "srun" as opposed to "mpirun". For example, is this command OK ? :

srun -n #cores Foam_Solver -parallel ?

Note, in your "allocation" example "srun" is still launching a parallel job -- bash. Perhaps, it is not a good way to allocate resources for interactive jobs in comparison with "salloc".

-SF

[clapointe]

I tested the previous series of commands and was successful in submitting the job/getting the resources and running openFoam in parallel. The "--pty bash" is intentional, giving you the opportunity to use "mpirun" in the form of an interactive job.

If this is not your intention, srun is not required and the script included in a previous post will work.

[AshaEgreck]

Originally Posted by SymplPilot

Thanks for the suggestions. I agree "srun" can be used to allocate resources (under circumstances). However, its main purpose to launch parallel jobs in SLURM. I wonder if OF would work with "srun" as opposed to "mpirun". For example, is this command OK ? :

srun -n #cores Foam_Solver -parallel ?

Note, in your "allocation" example "srun" is still launching a parallel job -- bash. Perhaps, it is not a good way to allocate resources for interactive jobs in comparison with "salloc".

-SF

Thanks for the help. Following the guidance from https://www.open-mpi.org/faq/?catego...slurm-run-jobs I made a file called my_mpi_application with the following :

#!/bin/sh
caseName=${PWD##*/}
mpirun --oversubscribe -np 4 buoyantBoussinesqPimpleFoam -parallel > log.out_$caseName

Then I made a submit.sh file with the following :

#!/bin/bash
# Submission script for Cluster
#SBATCH --job-name=boussForooz
#SBATCH --time=01:00:00 # hh:mm:ss
#
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=2625 # megabytes
#SBATCH --partition=debug
#
#SBATCH --comment=openFOAM
export OMP_NUM_THREADS=4
export MKL_NUM_THREADS=4

blockMesh
setFields
decomposePar

srun -n 1 my_mpi_application

Then I run the following from the terminal

sbatch submit.sh

I needed to add the --oversubscribe after mpirun (NB I don't really know the implications of this but it worked).

Hopefully that will help someone some day.

[losiola]

Hello,

I am having an issue runing my jobs in slurm ,


once i submit my script using (sbatch job.sh ) i get a message that my job is submitted and by using squeue i can see that my job is runing on the cluster but unfortunatlly i cant find any results once the job is done , my case file is always still like on the begining and i dont find any of time directories (I use openFoam ).


Can you please help me with this issue?


this is the script i submit

#!/bin/bash
#SBATCH --partition=A
#SBATCH --account=AA
#SBATCH -N 1
#SBATCH --tasks-per-node=12
#SBATCH --job-name=losiola
#SBATCH -t 00:30:00
#SBATCH --mem=18GB
#SBATCH --output=log-%j.out
#SBATCH --error=log-%j.err
nodes=$SLURM_JOB_NEM_NODES
cores=12
module load openfoam/4.1 
module load openmpi/1.6.5
module load gcc/4.7.2 
export OMP_NUM_THREADS=12
export MKL_NUM_THREADS=12
decomposePar -cellDist > log.decomposeParDict 2>&1 
mpirun -np 12 simpleFoam -parallel>log.simpleFoam 2>&1

[clapointe]

Fair warning -- I have not used slurm in a long while. That said, I have two immediate thoughts : are you sure the job is running (not just submitted and in the queue), and unless you have already made the mesh, you will need to run e.g. blockMesh before decomposePar.

Caelan

[losiola]

Originally Posted by clapointe

Fair warning -- I have not used slurm in a long while. That said, I have two immediate thoughts : are you sure the job is running (not just submitted and in the queue), and unless you have already made the mesh, you will need to run e.g. blockMesh before decomposePar.

Caelan

Hello,


Yeah i am sure that the job is runing because it's status is R and i can see runTime increase

[Sgvaa]

Hi, I have to run multiple sbatch slurm scripts for cluster. Say, I have 50 sbatch files and I am running them sequentially in terminal (am using Ubundu) as follows.
sbatch file1.sbatch
sbatch file2.sbatch .
.
.
sbatch file50.sbatch

I want to simplify this 50 different commands to run in single command. As I am new working with terminal as well as cluster, I really donno how to approach this problem. Kindly, suggest me some solution to perform this action (I guess that I need to use some for loop statements, but of which syntax is my doubt). Am completely confused, some relevant documents might also be helpful.

Thank you.

[clapointe]

If they are to be run sequentially, then you can combine them to form one (long) script. Or write a script that calls the others. In pseudo code :

slurm stuff here

load modules

source script 1
source script 2
source script 3
...
source script 50

Caelan

[obiscolly50]

Originally Posted by Sgvaa

Hi, I have to run multiple sbatch slurm scripts for cluster. Say, I have 50 sbatch files and I am running them sequentially in terminal (am using Ubundu) as follows.
sbatch file1.sbatch
sbatch file2.sbatch .
.
.
sbatch file50.sbatch

I want to simplify this 50 different commands to run in single command. As I am new working with terminal as well as cluster, I really donno how to approach this problem. Kindly, suggest me some solution to perform this action (I guess that I need to use some for loop statements, but of which syntax is my doubt). Am completely confused, some relevant documents might also be helpful.

Thank you.

Try this in a script:

#!/bin/bash
for i in {1..50}
do
   sbatch file$i.sbatch
done

[Sgvaa]

Originally Posted by obiscolly50

Try this in a script:

#!/bin/bash
for i in {1..50}
do
   sbatch file$i.sbatch
done

I tried this code as follows for just three files..

#!/bin/bash
for i in {1..3}
do
sbatch layer$i.sbatch
done

But, it didn't create as separate jobs I guess..Only single job is submitted as whole. So, this dint work for me.

$ ~/Marabou% sbatch call_sbatch.sbatch
Submitted batch job 4576049

Thanks.