CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > Siemens > STAR-CCM+

How to use SLURM to run STAR-CCM+ simulation

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   December 19, 2017, 10:17
Default How to use SLURM to run STAR-CCM+ simulation
  #1
New Member
 
Join Date: Mar 2017
Posts: 15
Rep Power: 5
calmdown911 is on a distinguished road
Hi,
I have a difficult problem which I hope someone would be so kind to help me out:
I need to use SLURM to run a STAR-CCM+ simulation. But I don't know how. I tried to write a script by myself. But the job could be submitted but will immediately failed with ExitCode 2.
Here is my script:
#!/bin/bash

#SBATCH --job-name 1218
#SBATCH --partition=DELL
#SBATCH --nodes=4
#SBATCH --ntasks=28
#SBATCH -n 84

#SBATCH -output=1218.out
#SBATCH -e error_flie.e

#SBATCH -t 12:30:00
#SBATCH --mail-type=all
#SBATCH --mail-user=abc@def.com

module load starccm/12.02
module load mpi

sim_file="1218.sim"
STARTMACRO="/home/MYCASE/run.java"

starccm+ 1999@abc.com -np 84 $1218.sim -batch $STARTMACRO -batch-report
calmdown911 is offline   Reply With Quote

Old   December 19, 2017, 13:39
Default
  #2
New Member
 
Paul Hancock
Join Date: Mar 2009
Location: Bellingham, WA
Posts: 28
Rep Power: 13
Paulh is on a distinguished road
Take a look in the STEVE-Portal. There is an article with a couple of script examples.
Paulh is offline   Reply With Quote

Old   December 19, 2017, 14:15
Default
  #3
New Member
 
Join Date: Mar 2017
Posts: 15
Rep Power: 5
calmdown911 is on a distinguished road
Quote:
Originally Posted by Paulh View Post
Take a look in the STEVE-Portal. There is an article with a couple of script examples.
I submitted that question and the scripts seems not working.
calmdown911 is offline   Reply With Quote

Old   December 20, 2017, 12:22
Default
  #4
Senior Member
 
Join Date: Nov 2010
Location: USA
Posts: 1,228
Rep Power: 20
me3840 is on a distinguished road
Please post the complete error the scheduler is writing out.
me3840 is offline   Reply With Quote

Old   December 21, 2017, 10:29
Default
  #5
New Member
 
Join Date: Mar 2017
Posts: 15
Rep Power: 5
calmdown911 is on a distinguished road
SLURM Job_id=56375 Name=1214 Failed, Run time 00:00:00, FAILED, ExitCode 2
calmdown911 is offline   Reply With Quote

Old   December 21, 2017, 14:40
Default
  #6
New Member
 
Join Date: Mar 2017
Posts: 15
Rep Power: 5
calmdown911 is on a distinguished road
Quote:
Originally Posted by me3840 View Post
Please post the complete error the scheduler is writing out.
The complete error message is as below:

SLURM Job_id=56375 Name=1214 Failed, Run time 00:00:00, FAILED, ExitCode 2

Thank you so much for your help!
calmdown911 is offline   Reply With Quote

Old   December 22, 2017, 11:51
Default
  #7
New Member
 
Andre C
Join Date: Nov 2013
Posts: 9
Rep Power: 8
spartan42 is on a distinguished road
Quote:
Originally Posted by calmdown911 View Post
The complete error message is as below:

SLURM Job_id=56375 Name=1214 Failed, Run time 00:00:00, FAILED, ExitCode 2

Thank you so much for your help!

Is that the error given to you by the output file? [1218.out]
spartan42 is offline   Reply With Quote

Old   December 22, 2017, 12:03
Default
  #8
New Member
 
Andre C
Join Date: Nov 2013
Posts: 9
Rep Power: 8
spartan42 is on a distinguished road
Quote:
Originally Posted by calmdown911 View Post
The complete error message is as below:

SLURM Job_id=56375 Name=1214 Failed, Run time 00:00:00, FAILED, ExitCode 2

Thank you so much for your help!
Maybe you can benefit out of my slurm file:

#!/bin/sh
#SBATCH --account= [your account]

#SBATCH --time=20:00:00
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=28
#SBATCH --mem-per-cpu=1990
#SBATCH --job-name= [job namr]

#SBATCH --error=Job_Results_ERROR-%J.err
#SBATCH --output=Job_Results.out

#SBATCH --export CDLMD_LICENSE_FILE = [1999@ your license]
export CDLMD_LICENSE_FILE=[1999@ your license]

## Simulation File: ====> Source
STARSOURCEFILE=[simulation name.sim]

## SLURM: ====> Job Node List (DO NOT MODIFY)
echo "Slurm nodes assigned :$SLURM_JOB_NODELIST"
echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES
echo "SLURMTMPDIR="$SLURMTMPDIR
echo "working directory = "$SLURM_SUBMIT_DIR
echo "SLURM_NTASKS="$SLURM_NTASKS


## Modules: ====> Loading desired mudule (DO NOT MODIFY)
module load starccm/starccm-10.02.012


## Working Directory: ====> Job Node List (DO NOT MODIFY)
cd $SLURM_SUBMIT_DIR
scontrol show hostname $SLURM_NODELIST | tr 'ec' 'ic'> machinefile_${SLURM_JOB_ID}

starccm+ -batch -np $SLURM_NTASKS -rsh ssh -machinefile machinefile_${SLURM_JOB_ID} -power $STARSOURCEFILE &>> job_$SLURM_JOB_ID.log

sleep 3

##rm $hostfile
echo "Run Finished" >> job_$SLURM_JOB_ID.log

Last edited by spartan42; December 25, 2017 at 11:30.
spartan42 is offline   Reply With Quote

Old   December 23, 2017, 10:08
Default
  #9
New Member
 
Join Date: Mar 2017
Posts: 15
Rep Power: 5
calmdown911 is on a distinguished road
Quote:
Originally Posted by spartan42 View Post
Maybe you can benefit out of my slurm file:

#!/bin/sh
#SBATCH --account= [your account]

#SBATCH --time=20:00:00
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=28
#SBATCH --mem-per-cpu=1990
#SBATCH --job-name= [job namr]

#SBATCH --error=Job_Results_ERROR-%J.err
#SBATCH --output=Job_Results.out

#SBATCH --export CDLMD_LICENSE_FILE=[1999@ your license]

export CDLMD_LICENSE_FILE=[1999@ your license]

## Simulation File: ====> Source
STARSOURCEFILE=[simulation name.sim]

## SLURM: ====> Job Node List (DO NOT MODIFY)
echo "Slurm nodes assigned :$SLURM_JOB_NODELIST"
echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES
echo "SLURMTMPDIR="$SLURMTMPDIR
echo "working directory = "$SLURM_SUBMIT_DIR
echo "SLURM_NTASKS="$SLURM_NTASKS


## Modules: ====> Loading desired mudule (DO NOT MODIFY)
module load starccm/starccm-10.02.012


## Working Directory: ====> Job Node List (DO NOT MODIFY)
cd $SLURM_SUBMIT_DIR
scontrol show hostname $SLURM_NODELIST | tr 'ec' 'ic'> machinefile_${SLURM_JOB_ID}

starccm+ -batch -np $SLURM_NTASKS -rsh ssh -machinefile machinefile_${SLURM_JOB_ID} -power $STARSOURCEFILE &>> job_$SLURM_JOB_ID.log

sleep 3

##rm $hostfile
echo "Run Finished" >> job_$SLURM_JOB_ID.log
Thank you so much! I'll try your script. Hope it works!
Merry Christmas!

Last edited by calmdown911; December 25, 2017 at 11:39.
calmdown911 is offline   Reply With Quote

Old   December 23, 2017, 14:36
Default
  #10
New Member
 
Join Date: Mar 2017
Posts: 15
Rep Power: 5
calmdown911 is on a distinguished road
Quote:
Originally Posted by spartan42 View Post
Is that the error given to you by the output file? [1218.out]
The error is send by email.
calmdown911 is offline   Reply With Quote

Old   December 23, 2017, 14:39
Default
  #11
New Member
 
Andre C
Join Date: Nov 2013
Posts: 9
Rep Power: 8
spartan42 is on a distinguished road
tmas to you as well!
spartan42 is offline   Reply With Quote

Old   December 23, 2017, 14:41
Default
  #12
New Member
 
Andre C
Join Date: Nov 2013
Posts: 9
Rep Power: 8
spartan42 is on a distinguished road
Quote:
Originally Posted by calmdown911 View Post
The error is send by email.
I hope it works. Let me know if you need some help.

Merry Christmas to you as well!
spartan42 is offline   Reply With Quote

Old   December 24, 2017, 11:53
Default
  #13
New Member
 
Join Date: Mar 2017
Posts: 15
Rep Power: 5
calmdown911 is on a distinguished road
Quote:
Originally Posted by spartan42 View Post
I hope it works. Let me know if you need some help.

Merry Christmas to you as well!
Hi, Spartan42,
I tried your scripts and got the error as below:
ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'starccm'
What shall I do for this kind of error?
Thank you so much for your help!
calmdown911 is offline   Reply With Quote

Old   December 24, 2017, 12:01
Default
  #14
New Member
 
Andre C
Join Date: Nov 2013
Posts: 9
Rep Power: 8
spartan42 is on a distinguished road
Quote:
Originally Posted by calmdown911 View Post
Hi, Spartan42,
I tried your scripts and got the error as below:
ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'starccm'
What shall I do for this kind of error?
Thank you so much for your help!
If you look at the script/ the section on the bottom is loading the module for starccm on the server. You need to find out the file directory.

I imagine your using some sort of command/ or putty. Correct?

In the command promp you can type [module avail] and it should populate a list with all the available modules on the server. From there you can search for the starccm module and replace the red part with yours


## Modules: ====> Loading desired mudule (DO NOT MODIFY)

module load starccm/starccm-10.02.012

Last edited by spartan42; December 24, 2017 at 12:04. Reason: Wrong command:
spartan42 is offline   Reply With Quote

Old   December 25, 2017, 09:40
Default
  #15
New Member
 
Join Date: Mar 2017
Posts: 15
Rep Power: 5
calmdown911 is on a distinguished road
Quote:
Originally Posted by spartan42 View Post
If you look at the script/ the section on the bottom is loading the module for starccm on the server. You need to find out the file directory.

I imagine your using some sort of command/ or putty. Correct?

In the command promp you can type [module avail] and it should populate a list with all the available modules on the server. From there you can search for the starccm module and replace the red part with yours


## Modules: ====> Loading desired mudule (DO NOT MODIFY)

module load starccm/starccm-10.02.012
Hi, Spartan42,
It seems I finally got the simulation running on our cluster. Yet I came across another problem. I activated autosave when I built the case. Yet I cannot get the calculation result. The case file is exactly same before and after the calculation. The residual is recorded in the output file. Yet the result seems lost. What shall I do to get the calculation result from SLURM?
calmdown911 is offline   Reply With Quote

Old   December 25, 2017, 09:55
Default
  #16
New Member
 
Andre C
Join Date: Nov 2013
Posts: 9
Rep Power: 8
spartan42 is on a distinguished road
Quote:
Originally Posted by calmdown911 View Post
Hi, Spartan42,
It seems I finally got the simulation running on our cluster. Yet I came across another problem. I activated autosave when I built the case. Yet I cannot get the calculation result. The case file is exactly same before and after the calculation. The residual is recorded in the output file. Yet the result seems lost. What shall I do to get the calculation result from SLURM?
Can you elaborate a little more?

This is what I got:

You set auto save every x amount of iterations? But its not saving the new sim file correct?

You are getting the residuals from the output file because they are the only active reports in your simulation. If you create other reports and graphs you will see those results in the output file.

When it comes to simulation saving/ - you need to set up the autosave for every x iteration - might be not set up correctly and that is why your not seeing anything. If you getting residuals it means the simulation is running. Maybe take a snip of your autosave setting so i can look at it.
spartan42 is offline   Reply With Quote

Old   December 25, 2017, 11:42
Default
  #17
New Member
 
Join Date: Mar 2017
Posts: 15
Rep Power: 5
calmdown911 is on a distinguished road
Quote:
Originally Posted by spartan42 View Post
Can you elaborate a little more?

This is what I got:

You set auto save every x amount of iterations? But its not saving the new sim file correct?

You are getting the residuals from the output file because they are the only active reports in your simulation. If you create other reports and graphs you will see those results in the output file.

When it comes to simulation saving/ - you need to set up the autosave for every x iteration - might be not set up correctly and that is why your not seeing anything. If you getting residuals it means the simulation is running. Maybe take a snip of your autosave setting so i can look at it.
My autosaving is as attachment. Is it right?
Attached Images
File Type: jpg autosave.jpg (60.5 KB, 19 views)
calmdown911 is offline   Reply With Quote

Old   July 1, 2020, 04:18
Default
  #18
Senior Member
 
MA
Join Date: Mar 2020
Posts: 122
Rep Power: 2
mazhar16823 is on a distinguished road
Quote:
Originally Posted by calmdown911 View Post
Thank you so much! I'll try your script. Hope it works!
Merry Christmas!



Hi,


I have used this script but it gives me an error "invalid argument" for my account. Why is this happening?
mazhar16823 is offline   Reply With Quote

Old   July 1, 2020, 10:00
Default
  #19
New Member
 
KimberlyJames's Avatar
 
Kimberly James
Join Date: Jun 2020
Location: Sydney
Posts: 2
Rep Power: 0
KimberlyJames is on a distinguished road
I'm also trying to figure it out, thank you for your suggestions.
KimberlyJames is offline   Reply With Quote

Old   July 11, 2020, 19:34
Default
  #20
New Member
 
Erik Lönroth
Join Date: Jul 2020
Location: Sweden
Posts: 3
Rep Power: 2
erik_lonroth is on a distinguished road
Perhaps this could be of help for you aswell https://eriklonroth.com/2020/07/11/r...nd-infiniband/
erik_lonroth is offline   Reply With Quote

Reply

Tags
clusters, mpi, slurm, star-ccm+

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Marine Propeller Hydroacoustic Simulation in Star CCM BlackEagle STAR-CCM+ 1 December 8, 2017 21:33
abaqus star ccm co simulation Mustafalboro STAR-CCM+ 0 July 25, 2017 16:13
Run STAR CCM+ with -power by default Jorgror STAR-CCM+ 2 February 16, 2016 08:46
Thermal Comfort Simulation in STAR CCM+ anupmu STAR-CCM+ 1 February 27, 2013 14:25
Procedure to run unsteady simulation? STN Main CFD Forum 2 February 16, 2002 04:37


All times are GMT -4. The time now is 01:24.