CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM

Restarting a simulation using latestTime in HPC

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree4Likes
  • 1 Post By clapointe
  • 1 Post By HPE
  • 1 Post By HPE
  • 1 Post By HPE

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   April 7, 2020, 11:57
Default Restarting a simulation using latestTime in HPC
  #1
New Member
 
hayat EL BAAMRANI
Join Date: Nov 2017
Posts: 9
Rep Power: 4
hayat elb is on a distinguished road
Hello,
I was running a simulation on HPC until a given period, but I found that the steady state was not reached. I would like to start the simulations from the point where I stopped as starting value . I was trying to implement this by changing the start value in the controlDict file, but I couldn't be of help. Could someone tell me how to restart a simulation

The script is illustrated below:

#!/bin/bash
#SBATCH -J OpenFoam
#SBATCH -N 1
#SBATCH --ntasks=16
#SBATCH --ntasks-per-core=1
#SBATCH --partition=longq
#SBATCH -o /data/%x-%j.out
#SBATCH -e /data/%x-%j.err

module load openmpi/gcc/64/1.10.7
module load boost/gcc/64
module load qt/gcc/64

module load OpenFoam/openmpi/gcc/64/v1906
source $OPENFOAM_BASH
export WORK_DIR=/data/$USER_${SLURM_JOB_ID}
export INPUT_DIR=/home/$USER/OpenFoam
mkdir -p $WORK_DIR
cp -r $INPUT_DIR/* $WORK_DIR
cd $WORK_DIR


echo "Lancement de OpenFoam with $SLURM_NTASKS at $WORK_DIR"


decomposePar
mpirun -np $SLURM_NTASKS icoReactingMultiphaseInterFoam -parallel
reconstructPar
echo " Termine "

I would appreciate if you guys can throw your suggestions.

Thanks
Hayat
hayat elb is offline   Reply With Quote

Old   April 7, 2020, 18:29
Default
  #2
Senior Member
 
Join Date: Aug 2015
Posts: 393
Rep Power: 9
clapointe is on a distinguished road
To restart from your latest simulation point, make sure startFrom is latestTime in controlDict. Also note that if your simulation is already decomposed, you will not need to decompose it again upon restart.

Caelan
hayat elb likes this.
clapointe is offline   Reply With Quote

Old   April 7, 2020, 19:17
Default
  #3
New Member
 
hayat EL BAAMRANI
Join Date: Nov 2017
Posts: 9
Rep Power: 4
hayat elb is on a distinguished road
Quote:
Originally Posted by clapointe View Post
To restart from your latest simulation point, make sure startFrom is latestTime in controlDict. Also note that if your simulation is already decomposed, you will not need to decompose it again upon restart.

Caelan
thank you for your reply
I did that but it doesn't work
hayat elb is offline   Reply With Quote

Old   April 11, 2020, 17:20
Default
  #4
HPE
Senior Member
 
Herpes Free Engineer
Join Date: Sep 2019
Location: The Home Under The Ground with the Lost Boys
Posts: 583
Rep Power: 5
HPE is on a distinguished road
can you share the `controlDict` please?
hayat elb likes this.
HPE is online now   Reply With Quote

Old   April 11, 2020, 17:52
Default
  #5
New Member
 
hayat EL BAAMRANI
Join Date: Nov 2017
Posts: 9
Rep Power: 4
hayat elb is on a distinguished road
Quote:
Originally Posted by HPE View Post
can you share the `controlDict` please?
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: v1906 |
| \\ / A nd | Web: www.OpenFOAM.com |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

application icoReactingMultiphaseInterFoam;

startFrom latestTime;

startTime 0;

stopAt endTime;

endTime 3600;

deltaT 0.00001;

writeControl adjustableRunTime;

writeInterval 1;

purgeWrite 0;

writeFormat ascii;

writePrecision 6;

writeCompression off;

timeFormat general;

timePrecision 6;

runTimeModifiable yes;
adjustTimeStep yes;

maxDeltaT 2;

maxCo 1;
maxAlphaCo 1.5;//maxAlphaCo 1.5;
maxAlphaDdt 1;//maxAlphaDdt 1;
hayat elb is offline   Reply With Quote

Old   April 12, 2020, 06:46
Default
  #6
HPE
Senior Member
 
Herpes Free Engineer
Join Date: Sep 2019
Location: The Home Under The Ground with the Lost Boys
Posts: 583
Rep Power: 5
HPE is on a distinguished road
The `controlDict` seems fine.

In the script, you `decompose` the given case. Remove `decomp` and `reconst` from the script, just execute them once per simulation. Keep `mpirun ..`. An already parallelised case is not needed to be reparallelised or smt like that. May be you decompose the same time step at every execution of the script?
hayat elb likes this.
HPE is online now   Reply With Quote

Old   April 12, 2020, 18:16
Default
  #7
New Member
 
hayat EL BAAMRANI
Join Date: Nov 2017
Posts: 9
Rep Power: 4
hayat elb is on a distinguished road
Quote:
Originally Posted by HPE View Post
The `controlDict` seems fine.

In the script, you `decompose` the given case. Remove `decomp` and `reconst` from the script, just execute them once per simulation. Keep `mpirun ..`. An already parallelised case is not needed to be reparallelised or smt like that. May be you decompose the same time step at every execution of the script?
thank you for your reply
I have been making changes in my script(remove decomp and reconst) but the simulation is not running and giving me this error message
[node05][[31961,1],9][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[node05][[31961,1],5][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[node05][[31961,1],2][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[node05][[31961,1],7][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[node05][[31961,1],4][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[node05][[31961,1],6][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[node05][[31961,1],8][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[node05][[31961,1],0][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[node05][[31961,1],3][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[node05][[31961,1],1][btl_openib.c:880:mca_btl_openib_add_procs] could not prepare openib device for use
[8] #0 Foam::error:rintStack(Foam::Ostream&) at ??:?
[8] #1 Foam::sigFpe::sigHandler(int) at ??:?
[8] #2 ? in /lib64/libc.so.6
[8] #3 Foam::divide(Foam::Field<double>&, Foam::UList<double> const&, Foam::UList<double> const&) at ??:?
[8] #4 void Foam::divide<Foam::fvPatchField, Foam::volMesh>(Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh>&, Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> const&, Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> const&) at ??:?
[8] #5 Foam::tmp<Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> > Foam:perator/<Foam::fvPatchField, Foam::volMesh>(Foam::tmp<Foam::GeometricField<doub le, Foam::fvPatchField, Foam::volMesh> > const&, Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> const&) at ??:?
[8] #6 Foam::MultiComponentPhaseModel<Foam:haseModel, Foam::rhoReactionThermo>::calculateMassFractions() at ??:?
[8] #7 Foam::MultiComponentPhaseModel<Foam:haseModel, Foam::rhoReactionThermo>::solveYi(Foam::PtrList<Fo am:imensionedField<double, Foam::volMesh> >&, Foam::PtrList<Foam:imensionedField<double, Foam::volMesh> >&) at ??:?
[8] #8 ? at ??:?
[8] #9 __libc_start_main in /lib64/libc.so.6
[8] #10 ? at ??:?
[node05:116862] *** Process received signal ***
[node05:116862] Signal: Floating point exception (8)
[node05:116862] Signal code: (-6)
[node05:116862] Failing at address: 0x5240001c87e
[node05:116862] [ 0] /lib64/libc.so.6(+0x36280)[0x2aaab272f280]
[node05:116862] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2aaab272f207]
[node05:116862] [ 2] /lib64/libc.so.6(+0x36280)[0x2aaab272f280]
[node05:116862] [ 3] /data/shared/apps2/OpenFOAM/OpenFOAM-v1906/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam6divideERNS_5FieldIdEERKNS_ 5UListIdEES6_+0x115)[0x2aaab16ee705]
[node05:116862] [ 4] /data/shared/apps2/OpenFOAM/OpenFOAM-v1906/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam6divideINS_12fvPatchFie ldENS_7volMeshEEEvRNS_14GeometricFieldIdT_T0_EERKS 6_S9_+0xa8)[0x2aaaabf526e8]
[node05:116862] [ 5] /data/shared/apps2/OpenFOAM/OpenFOAM-v1906/platforms/linux64GccDPInt32Opt/lib/libturbulenceModels.so(_ZN4FoamdvINS_12fvPatchFiel dENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_ EEEERKS8_RKS7_+0x23b)[0x2aaaae6cf13b]
[node05:116862] [ 6] /data/shared/apps2/OpenFOAM/OpenFOAM-v1906/platforms/linux64GccDPInt32Opt/lib/libIncompressibleMultiphaseSystems.so(_ZN4Foam24Mu ltiComponentPhaseModelINS_10phaseModelENS_17rhoRea ctionThermoEE22calculateMassFractionsEv+0x507)[0x2aaaafc6b617]
[node05:116862] [ 7] /data/shared/apps2/OpenFOAM/OpenFOAM-v1906/platforms/linux64GccDPInt32Opt/lib/libIncompressibleMultiphaseSystems.so(_ZN4Foam24Mu ltiComponentPhaseModelINS_10phaseModelENS_17rhoRea ctionThermoEE7solveYiERNS_7PtrListINS_16Dimensione dFieldIdNS_7volMeshEEEEES9_+0x1b2d)[0x2aaaafc7d70d]
[node05:116862] [ 8] icoReactingMultiphaseInterFoam[0x436e9f]
[node05:116862] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaab271b3d5]
[node05:116862] [10] icoReactingMultiphaseInterFoam[0x43c1c2]
[node05:116862] *** End of error message ***
[node05:116823] 5 more processes have sent help message help-mpi-btl-openib.txt / init-fail-no-mem
[node05:116823] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
hayat elb is offline   Reply With Quote

Old   April 13, 2020, 09:24
Default
  #8
HPE
Senior Member
 
Herpes Free Engineer
Join Date: Sep 2019
Location: The Home Under The Ground with the Lost Boys
Posts: 583
Rep Power: 5
HPE is on a distinguished road
Have you decomposed your case before running the application?
hayat elb likes this.
HPE is online now   Reply With Quote

Old   April 13, 2020, 10:19
Default
  #9
New Member
 
hayat EL BAAMRANI
Join Date: Nov 2017
Posts: 9
Rep Power: 4
hayat elb is on a distinguished road
Quote:
Originally Posted by HPE View Post
Have you decomposed your case before running the application?
Yes I have decomposed the case before
Attached Images
File Type: png Capture.PNG (8.3 KB, 6 views)
hayat elb is offline   Reply With Quote

Old   April 13, 2020, 15:10
Default
  #10
HPE
Senior Member
 
Herpes Free Engineer
Join Date: Sep 2019
Location: The Home Under The Ground with the Lost Boys
Posts: 583
Rep Power: 5
HPE is on a distinguished road
Honestly, I really don't know what is going on based on the information you have shared so far.

Any chance to try to run it without using the script? On your desktop? I think it should be OK since you have 8 processors in use?
HPE is online now   Reply With Quote

Old   April 13, 2020, 18:10
Default
  #11
New Member
 
hayat EL BAAMRANI
Join Date: Nov 2017
Posts: 9
Rep Power: 4
hayat elb is on a distinguished road
Quote:
Originally Posted by HPE View Post
Honestly, I really don't know what is going on based on the information you have shared so far.

Any chance to try to run it without using the script? On your desktop? I think it should be OK since you have 8 processors in use?
I work on high performance cluster since I have enough CPU power, in the beginning the calculation starts well but to restart the simulation from the time where I stopped as the starting value I found this problem.

Thanks
Hayat
hayat elb is offline   Reply With Quote

Old   May 23, 2020, 13:43
Default
  #12
Senior Member
 
alberto
Join Date: Apr 2016
Location: Mexico
Posts: 102
Rep Power: 6
dewey is on a distinguished road
hi



I have a question, do you stop the simulation or the simulation ends alone?


Sometime when we stop the simulation, or ends by other reason, it does not finish writing the last timestep.



Something that you could probe is to delete the last timestep in each processor files, and then now start from the latestTime.
dewey is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Restart InterDymFoam simulation from latestTime Giovanni_Do OpenFOAM Running, Solving & CFD 2 April 11, 2019 05:22
OpenFoam restarting simulation, stops at first Time loop syntex OpenFOAM Running, Solving & CFD 4 April 5, 2019 11:20
Restarting a CFX simulation from last stop selvam2487 CFX 9 September 9, 2014 11:00
Convergence of jet flow simulation MiraLisa FLUENT 0 August 15, 2013 04:44
Problem restarting a simulation from an intermediate point. sergiorbrett CFX 2 April 21, 2010 17:11


All times are GMT -4. The time now is 08:44.