Setting up Lid driven Cavity Benchmark with 1M cells for multiple cores

puneet336 · April 6, 2019, 04:52

Hi,
I have a new processor, on which i am planning to carry out some benchmarks with openfoam-6. I am getting floating point exception while running the benchmark , so sharing the detailed setup procedure (which i followed) here -

I came across Lid Driven cavity benchmark numbers - here and here, So, here is how i setup the input for cavity 1M cells -

Code:

mkdir -p $FOAM_RUN 
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/icoFoam/cavity/cavity .
cd cavity

As i need to run the setup on multiple cores, as per instructions here i copied the decomposeParDict from damBreak case -

Code:

cp -r $FOAM_TUTORIALS/multiphase/interFoam/laminar/damBreak/damBreak/system/decomposeParDict system/

from Allrun script of cavity -

Code:

$FOAM_TUTORIALS/incompressible/icoFoam/cavity/Allrun

It seems for cavity case i need to run following in sequence for a single core setup -
a. blockMesh
b. icoFoam

1. i ran "blockMesh" utility, then as per this link in CFD Forum, it seems the default tutorial is for 400 (blocks - 20 x 20 x 1 ?) cells.

Code:

...
blocks
(
    hex (0 1 2 3 4 5 6 7) (20 20 1) simpleGrading (1 1 1)
);
...

So i modified aforementioned blocks "section" in system/blockMeshDict to -

Code:

...
blocks
(
    hex (0 1 2 3 4 5 6 7) (1000 1000 1) simpleGrading (1 1 1)
);
...

for 1000000 (1000 x 1000 -> 1M) cells before running blockMesh

here is the breif stdout from the run-

Code:

Creating block mesh topology

Check topology

        Basic statistics
                Number of internal faces : 0
                Number of boundary faces : 6
                Number of defined boundary faces : 6
                Number of undefined boundary faces : 0
        Checking patch -> block consistency

Creating block offsets
Creating merge list .

Creating polyMesh from blockMesh
Creating patches
Creating cells
Creating points with scale 0.1
    Block 0 cell size :
        i : 0.0001 .. 0.0001
        j : 0.0001 .. 0.0001
        k : 0.01 .. 0.01


Writing polyMesh
----------------
Mesh Information
----------------
  boundingBox: (0 0 0) (0.1 0.1 0.01)
  nPoints: 2004002
  nCells: 1000000
  nFaces: 4002000
  nInternalFaces: 1998000
----------------
Patches
----------------
  patch 0 (start: 1998000 size: 1000) name: movingWall
  patch 1 (start: 1999000 size: 3000) name: fixedWalls
  patch 2 (start: 2002000 size: 2000000) name: frontAndBack

End

2. As i need to run this case on multiple cores (16 for a start), i ran - decomposePar utility after following modification in system/decomposeParDict -

Code:

numberOfSubdomains 16;
...
simpleCoeffs
{
    n               (4 4 1);
    delta           0.001;
}

changed simpleCoeffs as with original n ( 2 2 1) i was getting -

Code:

--> FOAM FATAL ERROR:
Wrong number of processor divisions in geomDecomp:
Number of domains    : 16
Wanted decomposition : (2 2 1)

So, with n ( 4 4 1) , the stdout of decomposePar was -

Code:

.....
Processor 14
    Number of cells = 62500
    Number of faces shared with processor 10 = 250
    Number of faces shared with processor 13 = 250
    Number of faces shared with processor 15 = 250
    Number of processor patches = 3
    Number of processor faces = 750
    Number of boundary faces = 125250
....

3. Finally, i ran icoFoam as -
time mpirun -np 16 icoFoam -parallel 2>&1|tee logs_run

the simulation crashed with following error message in 18 seconds, here are some error messages -

Code:

Starting time loop

Time = 0.005

Courant Number mean: 0 max: 0
smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 0.00680461, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.21807, No Iterations 1000
time step continuity errors : sum local = 0.000782014, global = 1.19114e-18, cumulative = 1.19114e-18
DICPCG:  Solving for p, Initial residual = 0.953973, Final residual = 0.074792, No Iterations 1000
time step continuity errors : sum local = 0.000371379, global = -3.79841e-19, cumulative = 8.11299e-19
ExecutionTime = 6.05 s  ClockTime = 6 s

Time = 0.01

Courant Number mean: 3.27516 max: 47.7626
smoothSolver:  Solving for Ux, Initial residual = 0.204174, Final residual = 0.092896, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.970606, Final residual = 0.613291, No Iterations 1000
DICPCG:  Solving for p, Initial residual = 0.998811, Final residual = 0.905489, No Iterations 1000
time step continuity errors : sum local = 0.469862, global = 2.03288e-17, cumulative = 2.11401e-17
DICPCG:  Solving for p, Initial residual = 0.735387, Final residual = 0.418723, No Iterations 1000
time step continuity errors : sum local = 0.497166, global = -5.45489e-17, cumulative = -3.34088e-17
ExecutionTime = 12.35 s  ClockTime = 13 s

Time = 0.015

Courant Number mean: 19.9394 max: 1597.49
smoothSolver:  Solving for Ux, Initial residual = 0.98995, Final residual = 0.732227, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.976188, Final residual = 0.558427, No Iterations 1000
DICPCG:  Solving for p, Initial residual = 0.998597, Final residual = 1.7475, No Iterations 1000
time step continuity errors : sum local = 623.502, global = 3.51108e-13, cumulative = 3.51075e-13
DICPCG:  Solving for p, Initial residual = 0.843665, Final residual = 0.0584327, No Iterations 1000
time step continuity errors : sum local = 66.8301, global = 2.84495e-14, cumulative = 3.79524e-13
ExecutionTime = 18.65 s  ClockTime = 19 s

Time = 0.02

Courant Number mean: 8657.74 max: 935300
[13] #0  Foam::error::printStack(Foam::Ostream&)[14] #0  Foam::error::printStack(Foam::Ostream&)[0] #0  Foam::error::printStack(Foam::Ostream&)--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[37276,1],13] (PID 186782)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
 at ??:?
 at ??:?
 at ??:?
[13] #1  [14] #1  [0] #1  Foam::sigFpe::sigHandler(int)Foam::sigFpe::sigHandler(int)Foam::sigFpe::sigHandler(int) at ??:?
[0] #2  ? at ??:?
[13] #2  ? at ??:?
[14] #2  ? in "/lib64/libc.so.6"
[0] #3  Foam::symGaussSeidelSmoother::smooth(Foam::word const&, Foam::Field<double>&, Foam::lduMatrix const&, Foam::Field<double> const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char, int) in "/lib64/libc.so.6"
[13] #3  Foam::symGaussSeidelSmoother::smooth(Foam::word const&, Foam::Field<double>&, Foam::lduMatrix const&, Foam::Field<double> const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char, int) in "/lib64/libc.so.6"
[14] #3  Foam::symGaussSeidelSmoother::smooth(Foam::word const&, Foam::Field<double>&, Foam::lduMatrix const&, Foam::Field<double> const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char, int) at ??:?
[0] #4  Foam::symGaussSeidelSmoother::smooth(Foam::Field<double>&, Foam::Field<double> const&, unsigned char, int) const at ??:?
[13] #4  Foam::symGaussSeidelSmoother::smooth(Foam::Field<double>&, Foam::Field<double> const&, unsigned char, int) const at ??:?
[14] #4  Foam::symGaussSeidelSmoother::smooth(Foam::Field<double>&, Foam::Field<double> const&, unsigned char, int) const at ??:?
[0] #5  Foam::smoothSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[13] #5  Foam::smoothSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[14] #5  Foam::smoothSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[0] #6   at ??:?
[13] #6   at ??:?
[14] #6  ??? at ??:?
[0] #7   at ??:?
[13] #7   at ??:?
[14] #7  ??? at ??:?
[0] #8   at ??:?
[13] #8   at ??:?
[14] #8  ??? at ??:?
[0] #9   at ??:?
[13] #9   at ??:?
[14] #9  ??? at ??:?
[0] #10  __libc_start_main at ??:?
[13] #10  __libc_start_main at ??:?
[14] #10  __libc_start_main in "/lib64/libc.so.6"
[0] #11   in "/lib64/libc.so.6"
[14] #11   in "/lib64/libc.so.6"
[13] #11  ??? at ??:?
[skl035:186769] *** Process received signal ***
[skl035:186769] Signal: Floating point exception (8)
[skl035:186769] Signal code:  (-6)
[skl035:186769] Failing at address: 0x4110002d991
[skl035:186769] [ 0] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[skl035:186769] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2aaaae7851f7]
[skl035:186769] [ 2] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[skl035:186769] [ 3] /home/puneet/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam22symGaussSeidelSmoother6smoothERKNS_4wordERNS_5FieldIdEERKNS_9lduMatrixERKS5_RKNS_10FieldFieldIS4_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEhi+0x3cc)[0x2aaaad69d64c]
[skl035:186769] [ 4] /home/puneet/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam22symGaussSeidelSmoother6smoothERNS_5FieldIdEERKS2_hi+0x2d)[0x2aaaad69d92d]
[skl035:186769] [ 5] /home/puneet/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam12smoothSolver5solveERNS_5FieldIdEERKS2_h+0x377)[0x2aaaad6963f7]
[skl035:186769] [ 6] icoFoam[0x445819]
[skl035:186769] [ 7] icoFoam[0x44be5c]
[skl035:186769] [ 8] icoFoam[0x44c179]
[skl035:186769] [ 9] icoFoam[0x41c161]
[skl035:186769] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaaae771c05]
[skl035:186769] [11] icoFoam[0x41cba7]
[skl035:186769] *** End of error message ***
 at ??:?
[skl035:186783] *** Process received signal ***
[skl035:186783] Signal: Floating point exception (8)
[skl035:186783] Signal code:  (-6)
[skl035:186783] Failing at address: 0x4110002d99f
[skl035:186783] [ 0] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[skl035:186783] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2aaaae7851f7]
[skl035:186783] [ 2] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[skl035:186783] [ 3]  at ??:?
[skl035:186782] *** Process received signal ***
[skl035:186782] Signal: Floating point exception (8)
[skl035:186782] Signal code:  (-6)
[skl035:186782] Failing at address: 0x4110002d99e
[skl035:186782] [ 0] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[skl035:186782] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2aaaae7851f7]
[skl035:186782] [ 2] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[skl035:186782] [ 3] /home/puneet/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam22symGaussSeidelSmoother6smoothERKNS_4wordERNS_5FieldIdEERKNS_9lduMatrixERKS5_RKNS_10FieldFieldIS4_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEhi+0x3cc)[0x2aaaad69d64c]
[skl035:186783] [ 4] /home/puneet/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam22symGaussSeidelSmoother6smoothERKNS_4wordERNS_5FieldIdEERKNS_9lduMatrixERKS5_RKNS_10FieldFieldIS4_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEhi+0x2d5)[0x2aaaad69d555]
[skl035:186782] [ 4] /home/puneet/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam22symGaussSeidelSmoother6smoothERNS_5FieldIdEERKS2_hi+0x2d)[0x2aaaad69d92d]
[skl035:186783] [ 5] /home/puneet/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam22symGaussSeidelSmoother6smoothERNS_5FieldIdEERKS2_hi+0x2d)[0x2aaaad69d92d]
[skl035:186782] [ 5] /home/puneet/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam12smoothSolver5solveERNS_5FieldIdEERKS2_h+0x377)[0x2aaaad6963f7]
[skl035:186783] [ 6] icoFoam[0x445819]
[skl035:186783] [ 7] icoFoam[0x44be5c]
[skl035:186783] [ 8] icoFoam[0x44c179]
[skl035:186783] [ 9] icoFoam[0x41c161]
[skl035:186783] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaaae771c05]
[skl035:186783] [11] icoFoam[0x41cba7]
[skl035:186783] *** End of error message ***
/home/puneet/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam12smoothSolver5solveERNS_5FieldIdEERKS2_h+0x377)[0x2aaaad6963f7]
[skl035:186782] [ 6] icoFoam[0x445819]
[skl035:186782] [ 7] icoFoam[0x44be5c]
[skl035:186782] [ 8] icoFoam[0x44c179]
[skl035:186782] [ 9] icoFoam[0x41c161]
[skl035:186782] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaaae771c05]
[skl035:186782] [11] icoFoam[0x41cba7]
[skl035:186782] *** End of error message ***
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 186769 on node skl035 exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------
[skl035:186763] 2 more processes have sent help message help-opal-runtime.txt / opal_init:warn-fork
[skl035:186763] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Question -
1. Is aforementioned method correct for setting up/running a synthetic 1M cavity benchmark? Please let me know if there are other config files which i should have modified before running this case.
Please let me know if i can provide you with any further information.

simrego · April 6, 2019, 05:29

Hi!

Decrease the time step! You have a giant courant number.... It shouldn't be higher than one with icoFoam.

puneet336 · April 6, 2019, 06:07

i tried with 4 processes,
after 4 minutes, simulation crashed as -

Code:

time mpirun --oversubscribe -np 4 icoFoam 2>&1|tee logs_run4
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  6
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 6
Exec   : icoFoam
Date   : Apr 06 2019
Time   : 04:56:19
Host   : "cn003"
PID    : 191573
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  6
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 6
Exec   : icoFoam
Date   : Apr 06 2019
Time   : 04:56:19
Host   : "cn003"
PID    : 191574
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  6
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 6
Exec   : icoFoam
Date   : Apr 06 2019
Time   : 04:56:19
Host   : "cn003"
PID    : 191575
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  6
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 6
Exec   : icoFoam
Date   : Apr 06 2019
Time   : 04:56:19
Host   : "cn003"
PID    : 191576
I/O    : uncollated
Case   : /home/user/MySoftwares/UTILS/OpenFOAM/user-6/run/cavity
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

I/O    : uncollated
Case   : /home/user/MySoftwares/UTILS/OpenFOAM/user-6/run/cavity
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

I/O    : uncollated
Case   : /home/user/MySoftwares/UTILS/OpenFOAM/user-6/run/cavity
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Create mesh for time = 0

Create mesh for time = 0

I/O    : uncollated
Case   : /home/user/MySoftwares/UTILS/OpenFOAM/user-6/run/cavity
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi

Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi


Starting time loop

Time = 0.005

Courant Number mean: 0 max: 0
Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi


Starting time loop

Time = 0.005

Courant Number mean: 0 max: 0
Reading transportProperties

Reading field p

Reading field U


Starting time loop

Time = 0.005

Reading/calculating face flux field phi

Courant Number mean: 0 max: 0

Starting time loop

Time = 0.005

Courant Number mean: 0 max: 0
smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 0.00678545, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 0.00678545, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 0.00678545, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 0.00678545, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.687191, No Iterations 1000
time step continuity errors : sum local = 0.0023281, global = 3.56267e-19, cumulative = 3.56267e-19
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.687191, No Iterations 1000
time step continuity errors : sum local = 0.0023281, global = 3.56267e-19, cumulative = 3.56267e-19
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.687191, No Iterations 1000
time step continuity errors : sum local = 0.0023281, global = 3.56267e-19, cumulative = 3.56267e-19
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.687191, No Iterations 1000
time step continuity errors : sum local = 0.0023281, global = 3.56267e-19, cumulative = 3.56267e-19
DICPCG:  Solving for p, Initial residual = 0.945219, Final residual = 0.0716426, No Iterations 1000
time step continuity errors : sum local = 0.000511614, global = -3.8407e-19, cumulative = -2.78036e-20
ExecutionTime = 83.57 s  ClockTime = 83 s

Time = 0.01

Courant Number mean: 3.28089 max: 47.7257
DICPCG:  Solving for p, Initial residual = 0.945219, Final residual = 0.0716426, No Iterations 1000
time step continuity errors : sum local = 0.000511614, global = -3.8407e-19, cumulative = -2.78036e-20
ExecutionTime = 83.76 s  ClockTime = 84 s

Time = 0.01

Courant Number mean: 3.28089 max: 47.7257
DICPCG:  Solving for p, Initial residual = 0.945219, Final residual = 0.0716426, No Iterations 1000
time step continuity errors : sum local = 0.000511614, global = -3.8407e-19, cumulative = -2.78036e-20
ExecutionTime = 84.37 s  ClockTime = 84 s

Time = 0.01

Courant Number mean: 3.28089 max: 47.7257
DICPCG:  Solving for p, Initial residual = 0.945219, Final residual = 0.0716426, No Iterations 1000
time step continuity errors : sum local = 0.000511614, global = -3.8407e-19, cumulative = -2.78036e-20
ExecutionTime = 84.64 s  ClockTime = 84 s

Time = 0.01

Courant Number mean: 3.28089 max: 47.7257
smoothSolver:  Solving for Ux, Initial residual = 0.275226, Final residual = 0.131304, No Iterations 1000
smoothSolver:  Solving for Ux, Initial residual = 0.275226, Final residual = 0.131304, No Iterations 1000
smoothSolver:  Solving for Ux, Initial residual = 0.275226, Final residual = 0.131304, No Iterations 1000
smoothSolver:  Solving for Ux, Initial residual = 0.275226, Final residual = 0.131304, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.97424, Final residual = 0.62469, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.97424, Final residual = 0.62469, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.97424, Final residual = 0.62469, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.97424, Final residual = 0.62469, No Iterations 1000
DICPCG:  Solving for p, Initial residual = 0.998548, Final residual = 0.794578, No Iterations 1000
time step continuity errors : sum local = 0.718303, global = 2.56095e-17, cumulative = 2.55817e-17
DICPCG:  Solving for p, Initial residual = 0.998548, Final residual = 0.794578, No Iterations 1000
time step continuity errors : sum local = 0.718303, global = 2.56095e-17, cumulative = 2.55817e-17
DICPCG:  Solving for p, Initial residual = 0.998548, Final residual = 0.794578, No Iterations 1000
time step continuity errors : sum local = 0.718303, global = 2.56095e-17, cumulative = 2.55817e-17
DICPCG:  Solving for p, Initial residual = 0.998548, Final residual = 0.794578, No Iterations 1000
time step continuity errors : sum local = 0.718303, global = 2.56095e-17, cumulative = 2.55817e-17
DICPCG:  Solving for p, Initial residual = 0.779639, Final residual = 0.374333, No Iterations 1000
time step continuity errors : sum local = 0.663485, global = 1.46907e-17, cumulative = 4.02724e-17
ExecutionTime = 188.34 s  ClockTime = 188 s

Time = 0.015

Courant Number mean: 20.1784 max: 835.394
DICPCG:  Solving for p, Initial residual = 0.779639, Final residual = 0.374333, No Iterations 1000
time step continuity errors : sum local = 0.663485, global = 1.46907e-17, cumulative = 4.02724e-17
ExecutionTime = 188.69 s  ClockTime = 189 s

Time = 0.015

Courant Number mean: 20.1784 max: 835.394
DICPCG:  Solving for p, Initial residual = 0.779639, Final residual = 0.374333, No Iterations 1000
time step continuity errors : sum local = 0.663485, global = 1.46907e-17, cumulative = 4.02724e-17
ExecutionTime = 191.54 s  ClockTime = 192 s

Time = 0.015

Courant Number mean: 20.1784 max: 835.394
DICPCG:  Solving for p, Initial residual = 0.779639, Final residual = 0.374333, No Iterations 1000
time step continuity errors : sum local = 0.663485, global = 1.46907e-17, cumulative = 4.02724e-17
ExecutionTime = 191.82 s  ClockTime = 192 s

Time = 0.015

Courant Number mean: 20.1784 max: 835.394
smoothSolver:  Solving for Ux, Initial residual = 0.983178, Final residual = 0.674171, No Iterations 1000
smoothSolver:  Solving for Ux, Initial residual = 0.983178, Final residual = 0.674171, No Iterations 1000
smoothSolver:  Solving for Ux, Initial residual = 0.983178, Final residual = 0.674171, No Iterations 1000
smoothSolver:  Solving for Ux, Initial residual = 0.983178, Final residual = 0.674171, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.980638, Final residual = 0.584039, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.980638, Final residual = 0.584039, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.980638, Final residual = 0.584039, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.980638, Final residual = 0.584039, No Iterations 1000
DICPCG:  Solving for p, Initial residual = 0.998725, Final residual = 0.453711, No Iterations 1000
time step continuity errors : sum local = 207.333, global = -7.3029e-14, cumulative = -7.29887e-14
DICPCG:  Solving for p, Initial residual = 0.998725, Final residual = 0.453711, No Iterations 1000
time step continuity errors : sum local = 207.333, global = -7.3029e-14, cumulative = -7.29887e-14
DICPCG:  Solving for p, Initial residual = 0.998725, Final residual = 0.453711, No Iterations 1000
time step continuity errors : sum local = 207.333, global = -7.3029e-14, cumulative = -7.29887e-14
DICPCG:  Solving for p, Initial residual = 0.998725, Final residual = 0.453711, No Iterations 1000
time step continuity errors : sum local = 207.333, global = -7.3029e-14, cumulative = -7.29887e-14
DICPCG:  Solving for p, Initial residual = 0.873432, Final residual = 0.000718764, No Iterations 1000
time step continuity errors : sum local = 0.469363, global = -9.89589e-16, cumulative = -7.39783e-14
ExecutionTime = 292.86 s  ClockTime = 293 s

Time = 0.02

Courant Number mean: 5377.55 max: 79240
#0  Foam::error::printStack(Foam::Ostream&) at ??:?
#1  Foam::sigFpe::sigHandler(int) at ??:?
#2  ? in "/lib64/libc.so.6"
#3  Foam::symGaussSeidelSmoother::smooth(Foam::word const&, Foam::Field<double>&, Foam::lduMatrix const&, Foam::Field<double> const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char, int) at ??:?
#4  Foam::symGaussSeidelSmoother::smooth(Foam::Field<double>&, Foam::Field<double> const&, unsigned char, int) constDICPCG:  Solving for p, Initial residual = 0.873432, Final residual = 0.000718764, No Iterations 1000
 at ??:?
#5  Foam::smoothSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
#6  ?time step continuity errors : sum local = 0.469363, global = -9.89589e-16, cumulative = -7.39783e-14
 at ??:?
#7  ? at ??:?
#8  ExecutionTime = 293.25 s  ClockTime = 294 s

Time = 0.02

? at ??:?
#9  Courant Number mean: 5377.55 max: 79240
? at ??:?
#10  __libc_start_main in "/lib64/libc.so.6"
#11  ? at ??:?
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 191575 on node cn003 exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------

real    4m54.460s
user    4m52.835s
sys     0m0.625s

simrego · April 6, 2019, 06:11

You missed the -parallel switch. That's why you get many things four times.
But as i said decrease the time step! This is your main problem.
Your sell size is ~50x smaller than the original one. Try with at least 50x smaller time steps than the original.

puneet336 · April 6, 2019, 06:26

Quote:

Originally Posted by simrego

Hi!

Decrease the time step! You have a giant courant number.... It shouldn't be higher than one with icoFoam.

thank you simrego for the reply, i am not sure about where to put the courant number setting and is there any thumb rule to select the courant number ?

adding following lines in system/controlDict will be enough for my current setup?

Code:

adjustTimeStep  yes;
maxCo           1;

Update:
Thankyou again for the quick reply on np=4 failure, i will use -parallel switch.
If am correct , timestep can be adjusted in system/controlDict's deltaT.
i will now try with 50x smaller deltaT -

Code:

deltaT         .0001;

default was -

Code:

deltaT          0.005;

i will skip modification of adjustTimeStep & maxCo in system/controlDict for now.

simrego · April 6, 2019, 06:33

You select the timestep not he courant number. Courant number is "how many cells will the fluid pass through in one time step". Namely velocity/cellLength. For icoFoam (PISO algorithm) you need max Co under one. If you are using pimpleFoam (PIMPLE algorithm), then you can have greater Co.

puneet336 · April 6, 2019, 07:17

Quote:

Originally Posted by simrego

You select the timestep not he courant number. Courant number is "how many cells will the fluid pass through in one time step". Namely velocity/cellLength. For icoFoam (PISO algorithm) you need max Co under one. If you are using pimpleFoam (PIMPLE algorithm), then you can have greater Co.

Thank you for the explanation and Yes, adjusting deltaT did not help.
As I am using icoFoam so seems i need to figure out way to keep courant number <1.
from here it seems that system/controlDict file is once place where we can specify the same. and i assume that

maxCO = maximum allowed value of courant number.

Though, still unsure where to specify maxCO , with default deltaT (0.005) i tried following -

attempt1:

Code:

adjustTimeStep  yes;
maxCo           1;

attempt2

Code:

adjustTimeStep  yes;
maxCo           0.8;

on both attempts i got error. So if maxCO is still the root cause, seems still i am unable to specify it correctly.

error log with attempt1:

Code:

mpirun -np 4 icoFoam -parallel 2>&1|tee logs_run4
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  6
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 6
Exec   : icoFoam -parallel
Date   : Apr 06 2019
Time   : 05:42:26
Host   : "node003"
PID    : 194943
I/O    : uncollated
Case   : /home/user/MySoftwares/UTILS/OpenFOAM/user-6/run/cavity
nProcs : 4
Slaves :
3
(
"node003.194944"
"node003.194945"
"node003.194946"
)

Pstream initialized with:
    floatTransfer      : 0
    nProcsSimpleSum    : 0
    commsType          : nonBlocking
    polling iterations : 0
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
...
Time = 0.015

Courant Number mean: 20.0977 max: 1474.94
smoothSolver:  Solving for Ux, Initial residual = 0.989775, Final residual = 0.732099, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.977238, Final residual = 0.564603, No Iterations 1000
DICPCG:  Solving for p, Initial residual = 0.998722, Final residual = 1.20531, No Iterations 1000
time step continuity errors : sum local = 476.907, global = -1.4988e-13, cumulative = -1.49952e-13
DICPCG:  Solving for p, Initial residual = 0.839127, Final residual = 0.0492843, No Iterations 1000
time step continuity errors : sum local = 49.0106, global = -1.02696e-13, cumulative = -2.52647e-13
ExecutionTime = 67.22 s  ClockTime = 68 s

Time = 0.02

Courant Number mean: 8382.29 max: 768785
[2] #0  Foam::error::printStack(Foam::Ostream&)--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[45422,1],2] (PID 194945)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
[0] #0  Foam::error::printStack(Foam::Ostream&) at ??:?
 at ??:?
[2] #1  Foam::sigFpe::sigHandler(int)[0] #1  Foam::sigFpe::sigHandler(int) at ??:?
[2] #2  ? at ??:?
[0] #2  ? in "/lib64/libc.so.6"
[2] #3  Foam::symGaussSeidelSmoother::smooth(Foam::word const&, Foam::Field<double>&, Foam::lduMatrix const&, Foam::Field<double> const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char, int) in "/lib64/libc.so.6"
[0] #3  Foam::symGaussSeidelSmoother::smooth(Foam::word const&, Foam::Field<double>&, Foam::lduMatrix const&, Foam::Field<double> const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char, int) at ??:?
[2] #4  Foam::symGaussSeidelSmoother::smooth(Foam::Field<double>&, Foam::Field<double> const&, unsigned char, int) const at ??:?
[0] #4  Foam::symGaussSeidelSmoother::smooth(Foam::Field<double>&, Foam::Field<double> const&, unsigned char, int) const at ??:?
[2] #5  Foam::smoothSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[0] #5  Foam::smoothSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[2] #6   at ??:?
[0] #6  ?? at ??:?
[0] #7   at ??:?
[2] #7  ?? at ??:?
[0] #8   at ??:?
[2] #8  ?? at ??:?
[2] #9   at ??:?
[0] #9  ?? at ??:?
[2] #10  __libc_start_main at ??:?
[0] #10  __libc_start_main in "/lib64/libc.so.6"
[2] #11   in "/lib64/libc.so.6"
[0] #11  ?? at ??:?
[node003:194945] *** Process received signal ***
[node003:194945] Signal: Floating point exception (8)
[node003:194945] Signal code:  (-6)
[node003:194945] Failing at address: 0x4110002f981
[node003:194945] [ 0] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[node003:194945] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2aaaae7851f7]
[node003:194945] [ 2] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[node003:194945] [ 3]  at ??:?
[node003:194943] *** Process received signal ***
[node003:194943] Signal: Floating point exception (8)
[node003:194943] Signal code:  (-6)
[node003:194943] Failing at address: 0x4110002f97f
[node003:194943] [ 0] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam22symGaussSeidelSmoother6smoothERKNS_4wordERNS_5FieldIdEERKNS_9lduMatrixERKS5_RKNS_10FieldFieldIS4_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEhi+0x3cc)[0x2aaaad69d64c]
[node003:194945] [ 4] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[node003:194943] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2aaaae7851f7]
[node003:194943] [ 2] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[node003:194943] [ 3] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam22symGaussSeidelSmoother6smoothERNS_5FieldIdEERKS2_hi+0x2d)[0x2aaaad69d92d]
[node003:194945] [ 5] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam22symGaussSeidelSmoother6smoothERKNS_4wordERNS_5FieldIdEERKNS_9lduMatrixERKS5_RKNS_10FieldFieldIS4_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEhi+0x3cc)[0x2aaaad69d64c]
[node003:194943] [ 4] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam12smoothSolver5solveERNS_5FieldIdEERKS2_h+0x377)[0x2aaaad6963f7]
[node003:194945] [ 6] icoFoam[0x445819]
[node003:194945] [ 7] icoFoam[0x44be5c]
[node003:194945] [ 8] icoFoam[0x44c179]
[node003:194945] [ 9] icoFoam[0x41c161]
[node003:194945] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaaae771c05]
[node003:194945] [11] icoFoam[0x41cba7]
[node003:194945] *** End of error message ***
/home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam22symGaussSeidelSmoother6smoothERNS_5FieldIdEERKS2_hi+0x2d)[0x2aaaad69d92d]
[node003:194943] [ 5] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam12smoothSolver5solveERNS_5FieldIdEERKS2_h+0x377)[0x2aaaad6963f7]
[node003:194943] [ 6] icoFoam[0x445819]
[node003:194943] [ 7] icoFoam[0x44be5c]
[node003:194943] [ 8] icoFoam[0x44c179]
[node003:194943] [ 9] icoFoam[0x41c161]
[node003:194943] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaaae771c05]
[node003:194943] [11] icoFoam[0x41cba7]
[node003:194943] *** End of error message ***
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 194945 on node node003 exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------
[node003:194937] 1 more process has sent help message help-opal-runtime.txt / opal_init:warn-fork
[node003:194937] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

error log with attempt2:

Code:

/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  6
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 6
Exec   : icoFoam -parallel
Date   : Apr 06 2019
Time   : 05:53:28
Host   : "node003"
PID    : 195862
I/O    : uncollated
Case   : /home/user/MySoftwares/UTILS/OpenFOAM/user-6/run/cavity
nProcs : 4
Slaves :
3
(
"node003.195863"
"node003.195864"
"node003.195865"
)

Pstream initialized with:
    floatTransfer      : 0
    nProcsSimpleSum    : 0
    commsType          : nonBlocking
    polling iterations : 0
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi


Starting time loop

Time = 0.005

Courant Number mean: 0 max: 0
smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 0.00679184, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.23142, No Iterations 1000
time step continuity errors : sum local = 0.000813476, global = 1.25996e-18, cumulative = 1.25996e-18
DICPCG:  Solving for p, Initial residual = 0.958049, Final residual = 0.0759298, No Iterations 1000
time step continuity errors : sum local = 0.00037845, global = 1.02901e-18, cumulative = 2.28897e-18
ExecutionTime = 20 s  ClockTime = 20 s

Time = 0.01

Courant Number mean: 3.27801 max: 47.768
smoothSolver:  Solving for Ux, Initial residual = 0.203336, Final residual = 0.0898295, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.971246, Final residual = 0.615645, No Iterations 1000
DICPCG:  Solving for p, Initial residual = 0.998887, Final residual = 0.919699, No Iterations 1000
time step continuity errors : sum local = 0.488496, global = -1.01644e-17, cumulative = -7.87542e-18
DICPCG:  Solving for p, Initial residual = 0.749711, Final residual = 0.429908, No Iterations 1000
time step continuity errors : sum local = 0.528229, global = -6.36969e-17, cumulative = -7.15723e-17
ExecutionTime = 43.74 s  ClockTime = 44 s

Time = 0.015

Courant Number mean: 20.0977 max: 1474.94
smoothSolver:  Solving for Ux, Initial residual = 0.989775, Final residual = 0.732099, No Iterations 1000
smoothSolver:  Solving for Uy, Initial residual = 0.977238, Final residual = 0.564603, No Iterations 1000
DICPCG:  Solving for p, Initial residual = 0.998722, Final residual = 1.20531, No Iterations 1000
time step continuity errors : sum local = 476.907, global = -1.4988e-13, cumulative = -1.49952e-13
DICPCG:  Solving for p, Initial residual = 0.839127, Final residual = 0.0492843, No Iterations 1000
time step continuity errors : sum local = 49.0106, global = -1.02696e-13, cumulative = -2.52647e-13
ExecutionTime = 67.59 s  ClockTime = 68 s

Time = 0.02

Courant Number mean: 8382.29 max: 768785
[2] #0  Foam::error::printStack(Foam::Ostream&)--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[46365,1],2] (PID 195864)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
[0] #0  Foam::error::printStack(Foam::Ostream&) at ??:?
 at ??:?
[0] #1  [2] #1  Foam::sigFpe::sigHandler(int)Foam::sigFpe::sigHandler(int) at ??:?
[2] #2  ? at ??:?
[0] #2  ? in "/lib64/libc.so.6"
[2] #3  Foam::symGaussSeidelSmoother::smooth(Foam::word const&, Foam::Field<double>&, Foam::lduMatrix const&, Foam::Field<double> const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char, int) in "/lib64/libc.so.6"
[0] #3  Foam::symGaussSeidelSmoother::smooth(Foam::word const&, Foam::Field<double>&, Foam::lduMatrix const&, Foam::Field<double> const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char, int) at ??:?
[2] #4  Foam::symGaussSeidelSmoother::smooth(Foam::Field<double>&, Foam::Field<double> const&, unsigned char, int) const at ??:?
[0] #4  Foam::symGaussSeidelSmoother::smooth(Foam::Field<double>&, Foam::Field<double> const&, unsigned char, int) const at ??:?
[2] #5  Foam::smoothSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[0] #5  Foam::smoothSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[2] #6   at ??:?
[0] #6  ?? at ??:?
[2] #7   at ??:?
[0] #7  ?? at ??:?
[2] #8   at ??:?
[0] #8  ?? at ??:?
[2] #9   at ??:?
[0] #9  ?? at ??:?
[2] #10  __libc_start_main at ??:?
[0] #10  __libc_start_main in "/lib64/libc.so.6"
[2] #11  ? in "/lib64/libc.so.6"
[0] #11   at ??:?
[node003:195864] *** Process received signal ***
[node003:195864] Signal: Floating point exception (8)
[node003:195864] Signal code:  (-6)
[node003:195864] Failing at address: 0x4110002fd18
[node003:195864] [ 0] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[node003:195864] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2aaaae7851f7]
[node003:195864] [ 2] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[node003:195864] [ 3] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam22symGaussSeidelSmoother6smoothERKNS_4wordERNS_5FieldIdEERKNS_9lduMatrixERKS5_RKNS_10FieldFieldIS4_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEhi+0x3cc)[0x2aaaad69d64c]
[node003:195864] [ 4] ?/home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam22symGaussSeidelSmoother6smoothERNS_5FieldIdEERKS2_hi+0x2d)[0x2aaaad69d92d]
[node003:195864] [ 5] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam12smoothSolver5solveERNS_5FieldIdEERKS2_h+0x377)[0x2aaaad6963f7]
[node003:195864] [ 6] icoFoam[0x445819]
[node003:195864] [ 7] icoFoam[0x44be5c]
[node003:195864] [ 8] icoFoam[0x44c179]
[node003:195864] [ 9] icoFoam[0x41c161]
[node003:195864] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaaae771c05]
[node003:195864] [11] icoFoam[0x41cba7]
[node003:195864] *** End of error message ***
 at ??:?
[node003:195862] *** Process received signal ***
[node003:195862] Signal: Floating point exception (8)
[node003:195862] Signal code:  (-6)
[node003:195862] Failing at address: 0x4110002fd16
[node003:195862] [ 0] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[node003:195862] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2aaaae7851f7]
[node003:195862] [ 2] /lib64/libc.so.6(+0x35270)[0x2aaaae785270]
[node003:195862] [ 3] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam22symGaussSeidelSmoother6smoothERKNS_4wordERNS_5FieldIdEERKNS_9lduMatrixERKS5_RKNS_10FieldFieldIS4_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEEhi+0x3cc)[0x2aaaad69d64c]
[node003:195862] [ 4] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam22symGaussSeidelSmoother6smoothERNS_5FieldIdEERKS2_hi+0x2d)[0x2aaaad69d92d]
[node003:195862] [ 5] /home/user/MySoftwares/UTILS/OpenFOAM//OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam12smoothSolver5solveERNS_5FieldIdEERKS2_h+0x377)[0x2aaaad6963f7]
[node003:195862] [ 6] icoFoam[0x445819]
[node003:195862] [ 7] icoFoam[0x44be5c]
[node003:195862] [ 8] icoFoam[0x44c179]
[node003:195862] [ 9] icoFoam[0x41c161]
[node003:195862] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2aaaae771c05]
[node003:195862] [11] icoFoam[0x41cba7]
[node003:195862] *** End of error message ***
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 195864 on node node003 exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------
[node003:195850] 1 more process has sent help message help-opal-runtime.txt / opal_init:warn-fork
[node003:195850] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

real    1m8.916s
user    1m6.416s
sys     0m1.608s

puneet336 · April 6, 2019, 08:58

Found out one working setting, simulaiton ran to completion on 4 cores.

in system/controlDict, setting deltaT as

Code:

deltaT     0.001

and setting blocks to -

Code:

blocks
(
    hex (0 1 2 3 4 5 6 7) (100 100 1) simpleGrading (1 1 1)
);

simulation runs for 6 seconds and courant number mean/max was stable -
Courant Number mean: 0.225736 max: 0.970652

please help me for appropriate settings for 1M cavity benchmark like the one used by advisory council.

This post can serve as reference for newbies like me looking for benchmark setup on openfoam.

simrego · April 6, 2019, 09:08

As I mentioned I'm not sure if icoFoam supports adjustable time step, i googled it and it seems like i was correct. So that's why nothing happens if you turn it on. The solver won't read that information.

If you are good with 100x100 mesh and 0.001 sec time step, i would suggest you to try 1000x1000 mesh with 0.0001 sec time step. It could be okay.

simrego · April 6, 2019, 09:15

Quote:

Originally Posted by puneet336

Found out one working setting, simulaiton ran to completion on 4 cores.

in system/controlDict, setting deltaT as

Code:

deltaT     0.001

and setting blocks to -

Code:

blocks
(
    hex (0 1 2 3 4 5 6 7) (100 100 1) simpleGrading (1 1 1)
);

simulation runs for 6 seconds and courant number mean/max was stable -
Courant Number mean: 0.225736 max: 0.970652

please help me for appropriate settings for 1M cavity benchmark like the one used by advisory council.

This post can serve as reference for newbies like me looking for benchmark setup on openfoam.

Just to make everything clear:
https://en.wikipedia.org/wiki/Couran...Lewy_condition
As you see Co (courant number) is calculated as:
Co = U*dt/dx, where U is the velocity in the cell, dt is the timestep, and dx is the "length" of your cell. Rearrange, then:
dt = Co * dx / U
You need Co = 1 (ideally Co < 1), your dx = 0.0001 meters (with the default tutorial but 1000x1000 grid), and your U = 1m/s at the top ( This data is again from the default tutorial. Of course this velocity is different in every cell but let's assume this is the max velocity in the domain )
With these data you can easily calculate that the necessary time step for your simulation is ~0.0001 sec.

puneet336 · April 6, 2019, 09:56

Quote:

Originally Posted by simrego

Just to make everything clear:
https://en.wikipedia.org/wiki/Couran...Lewy_condition
As you see Co (courant number) is calculated as:
Co = U*dt/dx, where U is the velocity in the cell, dt is the timestep, and dx is the "length" of your cell. Rearrange, then:
dt = Co * dx / U
You need Co = 1 (ideally Co < 1), your dx = 0.0001 meters (with the default tutorial but 1000x1000 grid), and your U = 1m/s at the top ( This data is again from the default tutorial. Of course this velocity is different in every cell but let's assume this is the max velocity in the domain )
With these data you can easily calculate that the necessary time step for your simulation is ~0.0001 sec.

Hi,

deltaT 0.0001 + (1000 1000 1) didn't work

here is the stdout log -
https://bitbucket.org/puneet336/cavi...aster/logs.txt

you could have a look at the input setup which i am using here

P.S. As 1M benchmark features in number of publications on web, OpenFOAM developers could put this benchmark in its tutorial sections in their future releases.

puneet336 · April 7, 2019, 00:58

Came across an article at - pugetsystems

here are the list of changes i made to the default setup -

Code:

sed -i "s/0.01/0.001/g" constant/transportProperties
sed -i "s/20 20 1/1024 1024 1/g" system/blockMeshDict

sed -i "/^endTime/c endTime         0.001;"  system/controlDict
sed -i "/^deltaT/c deltaT          0.00001;"  system/controlDict
sed -i "/^writeControl/c writeControl    runTime;"  system/controlDict
sed -i "/^writeInterval/c writeInterval   0.1;"  system/controlDict
sed -i "/^runTimeModifiable/c runTimeModifiable false;" system/controlDict

sed -i "s/n               (2 2 1);/n               ( 1 1 1 );/g"  system/decomposeParDict
sed -i "s/delta           0.001;/delta           0.0001;/g" system/decomposeParDict

then, deleted following sections from system/decomposeParDict (i was using damBreak's decomposeParDict) -

Code:

 hierarchicalCoeffs
 {
     n               (1 1 1);
     delta           0.001;
     order           xyz;
 }

 manualCoeffs
 {
     dataFile        "";
  }

and changed the subdomains for 40 cores -

Code:

sed -i "s/numberOfSubdomains 4;/numberOfSubdomains 40;/g" system/decomposeParDict
sed -i "s/n               ( 1 1 1 );/n               ( 4 10 1 );/g" system/decomposeParDict

the simulation ran fine and finished in ~140s.

setup is here

I guess similar setup is being used for all/similar benchmarks results available on internet under title "Lid Driven Cavity Flow - 1 million elements, DP, 2D icoFoam solver for laminar, isothermal, incompressible flow"

thankyou simrego for help
and thanks to Dr Donald Kinghorn/pugetsystems for their super helpful KA.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[snappyHexMesh] SnappyHexMesh for internal Flow	vishwa	OpenFOAM Meshing & Mesh Conversion	24	June 27, 2016 08:54
Boundary condition for 2d lid driven cavity using ghost cells	quarkz	Main CFD Forum	9	January 20, 2013 05:54
[Netgen] Import netgen mesh to OpenFOAM	hsieh	OpenFOAM Meshing & Mesh Conversion	32	September 13, 2011 05:50
Lid Driven Cavity using Ghost Cell Method and in C++	illuminati5288	Main CFD Forum	0	August 12, 2011 22:05
physical boundary error!!	kris	Siemens	2	August 3, 2005 00:32

April 6, 2019, 05:29		#2
simrego Senior Member anonymous Join Date: Jan 2016 Posts: 416 Rep Power: 14	Hi! Decrease the time step! You have a giant courant number.... It shouldn't be higher than one with icoFoam.

April 6, 2019, 06:11		#4
simrego Senior Member anonymous Join Date: Jan 2016 Posts: 416 Rep Power: 14	You missed the -parallel switch. That's why you get many things four times. But as i said decrease the time step! This is your main problem. Your sell size is ~50x smaller than the original one. Try with at least 50x smaller time steps than the original.

April 6, 2019, 06:33		#6
simrego Senior Member anonymous Join Date: Jan 2016 Posts: 416 Rep Power: 14	You select the timestep not he courant number. Courant number is "how many cells will the fluid pass through in one time step". Namely velocity/cellLength. For icoFoam (PISO algorithm) you need max Co under one. If you are using pimpleFoam (PIMPLE algorithm), then you can have greater Co.

April 6, 2019, 08:58		#8
puneet336 New Member puneet Join Date: Nov 2015 Posts: 19 Rep Power: 10	Found out one working setting, simulaiton ran to completion on 4 cores. in system/controlDict, setting deltaT as Code: deltaT 0.001 and setting blocks to - Code: blocks ( hex (0 1 2 3 4 5 6 7) (100 100 1) simpleGrading (1 1 1) ); simulation runs for 6 seconds and courant number mean/max was stable - Courant Number mean: 0.225736 max: 0.970652 please help me for appropriate settings for 1M cavity benchmark like the one used by advisory council. This post can serve as reference for newbies like me looking for benchmark setup on openfoam.

April 6, 2019, 09:08		#9
simrego Senior Member anonymous Join Date: Jan 2016 Posts: 416 Rep Power: 14	As I mentioned I'm not sure if icoFoam supports adjustable time step, i googled it and it seems like i was correct. So that's why nothing happens if you turn it on. The solver won't read that information. If you are good with 100x100 mesh and 0.001 sec time step, i would suggest you to try 1000x1000 mesh with 0.0001 sec time step. It could be okay.