[msrinath80]

Hi,

I'm trying the same cavity case as explained in the user guide. But I've increased the mesh to 10000 cells and decreased the time step from 0.005 to 0.001. The case runs happily when in serial mode. When I decompose the domain and run in parallel, it seems to run fine for a while after which, suddenly a solution singluarity appears. The BCs and problem definition are OK as the case runs properly in serial. Any suggestions (please look at the time between 0.013 and 0.015 seconds). Thanks for your help.

PS: At Time = 0.024, I hit Ctrl + C.

Also, when running the case in parallel again (from scratch) sometimes it works fine up to 0.2 or 0.3 seconds, sometimes less, sometimes it just hangs there with no error messages as if waiting for something. Has anybody experienced such random behavior. I'm using OpenFOAM on a 64 bit IBM p360 (4-CPU) machine running Suse Linux 10.1 and lam (compiled from sources that came with OpenFOAM).


madhavan@varese:~/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam> ls
Allclean cavity cavityGrade cavity_large_mesh_backup machines
Allrun cavityClipped cavity_large_mesh elbow resetFixedWallsScr

madhavan@varese:~/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam> decomposePar . cavity_large_mesh

/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.3 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

Exec : decomposePar . cavity_large_mesh
Date : Jul 25 2006
Time : 19:20:21
Host : varese
PID : 32389
Root : /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam
Case : cavity_large_mesh
Nprocs : 1
Create time

Time = 0
Create mesh


Calculating distribution of cells
Selecting decompositionMethod simple

Finished decomposition in 0.45 s

Calculating original mesh data

Distributing cells to processors

Distributing faces to processors

Calculating processor boundary addressing

Distributing points to processors

Constructing processor meshes

Processor 0
Number of cells = 2500
Number of faces shared with processor 1 = 50
Number of faces shared with processor 2 = 50
Number of boundary faces = 5100

Processor 1
Number of cells = 2500
Number of faces shared with processor 0 = 50
Number of faces shared with processor 3 = 50
Number of boundary faces = 5100

Processor 2
Number of cells = 2500
Number of faces shared with processor 0 = 50
Number of faces shared with processor 3 = 50
Number of boundary faces = 5100

Processor 3
Number of cells = 2500
Number of faces shared with processor 1 = 50
Number of faces shared with processor 2 = 50
Number of boundary faces = 5100

Processor 0: field transfer
Processor 1: field transfer
Processor 2: field transfer
Processor 3: field transfer

End.

madhavan@varese:~/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam> mpirun -np 4 icoFoam . cavity_large_mesh -parallel
/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.3 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.3 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.3 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.3 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

Exec : icoFoam . cavity_large_mesh -parallel
Exec : icoFoam . cavity_large_mesh -parallel
Exec : icoFoam . cavity_large_mesh -parallel
Exec : icoFoam . cavity_large_mesh -parallel
[2] Date : Jul 25 2006
[2] Time : 19:21:14
[0] Date : Jul 25 2006
[0] Time : 19:21:14
[0] Host : varese
[2] Host : varese
[2] PID : 32393
[3] Date : Jul 25 2006
[3] Time : 19:21:14
[0] PID : 32391
[3] Host : varese
[3] PID : 32394
[1] Date : Jul 25 2006
[1] Time : 19:21:14
[1] Host : varese
[1] PID : 32392
[1] Root : /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam
[1] Case : cavity_large_mesh
[1] Nprocs : 4
[2] Root : /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam
[2] Case : cavity_large_mesh
[2] Nprocs : 4
[3] Root : /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam
[3] Case : cavity_large_mesh
[0] Root : /home/madhavan/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam
[0] Case : cavity_large_mesh
[0] Nprocs : 4
[0] Slaves :
[0] 3
[3] Nprocs : 4
[0] (
[0] varese.32392
[0] varese.32393
[0] varese.32394
[0] )
[0]
Create time

Create mesh for time = 0

Reading transportProperties

Reading field p

Reading field U

Reading/calculating face flux field phi


Starting time loop

Time = 0.001

Mean and max Courant Numbers = 0 0
BICCG: Solving for Ux, Initial residual = 1, Final residual = 9.04991e-06, No Iterations 18
BICCG: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
ICCG: Solving for p, Initial residual = 1, Final residual = 6.86206e-07, No Iterations 201
time step continuity errors : sum local = 4.28884e-10, global = -9.48776e-21, cumulative = -9.48776e-21
ICCG: Solving for p, Initial residual = 0.672536, Final residual = 8.34589e-07, No Iterations 200
time step continuity errors : sum local = 7.54547e-10, global = -5.38652e-20, cumulative = -6.3353e-20
ExecutionTime = 2.51 s ClockTime = 3 s

Time = 0.002

Mean and max Courant Numbers = 0.0296832 0.810069
BICCG: Solving for Ux, Initial residual = 0.119078, Final residual = 6.90333e-06, No Iterations 17
BICCG: Solving for Uy, Initial residual = 0.386927, Final residual = 6.9427e-06, No Iterations 18
ICCG: Solving for p, Initial residual = 0.722155, Final residual = 9.66376e-07, No Iterations 198
time step continuity errors : sum local = 9.31977e-10, global = -1.34216e-19, cumulative = -1.97569e-19
ICCG: Solving for p, Initial residual = 0.633029, Final residual = 9.07363e-07, No Iterations 197
time step continuity errors : sum local = 9.3323e-10, global = 1.01922e-19, cumulative = -9.56466e-20
ExecutionTime = 4.04 s ClockTime = 4 s

Time = 0.003

Mean and max Courant Numbers = 0.0449432 0.926016
BICCG: Solving for Ux, Initial residual = 0.047979, Final residual = 9.80318e-06, No Iterations 15
BICCG: Solving for Uy, Initial residual = 0.146506, Final residual = 4.93936e-06, No Iterations 17
ICCG: Solving for p, Initial residual = 0.711663, Final residual = 9.89994e-07, No Iterations 197
time step continuity errors : sum local = 7.55206e-10, global = 4.25957e-20, cumulative = -5.30509e-20
ICCG: Solving for p, Initial residual = 0.663622, Final residual = 8.84782e-07, No Iterations 197
time step continuity errors : sum local = 6.7351e-10, global = 8.68074e-21, cumulative = -4.43702e-20
ExecutionTime = 5.55 s ClockTime = 6 s

Time = 0.004

Mean and max Courant Numbers = 0.053069 0.930456
BICCG: Solving for Ux, Initial residual = 0.026967, Final residual = 8.26492e-06, No Iterations 14
BICCG: Solving for Uy, Initial residual = 0.104978, Final residual = 5.43694e-06, No Iterations 16
ICCG: Solving for p, Initial residual = 0.496791, Final residual = 9.2743e-07, No Iterations 196
time step continuity errors : sum local = 5.71826e-10, global = -6.46623e-22, cumulative = -4.50168e-20
ICCG: Solving for p, Initial residual = 0.45407, Final residual = 8.34512e-07, No Iterations 196
time step continuity errors : sum local = 5.17042e-10, global = 1.43151e-19, cumulative = 9.81339e-20
ExecutionTime = 7.07 s ClockTime = 7 s

Time = 0.005

Mean and max Courant Numbers = 0.0597545 0.939777
BICCG: Solving for Ux, Initial residual = 0.0206086, Final residual = 5.9467e-06, No Iterations 14
BICCG: Solving for Uy, Initial residual = 0.0574585, Final residual = 8.81759e-06, No Iterations 14
ICCG: Solving for p, Initial residual = 0.41837, Final residual = 7.66076e-07, No Iterations 195
time step continuity errors : sum local = 3.66536e-10, global = 1.93771e-19, cumulative = 2.91905e-19
ICCG: Solving for p, Initial residual = 0.38174, Final residual = 8.99953e-07, No Iterations 194
time step continuity errors : sum local = 4.33183e-10, global = -1.20382e-19, cumulative = 1.71523e-19
ExecutionTime = 8.55 s ClockTime = 9 s

Time = 0.006

Mean and max Courant Numbers = 0.0648963 0.946156
BICCG: Solving for Ux, Initial residual = 0.0139187, Final residual = 7.52264e-06, No Iterations 13
BICCG: Solving for Uy, Initial residual = 0.0455918, Final residual = 7.22159e-06, No Iterations 14
ICCG: Solving for p, Initial residual = 0.253609, Final residual = 8.96376e-07, No Iterations 192
time step continuity errors : sum local = 3.92679e-10, global = 5.5118e-20, cumulative = 2.26641e-19
ICCG: Solving for p, Initial residual = 0.229742, Final residual = 8.75365e-07, No Iterations 192
time step continuity errors : sum local = 3.8378e-10, global = -8.76987e-20, cumulative = 1.38943e-19
ExecutionTime = 10.02 s ClockTime = 10 s

Time = 0.007

Mean and max Courant Numbers = 0.0692539 0.949465
BICCG: Solving for Ux, Initial residual = 0.0116939, Final residual = 6.09785e-06, No Iterations 13
BICCG: Solving for Uy, Initial residual = 0.0320625, Final residual = 7.43667e-06, No Iterations 13
ICCG: Solving for p, Initial residual = 0.180605, Final residual = 9.52495e-07, No Iterations 189
time step continuity errors : sum local = 3.83623e-10, global = -1.58405e-19, cumulative = -1.94624e-20
ICCG: Solving for p, Initial residual = 0.163707, Final residual = 9.64875e-07, No Iterations 188
time step continuity errors : sum local = 3.90012e-10, global = -7.23862e-20, cumulative = -9.18486e-20
ExecutionTime = 11.47 s ClockTime = 12 s

Time = 0.008

Mean and max Courant Numbers = 0.0729413 0.952808
BICCG: Solving for Ux, Initial residual = 0.00909364, Final residual = 8.65935e-06, No Iterations 12
BICCG: Solving for Uy, Initial residual = 0.0269566, Final residual = 6.7032e-06, No Iterations 13
ICCG: Solving for p, Initial residual = 0.109123, Final residual = 8.15401e-07, No Iterations 188
time step continuity errors : sum local = 3.10857e-10, global = 1.21822e-20, cumulative = -7.96664e-20
ICCG: Solving for p, Initial residual = 0.0988474, Final residual = 9.23239e-07, No Iterations 187
time step continuity errors : sum local = 3.52258e-10, global = 1.454e-19, cumulative = 6.57332e-20
ExecutionTime = 12.91 s ClockTime = 13 s

Time = 0.009

Mean and max Courant Numbers = 0.0761604 0.954918
BICCG: Solving for Ux, Initial residual = 0.00788061, Final residual = 8.02051e-06, No Iterations 12
BICCG: Solving for Uy, Initial residual = 0.0216089, Final residual = 7.38888e-06, No Iterations 12
ICCG: Solving for p, Initial residual = 0.0751862, Final residual = 8.52566e-07, No Iterations 184
time step continuity errors : sum local = 3.22218e-10, global = -2.50286e-19, cumulative = -1.84553e-19
ICCG: Solving for p, Initial residual = 0.0683289, Final residual = 8.71176e-07, No Iterations 184
time step continuity errors : sum local = 3.2978e-10, global = 2.72231e-20, cumulative = -1.5733e-19
ExecutionTime = 14.32 s ClockTime = 14 s

Time = 0.01

Mean and max Courant Numbers = 0.078984 0.956873
BICCG: Solving for Ux, Initial residual = 0.00663957, Final residual = 5.93408e-06, No Iterations 12
BICCG: Solving for Uy, Initial residual = 0.0188549, Final residual = 6.51374e-06, No Iterations 12
ICCG: Solving for p, Initial residual = 0.0476218, Final residual = 7.87959e-07, No Iterations 183
time step continuity errors : sum local = 2.92796e-10, global = -3.27487e-20, cumulative = -1.90078e-19
ICCG: Solving for p, Initial residual = 0.0435532, Final residual = 8.46472e-07, No Iterations 182
time step continuity errors : sum local = 3.1477e-10, global = 9.94609e-20, cumulative = -9.06174e-20
ExecutionTime = 15.72 s ClockTime = 16 s

Time = 0.011

Mean and max Courant Numbers = 0.0815052 0.958373
BICCG: Solving for Ux, Initial residual = 0.00586825, Final residual = 9.88343e-06, No Iterations 11
BICCG: Solving for Uy, Initial residual = 0.0161717, Final residual = 9.9482e-06, No Iterations 11
ICCG: Solving for p, Initial residual = 0.0356458, Final residual = 9.13337e-07, No Iterations 182
time step continuity errors : sum local = 3.37795e-10, global = -2.71539e-19, cumulative = -3.62156e-19
ICCG: Solving for p, Initial residual = 0.0329001, Final residual = 7.4762e-07, No Iterations 182
time step continuity errors : sum local = 2.7672e-10, global = -1.66423e-19, cumulative = -5.28579e-19
ExecutionTime = 17.13 s ClockTime = 17 s

Time = 0.012

Mean and max Courant Numbers = 0.0837613 0.959674
BICCG: Solving for Ux, Initial residual = 0.00515329, Final residual = 8.0906e-06, No Iterations 11
BICCG: Solving for Uy, Initial residual = 0.0144256, Final residual = 7.93775e-06, No Iterations 11
ICCG: Solving for p, Initial residual = 0.0248828, Final residual = 8.89308e-07, No Iterations 179
time step continuity errors : sum local = 3.28367e-10, global = -6.074e-20, cumulative = -5.89319e-19
ICCG: Solving for p, Initial residual = 0.0232373, Final residual = 9.85799e-07, No Iterations 178
time step continuity errors : sum local = 3.64155e-10, global = 1.95186e-19, cumulative = -3.94132e-19
ExecutionTime = 18.5 s ClockTime = 19 s

Time = 0.013

Mean and max Courant Numbers = 0.0858044 0.960774
BICCG: Solving for Ux, Initial residual = 0.00462194, Final residual = 7.26935e-06, No Iterations 11
BICCG: Solving for Uy, Initial residual = 0.012806, Final residual = 7.36052e-06, No Iterations 11
ICCG: Solving for p, Initial residual = 0.0208117, Final residual = 8.85416e-07, No Iterations 179
time step continuity errors : sum local = 3.25548e-10, global = 9.66412e-20, cumulative = -2.97491e-19
ICCG: Solving for p, Initial residual = 0.0196253, Final residual = 8.99491e-07, No Iterations 179
time step continuity errors : sum local = 3.30787e-10, global = -7.37018e-21, cumulative = -3.04861e-19
ExecutionTime = 19.88 s ClockTime = 20 s

Time = 0.014

Mean and max Courant Numbers = 0.0876578 0.961719
BICCG: Solving for Ux, Initial residual = 0.00415071, Final residual = 6.38942e-06, No Iterations 11
BICCG: Solving for Uy, Initial residual = 0.0115789, Final residual = 1.44374e+95, No Iterations 1001
ICCG: Solving for p, Initial residual = 1, Final residual = 9.6264e-07, No Iterations 189
time step continuity errors : sum local = 1.29818e+87, global = 5.93881e+76, cumulative = 5.93881e+76
ICCG: Solving for p, Initial residual = 0.934444, Final residual = 9.29122e-07, No Iterations 189
time step continuity errors : sum local = 1.32389e+87, global = 5.63113e+75, cumulative = 6.50193e+76
ExecutionTime = 25.46 s ClockTime = 26 s

Time = 0.015

Mean and max Courant Numbers = 7.04549e+93 3.68526e+94
BICCG: Solving for Ux: solution singularity
BICCG: Solving for Uy: solution singularity
ICCG: Solving for p, Initial residual = 1, Final residual = 1.40439, No Iterations 5001
time step continuity errors : sum local = 1.88237e+101, global = 1.44676e+86, cumulative = 1.44676e+86
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 3.57117e+109, global = -6.875e+92, cumulative = -6.87499e+92
ExecutionTime = 39.12 s ClockTime = 39 s

Time = 0.016

Mean and max Courant Numbers = 1.45147e+109 1.83744e+112
BICCG: Solving for Ux: solution singularity
BICCG: Solving for Uy: solution singularity
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 1.67439e+110, global = 2.48722e+94, cumulative = 2.41847e+94
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 1.26382e+111, global = -1.14441e+95, cumulative = -9.0256e+94
ExecutionTime = 39.47 s ClockTime = 40 s

Time = 0.017

Mean and max Courant Numbers = 7.14503e+110 8.90101e+113
BICCG: Solving for Ux: solution singularity
BICCG: Solving for Uy: solution singularity
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 4.72906e+112, global = -1.80693e+96, cumulative = -1.89719e+96
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 2.80161e+113, global = 7.47804e+96, cumulative = 5.58085e+96
ExecutionTime = 39.82 s ClockTime = 40 s

Time = 0.018

Mean and max Courant Numbers = 1.77155e+113 4.14651e+116
BICCG: Solving for Ux: solution singularity
BICCG: Solving for Uy: solution singularity
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 6.67211e+114, global = 1.87091e+99, cumulative = 1.87649e+99
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 2.64446e+115, global = 5.34057e+99, cumulative = 7.21706e+99
ExecutionTime = 40.16 s ClockTime = 40 s

Time = 0.019

Mean and max Courant Numbers = 1.73067e+115 2.43263e+118
BICCG: Solving for Ux: solution singularity
BICCG: Solving for Uy: solution singularity
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 1.03355e+116, global = -2.80081e+99, cumulative = 4.41625e+99
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 8.83257e+116, global = 6.15057e+100, cumulative = 6.5922e+100
ExecutionTime = 40.51 s ClockTime = 41 s

Time = 0.02

Mean and max Courant Numbers = 5.19846e+116 7.38682e+119
BICCG: Solving for Ux: solution singularity
BICCG: Solving for Uy: solution singularity
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 3.63936e+117, global = 3.23517e+101, cumulative = 3.89439e+101
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 8.63794e+118, global = 5.14892e+102, cumulative = 5.53836e+102
ExecutionTime = 40.85 s ClockTime = 41 s

Time = 0.021

Mean and max Courant Numbers = 4.54631e+118 1.97763e+122
BICCG: Solving for Ux: solution singularity
BICCG: Solving for Uy: solution singularity
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 2.1432e+120, global = 1.32031e+104, cumulative = 1.3757e+104
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 4.05285e+121, global = 1.8166e+105, cumulative = 1.95417e+105
ExecutionTime = 41.2 s ClockTime = 41 s

Time = 0.022

Mean and max Courant Numbers = 2.11858e+121 8.97658e+124
BICCG: Solving for Ux: solution singularity
BICCG: Solving for Uy: solution singularity
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 1.7996e+122, global = 8.53168e+105, cumulative = 1.04858e+106
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 2.79712e+124, global = -1.02687e+108, cumulative = -1.01639e+108
ExecutionTime = 41.55 s ClockTime = 42 s

Time = 0.023

Mean and max Courant Numbers = 1.43656e+124 8.42775e+127
BICCG: Solving for Ux: solution singularity
BICCG: Solving for Uy: solution singularity
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 1.90088e+125, global = -4.21083e+109, cumulative = -4.31247e+109
ICCG: Solving for p: solution singularity
time step continuity errors : sum local = 1.54769e+126, global = -6.58994e+109, cumulative = -1.09024e+110
ExecutionTime = 41.89 s ClockTime = 42 s

Time = 0.024

Mean and max Courant Numbers = 1.07014e+126 1.86495e+129
-----------------------------------------------------------------------------
One of the processes started by mpirun has exited with a nonzero exit
code. This typically indicates that the process finished in error.
If your process did not finish in error, be sure to include a "return
0" or "exit(0)" in your C code before exiting the application.

PID 32391 failed on node n0 (127.0.0.1) due to signal 15.
-----------------------------------------------------------------------------
madhavan@varese:~/OpenFOAM/madhavan-1.3/run/tutorials/icoFoam>