| nguyenhung97 |
September 26, 2022 03:42 |
Divergence in parallel running after case reconstruction
Hi all!
I am running a simulation in a very complex geometry (human lung). Due to computer resources, I sometimes have to share clusters to other shared users. I constantly have to change my running processor number to 36, 72, 108, ... That's why I reconstruct and decompose the case again if I want to use different number of processor. My process is:
1. When someone asks for nodes, I stop my case (running in 108 cores) and reconstructPar -latestTime.
2. Then I decomposePar -latestTime the case to 36 cores and restart. This is when I face a divergence problem.
You can refer to the following results. After the second decomposing, my max parallel Courant number is 10.791 at time 0.1424 and keep doubling until divergence.
Quote:
Time = 0.1421s
Courant Number mean: 0.050839 max: 10.791
smoothSolver: Solving for Ux, Initial residual = 0.0138731, Final residual = 1.87886e-05, No Iterations 2
smoothSolver: Solving for Uy, Initial residual = 0.00704013, Final residual = 8.98269e-06, No Iterations 2
smoothSolver: Solving for Uz, Initial residual = 0.00496356, Final residual = 5.72542e-05, No Iterations 1
GAMG: Solving for p, Initial residual = 0.446619, Final residual = 5.04061e-05, No Iterations 13
GAMG: Solving for p, Initial residual = 0.182663, Final residual = 7.16055e-05, No Iterations 9
time step continuity errors : sum local = 6.20123e-08, global = -9.05988e-11, cumulative = -1.4971e-08
Current Inlet Flow Rate = -1.92092e-05
Current Inlet Flow Rate = -1.92092e-05
GAMG: Solving for p, Initial residual = 0.077038, Final residual = 8.26782e-05, No Iterations 8
GAMG: Solving for p, Initial residual = 0.0238152, Final residual = 8.96751e-05, No Iterations 4
time step continuity errors : sum local = 7.20512e-08, global = -1.06162e-09, cumulative = -1.60326e-08
Current Inlet Flow Rate = -1.92092e-05
GAMG: Solving for p, Initial residual = 0.0158456, Final residual = 8.69424e-05, No Iterations 6
GAMG: Solving for p, Initial residual = 0.00734297, Final residual = 8.69807e-05, No Iterations 3
time step continuity errors : sum local = 7.1083e-08, global = -3.9716e-10, cumulative = -1.64297e-08
Current Inlet Flow Rate = -1.92092e-05
GAMG: Solving for p, Initial residual = 0.00314104, Final residual = 5.60797e-05, No Iterations 3
GAMG: Solving for p, Initial residual = 0.00113054, Final residual = 7.12193e-07, No Iterations 7
time step continuity errors : sum local = 5.8211e-10, global = 4.59426e-11, cumulative = -1.63838e-08
Current Inlet Flow Rate = -1.92092e-05
ExecutionTime = 332.33 s ClockTime = 333 s
|
But when I try to run in one processor, it is only 5.50062 and keep running stably.
Quote:
Time = 0.1424s
Courant Number mean: 0.0509005 max: 5.50062
smoothSolver: Solving for Ux, Initial residual = 0.000300878, Final residual = 5.47137e-07, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 0.000388301, Final residual = 6.99841e-07, No Iterations 1
smoothSolver: Solving for Uz, Initial residual = 0.000328347, Final residual = 5.75167e-07, No Iterations 1
GAMG: Solving for p, Initial residual = 0.00066149, Final residual = 7.38323e-05, No Iterations 1
GAMG: Solving for p, Initial residual = 0.000248216, Final residual = 2.83435e-05, No Iterations 1
time step continuity errors : sum local = 1.15499e-08, global = 1.48838e-11, cumulative = -1.21772e-09
Current Inlet Flow Rate = -1.92498e-05
Current Inlet Flow Rate = -1.92498e-05
GAMG: Solving for p, Initial residual = 0.000128847, Final residual = 1.38249e-05, No Iterations 1
GAMG: Solving for p, Initial residual = 4.03098e-05, Final residual = 4.03098e-05, No Iterations 0
time step continuity errors : sum local = 1.64264e-08, global = -4.7946e-11, cumulative = -1.26567e-09
Current Inlet Flow Rate = -1.92498e-05
GAMG: Solving for p, Initial residual = 4.08922e-05, Final residual = 4.08922e-05, No Iterations 0
GAMG: Solving for p, Initial residual = 4.08922e-05, Final residual = 4.08922e-05, No Iterations 0
time step continuity errors : sum local = 1.66638e-08, global = -4.7946e-11, cumulative = -1.31361e-09
Current Inlet Flow Rate = -1.92498e-05
GAMG: Solving for p, Initial residual = 4.09558e-05, Final residual = 4.09558e-05, No Iterations 0
GAMG: Solving for p, Initial residual = 4.09558e-05, Final residual = 9.72793e-07, No Iterations 2
time step continuity errors : sum local = 3.96418e-10, global = 6.94878e-12, cumulative = -1.30666e-09
Current Inlet Flow Rate = -1.92498e-05
|
Note: I tried to reduce the time step to 1e-5 1e-6 (originally 1e-4) and still didn't solve the problem.
Anyone has any idea? Thanks. |
