[lakeat]

I ran a cavity case with icoFoam in parallel using 32 cores (i.e. two nodes) twice, but I got different results. By saying "different", I mean the log file shows residual difference.

log-1:

Time = 2e-05

Courant Number mean: 0 max: 0
smoothSolver: Solving for Ux, Initial residual = 1, Final residual = 4.22814e-13, No Iterations 1000
smoothSolver: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
GAMG: Solving for p, Initial residual = 1, Final residual = 4.09553e-11, No Iterations 1000
time step continuity errors : sum local = 8.15199e-17, global = -3.71519e-21, cumulative = -3.71519e-21
GAMG: Solving for p, Initial residual = 0.717435, Final residual = 4.10012e-13, No Iterations 1000
time step continuity errors : sum local = 1.13405e-18, global = 2.562e-22, cumulative = -3.45899e-21
ExecutionTime = 20.07 s ClockTime = 22 s

log-2:

Time = 2e-05

Courant Number mean: 0 max: 0
smoothSolver: Solving for Ux, Initial residual = 1, Final residual = 4.22814e-13, No Iterations 1000
smoothSolver: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0
GAMG: Solving for p, Initial residual = 1, Final residual = 4.12736e-11, No Iterations 1000
time step continuity errors : sum local = 8.22913e-17, global = -2.10757e-21, cumulative = -2.10757e-21
GAMG: Solving for p, Initial residual = 0.717435, Final residual = 4.10737e-13, No Iterations 1000
time step continuity errors : sum local = 1.13587e-18, global = 2.40503e-23, cumulative = -2.08352e-21
ExecutionTime = 20.26 s ClockTime = 22 s

You see, starting from this first step, the p final residual starts to differ. I feel very confused about this, the nodes used in both simulations belong to the same cluster, a new bought cluster. I have tried to tighten the tolerance even to 1e-30, but it does not help. The relTol is set as zero.

Any random behavior in CPU? Or can we blame AMG solver for this? (scotch partitioning method is used) Any ideas?