
[Sponsors] 
December 9, 2018, 04:18 
decompose dependent solution

#1 
New Member
behzad
Join Date: Dec 2010
Posts: 15
Rep Power: 15 
Hi, im trying to solve a dynamic mesh with AMI similar to the propeller tutorial. The simulation works fine with 8 cores n=(2 2 2) but fails with 48 cores. The 48 core simulation works better if n=(24 2 1) than (4 3 4) but fails anyway. Here is the end of log:
PIMPLE: Iteration 1 AMI: Creating addressing and weights between 23228 source faces and 17352 target faces AMI: Patch source sum(weights) min/max/average = 0.640126, 2.02482, 1.00383 AMI: Patch target sum(weights) min/max/average = 0.00764601, 1.60293, 1.0027 smoothSolver: Solving for Ux, Initial residual = 0.00314316, Final residual = 0.000161422, No Iterations 1 smoothSolver: Solving for Uy, Initial residual = 0.00336032, Final residual = 0.000150761, No Iterations 1 smoothSolver: Solving for Uz, Initial residual = 0.00311104, Final residual = 0.000151876, No Iterations 1 GAMG: Solving for p, Initial residual = 0.237415, Final residual = 0.00205794, No Iterations 4 GAMG: Solving for p, Initial residual = 0.0215434, Final residual = 0.000177966, No Iterations 5 time step continuity errors : sum local = 4.00436e07, global = 1.55868e07, cumulative = 8.40485e07 GAMG: Solving for p, Initial residual = 0.066189, Final residual = 0.00057829, No Iterations 3 GAMG: Solving for p, Initial residual = 0.00649573, Final residual = 3.07504e05, No Iterations 7 time step continuity errors : sum local = 7.23223e08, global = 1.27801e08, cumulative = 8.53265e07 GAMG: Solving for p, Initial residual = 0.0564967, Final residual = 0.00054468, No Iterations 3 GAMG: Solving for p, Initial residual = 0.00612875, Final residual = 6.4775e07, No Iterations 17 time step continuity errors : sum local = 1.5343e09, global = 5.26857e10, cumulative = 8.53792e07 smoothSolver: Solving for omega, Initial residual = 0.000311639, Final residual = 1.43584e05, No Iterations 1 smoothSolver: Solving for k, Initial residual = 0.000780975, Final residual = 4.56978e05, No Iterations 1 PIMPLE: Iteration 2 smoothSolver: Solving for Ux, Initial residual = 0.00162772, Final residual = 8.84785e05, No Iterations 1 smoothSolver: Solving for Uy, Initial residual = 0.00156759, Final residual = 8.06781e05, No Iterations 1 smoothSolver: Solving for Uz, Initial residual = 0.00162965, Final residual = 8.5398e05, No Iterations 1 GAMG: Solving for p, Initial residual = 0.0524565, Final residual = 0.000495011, No Iterations 4 GAMG: Solving for p, Initial residual = 0.0115723, Final residual = 0.000110676, No Iterations 5 time step continuity errors : sum local = 2.13065e07, global = 6.79215e08, cumulative = 7.8587e07 GAMG: Solving for p, Initial residual = 0.0228772, Final residual = 0.000101613, No Iterations 5 GAMG: Solving for p, Initial residual = 0.00414592, Final residual = 4.12058e05, No Iterations 4 time step continuity errors : sum local = 7.76397e08, global = 2.81646e08, cumulative = 7.57705e07 GAMG: Solving for p, Initial residual = 0.0198464, Final residual = 0.000194728, No Iterations 4 GAMG: Solving for p, Initial residual = 0.00385169, Final residual = 8.67885e07, No Iterations 16 time step continuity errors : sum local = 1.63919e09, global = 3.63495e10, cumulative = 7.57342e07 smoothSolver: Solving for omega, Initial residual = 0.000124945, Final residual = 6.16685e06, No Iterations 1 smoothSolver: Solving for k, Initial residual = 0.000327625, Final residual = 1.99492e05, No Iterations 1 PIMPLE: Iteration 3 smoothSolver: Solving for Ux, Initial residual = 0.00045945, Final residual = 2.53229e05, No Iterations 1 smoothSolver: Solving for Uy, Initial residual = 0.000640314, Final residual = 2.86446e05, No Iterations 1 smoothSolver: Solving for Uz, Initial residual = 0.000455598, Final residual = 2.39908e05, No Iterations 1 GAMG: Solving for p, Initial residual = 0.0255996, Final residual = 0.000100523, No Iterations 6 GAMG: Solving for p, Initial residual = 0.0082641, Final residual = 3.09009e05, No Iterations 4 time step continuity errors : sum local = 5.45629e08, global = 1.7299e08, cumulative = 7.40043e07 GAMG: Solving for p, Initial residual = 0.0109177, Final residual = 9.58104e05, No Iterations 4 GAMG: Solving for p, Initial residual = 0.00272103, Final residual = 1.16692e05, No Iterations 4 time step continuity errors : sum local = 2.05735e08, global = 4.27736e09, cumulative = 7.35766e07 GAMG: Solving for p, Initial residual = 0.00946239, Final residual = 7.99641e05, No Iterations 5 GAMG: Solving for p, Initial residual = 0.00267032, Final residual = 4.94202e07, No Iterations 11 time step continuity errors : sum local = 8.68681e10, global = 1.15411e10, cumulative = 7.3565e07 smoothSolver: Solving for omega, Initial residual = 5.18854e05, Final residual = 2.73978e06, No Iterations 1 smoothSolver: Solving for k, Initial residual = 0.000140115, Final residual = 8.89688e06, No Iterations 1 PIMPLE: Iteration 4 smoothSolver: Solving for Ux, Initial residual = 0.000125401, Final residual = 7.06938e06, No Iterations 1 smoothSolver: Solving for Uy, Initial residual = 0.000478161, Final residual = 2.31364e05, No Iterations 1 smoothSolver: Solving for Uz, Initial residual = 0.000125401, Final residual = 7.60537e06, No Iterations 1 GAMG: Solving for p, Initial residual = 0.0104569, Final residual = 7.24393e05, No Iterations 4 GAMG: Solving for p, Initial residual = 0.00217912, Final residual = 1.04679e05, No Iterations 4 time step continuity errors : sum local = 1.86154e08, global = 5.8645e09, cumulative = 7.29786e07 GAMG: Solving for p, Initial residual = 0.00436904, Final residual = 1.98028e05, No Iterations 4 GAMG: Solving for p, Initial residual = 0.000754537, Final residual = 4.36505e06, No Iterations 3 time step continuity errors : sum local = 7.75789e09, global = 4.332e10, cumulative = 7.30219e07 GAMG: Solving for p, Initial residual = 0.00374923, Final residual = 2.00379e05, No Iterations 4 GAMG: Solving for p, Initial residual = 0.000705467, Final residual = 6.62802e07, No Iterations 7 time step continuity errors : sum local = 1.17908e09, global = 3.78833e10, cumulative = 7.2984e07 smoothSolver: Solving for omega, Initial residual = 2.24531e05, Final residual = 1.25385e06, No Iterations 1 smoothSolver: Solving for k, Initial residual = 6.10787e05, Final residual = 4.03104e06, No Iterations 1 PIMPLE: Iteration 5 smoothSolver: Solving for Ux, Initial residual = 7.17316e05, Final residual = 4.35357e06, No Iterations 1 smoothSolver: Solving for Uy, Initial residual = 0.000222978, Final residual = 1.23042e05, No Iterations 1 smoothSolver: Solving for Uz, Initial residual = 7.08052e05, Final residual = 4.56745e06, No Iterations 1 GAMG: Solving for p, Initial residual = 0.00390302, Final residual = 3.13145e05, No Iterations 3 GAMG: Solving for p, Initial residual = 0.000409693, Final residual = 8.55303e06, No Iterations 3 time step continuity errors : sum local = 1.53258e08, global = 3.93791e09, cumulative = 7.33778e07 GAMG: Solving for p, Initial residual = 0.00171579, Final residual = 7.02324e06, No Iterations 4 GAMG: Solving for p, Initial residual = 0.000181272, Final residual = 8.61135e06, No Iterations 1 time step continuity errors : sum local = 1.54803e08, global = 6.64287e09, cumulative = 7.40421e07 GAMG: Solving for p, Initial residual = 0.00144378, Final residual = 6.86926e06, No Iterations 4 GAMG: Solving for p, Initial residual = 0.000162935, Final residual = 5.52017e07, No Iterations 7 time step continuity errors : sum local = 9.92553e10, global = 4.32425e10, cumulative = 7.40853e07 smoothSolver: Solving for omega, Initial residual = 1.02472e05, Final residual = 5.91612e07, No Iterations 1 smoothSolver: Solving for k, Initial residual = 2.72085e05, Final residual = 1.85781e06, No Iterations 1 PIMPLE: Iteration 6 smoothSolver: Solving for Ux, Initial residual = 2.65473e05, Final residual = 1.69321e06, No Iterations 1 smoothSolver: Solving for Uy, Initial residual = 6.88e05, Final residual = 4.54147e06, No Iterations 1 smoothSolver: Solving for Uz, Initial residual = 2.62085e05, Final residual = 1.77625e06, No Iterations 1 GAMG: Solving for p, Initial residual = 0.0017311, Final residual = 8.74477e06, No Iterations 5 GAMG: Solving for p, Initial residual = 0.000357266, Final residual = 4.89035e06, No Iterations 3 time step continuity errors : sum local = 8.82443e09, global = 3.3515e09, cumulative = 7.44205e07 GAMG: Solving for p, Initial residual = 0.000761454, Final residual = 9.82832e06, No Iterations 4 GAMG: Solving for p, Initial residual = 0.000133049, Final residual = 7.91594e06, No Iterations 1 time step continuity errors : sum local = 1.42964e08, global = 4.17428e09, cumulative = 7.48379e07 GAMG: Solving for p, Initial residual = 0.000644368, Final residual = 8.84289e06, No Iterations 4 GAMG: Solving for p, Initial residual = 0.000126339, Final residual = 9.66247e07, No Iterations 5 time step continuity errors : sum local = 1.74511e09, global = 7.31198e10, cumulative = 7.4911e07 smoothSolver: Solving for omega, Initial residual = 4.97623e06, Final residual = 2.86261e07, No Iterations 1 smoothSolver: Solving for k, Initial residual = 1.24484e05, Final residual = 8.72288e07, No Iterations 1 PIMPLE: Iteration 7 smoothSolver: Solving for Ux, Initial residual = 8.38714e06, Final residual = 5.55043e07, No Iterations 1 smoothSolver: Solving for Uy, Initial residual = 3.02269e05, Final residual = 1.62476e06, No Iterations 1 smoothSolver: Solving for Uz, Initial residual = 8.45018e06, Final residual = 6.00375e07, No Iterations 1 GAMG: Solving for p, Initial residual = 0.000741519, Final residual = 3.80987e06, No Iterations 5 GAMG: Solving for p, Initial residual = 0.000143734, Final residual = 8.36365e06, No Iterations 1 time step continuity errors : sum local = 1.51125e08, global = 2.89613e09, cumulative = 7.52006e07 GAMG: Solving for p, Initial residual = 0.000328853, Final residual = 7.51672e06, No Iterations 2 GAMG: Solving for p, Initial residual = 5.23058e05, Final residual = 6.16932e06, No Iterations 1 time step continuity errors : sum local = 1.11445e08, global = 1.51966e09, cumulative = 7.53526e07 GAMG: Solving for p, Initial residual = 0.000281089, Final residual = 7.02204e06, No Iterations 2 GAMG: Solving for p, Initial residual = 4.92908e05, Final residual = 4.39403e07, No Iterations 7 time step continuity errors : sum local = 7.93722e10, global = 1.68259e10, cumulative = 7.53694e07 smoothSolver: Solving for omega, Initial residual = 2.82637e06, Final residual = 1.47576e07, No Iterations 1 smoothSolver: Solving for k, Initial residual = 5.99283e06, Final residual = 4.22433e07, No Iterations 1 PIMPLE: Iteration 8 smoothSolver: Solving for Ux, Initial residual = 6.61349e06, Final residual = 7.97449e07, No Iterations 1 smoothSolver: Solving for Uy, Initial residual = 2.229e05, Final residual = 3.01211e07, No Iterations 2 smoothSolver: Solving for Uz, Initial residual = 6.64923e06, Final residual = 8.21615e07, No Iterations 1 GAMG: Solving for p, Initial residual = 0.155916, Final residual = 0.00105258, No Iterations 2 GAMG: Solving for p, Initial residual = 0.00890627, Final residual = 7.44728e05, No Iterations 3 time step continuity errors : sum local = 1.63909e07, global = 2.67794e08, cumulative = 7.26915e07 GAMG: Solving for p, Initial residual = 0.0401865, Final residual = 0.000183517, No Iterations 2 GAMG: Solving for p, Initial residual = 0.00172845, Final residual = 1.08675e05, No Iterations 4 time step continuity errors : sum local = 2.24388e08, global = 9.16245e09, cumulative = 7.17752e07 GAMG: Solving for p, Initial residual = 0.0135472, Final residual = 4.87362e05, No Iterations 2 GAMG: Solving for p, Initial residual = 0.000630845, Final residual = 7.14904e07, No Iterations 9 time step continuity errors : sum local = 1.44197e09, global = 4.89227e10, cumulative = 7.17263e07 smoothSolver: Solving for omega, Initial residual = 1.91752e05, Final residual = 1.05424e07, No Iterations 2 smoothSolver: Solving for k, Initial residual = 2.97993e05, Final residual = 2.37212e07, No Iterations 2 ExecutionTime = 141.79 s ClockTime = 145 s forces forces write: sum of forces: pressure : (0.0147785 2.259 0.0435249) viscous : (0.00239986 0.000924774 0.00203156) porous : (0 0 0) sum of moments: pressure : (0.000257647 0.0349593 0.00154703) viscous : (3.86898e05 0.00114474 5.58975e05) porous : (0 0 0) Courant Number mean: 0.13151 max: 4.97833 deltaT = 0.000143081 Time = 0.2255645 PIMPLE: Iteration 1 AMI: Creating addressing and weights between 23228 source faces and 17352 target faces AMI: Patch source sum(weights) min/max/average = 0.652674, 2.13182, 1.00375 AMI: Patch target sum(weights) min/max/average = 0, 1.68866, 1.00261 smoothSolver: Solving for Ux, Initial residual = 0.00314458, Final residual = 0.000161758, No Iterations 1 smoothSolver: Solving for Uy, Initial residual = 0.00324429, Final residual = 0.000149037, No Iterations 1 smoothSolver: Solving for Uz, Initial residual = 0.00310572, Final residual = 0.000151366, No Iterations 1 [11] #0 Foam::error:rintStack(Foam::Ostream&) at ??:? [11] #1 Foam::sigFpe::sigHandler(int) at ??:? [11] #2 ? in "/usr/lib64/libc.so.6" [11] #3 Foam::divide(Foam::Field<double>&, double const&, Foam::UList<double> const&) at ??:? [11] #4 Foam::tmp<Foam::GeometricField<double, Foam::volMesh> > Foam:perator/<Foam::fvPatchField, Foam::volMesh>(Foam::dimensioned<double> const&, Foam::tmp<Foam::GeometricField<double, Foam::volMesh> > const&) at ??:? [11] #5 Foam::tmp<Foam::GeometricField<double, Foam::volMesh> > Foam:perator/<Foam::fvPatchField, Foam::volMesh>(double const&, Foam::tmp<Foam::GeometricField<double, Foam::volMesh> > const&) at ??:? [11] #6 ? at ??:? [11] #7 __libc_start_main in "/usr/lib64/libc.so.6" [11] #8 ? at ??:? ================================================== ================================= = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 9823 RUNNING AT s12p29 = EXIT CODE: 8 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ================================================== ================================= Intel(R) MPI Library troubleshooting guide: https://software.intel.com/node/561764 ================================================== ================================= The mesh is Ok. I ran checkMesh with no errors. 

December 10, 2018, 05:17 

#2  
Member
Pekka Pasanen
Join Date: Feb 2012
Location: Finland
Posts: 87
Rep Power: 14 
I have two suggestions. First I would try to improve the match between rotating and nonrotating surfaces, because this could be better:
Quote:
Secondly, there is an option to keep AMI patches intact when running decomposePar. You will need to create faceSets out of cyclicAMI patches to do this and then use the created faceSets with singleProcessorFaceSets in decomposeParDict. See example in caseDicts file: https://github.com/OpenFOAM/OpenFOAM...composeParDict I don't exactly why the case is running with less decomposition, but I suspect that when you split it further, some of the bad AMI weights gets distributed in a numerically unstable way and it crashes. 

December 10, 2018, 06:49 

#3 
New Member
behzad
Join Date: Dec 2010
Posts: 15
Rep Power: 15 
how can i improve the interface matching?!


December 10, 2018, 09:36 

#4 
Member
Pekka Pasanen
Join Date: Feb 2012
Location: Finland
Posts: 87
Rep Power: 14 
Well that is hard to say when I don't know anything about your case or mesh..
In general, make sure that the rotational surfaces match as closely as possibly in geometry (use good quality .stl etc). Then try increasing mesh density on the interface, I also found out that using one boundary layer in snappyHexMesh (even with same cell size as inner mesh) makes the mesh more conforming because it extrudes from the rotational surfaces. Also make sure that edges are properly snapped. 

Thread Tools  Search this Thread 
Display Modes  


Similar Threads  
Thread  Thread Starter  Forum  Replies  Last Post 
Steady state solution as an initial condition for a transient problem  adnanakhtar  FLUENT  7  November 25, 2016 05:16 
grid dependancy  gueynard a.  Main CFD Forum  19  June 27, 2014 21:22 
boundary conditions dependent by the numerical solution  c120613  CFX  3  September 22, 2011 15:31 
reactingFoam solution is strongly dependent on time step size  andras  OpenFOAM Bugs  3  July 13, 2011 10:28 
Wall functions  Abhijit Tilak  Main CFD Forum  6  February 5, 1999 01:16 