CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM

decompose dependent solution

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   December 9, 2018, 04:18
Default decompose dependent solution
  #1
New Member
 
behzad
Join Date: Dec 2010
Posts: 15
Rep Power: 12
arionfard is on a distinguished road
Hi, im trying to solve a dynamic mesh with AMI similar to the propeller tutorial. The simulation works fine with 8 cores n=(2 2 2) but fails with 48 cores. The 48 core simulation works better if n=(24 2 1) than (4 3 4) but fails anyway. Here is the end of log:

PIMPLE: Iteration 1
AMI: Creating addressing and weights between 23228 source faces and 17352 target faces
AMI: Patch source sum(weights) min/max/average = 0.640126, 2.02482, 1.00383
AMI: Patch target sum(weights) min/max/average = 0.00764601, 1.60293, 1.0027
smoothSolver: Solving for Ux, Initial residual = 0.00314316, Final residual = 0.000161422, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 0.00336032, Final residual = 0.000150761, No Iterations 1
smoothSolver: Solving for Uz, Initial residual = 0.00311104, Final residual = 0.000151876, No Iterations 1
GAMG: Solving for p, Initial residual = 0.237415, Final residual = 0.00205794, No Iterations 4
GAMG: Solving for p, Initial residual = 0.0215434, Final residual = 0.000177966, No Iterations 5
time step continuity errors : sum local = 4.00436e-07, global = 1.55868e-07, cumulative = 8.40485e-07
GAMG: Solving for p, Initial residual = 0.066189, Final residual = 0.00057829, No Iterations 3
GAMG: Solving for p, Initial residual = 0.00649573, Final residual = 3.07504e-05, No Iterations 7
time step continuity errors : sum local = 7.23223e-08, global = 1.27801e-08, cumulative = 8.53265e-07
GAMG: Solving for p, Initial residual = 0.0564967, Final residual = 0.00054468, No Iterations 3
GAMG: Solving for p, Initial residual = 0.00612875, Final residual = 6.4775e-07, No Iterations 17
time step continuity errors : sum local = 1.5343e-09, global = 5.26857e-10, cumulative = 8.53792e-07
smoothSolver: Solving for omega, Initial residual = 0.000311639, Final residual = 1.43584e-05, No Iterations 1
smoothSolver: Solving for k, Initial residual = 0.000780975, Final residual = 4.56978e-05, No Iterations 1
PIMPLE: Iteration 2
smoothSolver: Solving for Ux, Initial residual = 0.00162772, Final residual = 8.84785e-05, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 0.00156759, Final residual = 8.06781e-05, No Iterations 1
smoothSolver: Solving for Uz, Initial residual = 0.00162965, Final residual = 8.5398e-05, No Iterations 1
GAMG: Solving for p, Initial residual = 0.0524565, Final residual = 0.000495011, No Iterations 4
GAMG: Solving for p, Initial residual = 0.0115723, Final residual = 0.000110676, No Iterations 5
time step continuity errors : sum local = 2.13065e-07, global = -6.79215e-08, cumulative = 7.8587e-07
GAMG: Solving for p, Initial residual = 0.0228772, Final residual = 0.000101613, No Iterations 5
GAMG: Solving for p, Initial residual = 0.00414592, Final residual = 4.12058e-05, No Iterations 4
time step continuity errors : sum local = 7.76397e-08, global = -2.81646e-08, cumulative = 7.57705e-07
GAMG: Solving for p, Initial residual = 0.0198464, Final residual = 0.000194728, No Iterations 4
GAMG: Solving for p, Initial residual = 0.00385169, Final residual = 8.67885e-07, No Iterations 16
time step continuity errors : sum local = 1.63919e-09, global = -3.63495e-10, cumulative = 7.57342e-07
smoothSolver: Solving for omega, Initial residual = 0.000124945, Final residual = 6.16685e-06, No Iterations 1
smoothSolver: Solving for k, Initial residual = 0.000327625, Final residual = 1.99492e-05, No Iterations 1
PIMPLE: Iteration 3
smoothSolver: Solving for Ux, Initial residual = 0.00045945, Final residual = 2.53229e-05, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 0.000640314, Final residual = 2.86446e-05, No Iterations 1
smoothSolver: Solving for Uz, Initial residual = 0.000455598, Final residual = 2.39908e-05, No Iterations 1
GAMG: Solving for p, Initial residual = 0.0255996, Final residual = 0.000100523, No Iterations 6
GAMG: Solving for p, Initial residual = 0.0082641, Final residual = 3.09009e-05, No Iterations 4
time step continuity errors : sum local = 5.45629e-08, global = -1.7299e-08, cumulative = 7.40043e-07
GAMG: Solving for p, Initial residual = 0.0109177, Final residual = 9.58104e-05, No Iterations 4
GAMG: Solving for p, Initial residual = 0.00272103, Final residual = 1.16692e-05, No Iterations 4
time step continuity errors : sum local = 2.05735e-08, global = -4.27736e-09, cumulative = 7.35766e-07
GAMG: Solving for p, Initial residual = 0.00946239, Final residual = 7.99641e-05, No Iterations 5
GAMG: Solving for p, Initial residual = 0.00267032, Final residual = 4.94202e-07, No Iterations 11
time step continuity errors : sum local = 8.68681e-10, global = -1.15411e-10, cumulative = 7.3565e-07
smoothSolver: Solving for omega, Initial residual = 5.18854e-05, Final residual = 2.73978e-06, No Iterations 1
smoothSolver: Solving for k, Initial residual = 0.000140115, Final residual = 8.89688e-06, No Iterations 1
PIMPLE: Iteration 4
smoothSolver: Solving for Ux, Initial residual = 0.000125401, Final residual = 7.06938e-06, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 0.000478161, Final residual = 2.31364e-05, No Iterations 1
smoothSolver: Solving for Uz, Initial residual = 0.000125401, Final residual = 7.60537e-06, No Iterations 1
GAMG: Solving for p, Initial residual = 0.0104569, Final residual = 7.24393e-05, No Iterations 4
GAMG: Solving for p, Initial residual = 0.00217912, Final residual = 1.04679e-05, No Iterations 4
time step continuity errors : sum local = 1.86154e-08, global = -5.8645e-09, cumulative = 7.29786e-07
GAMG: Solving for p, Initial residual = 0.00436904, Final residual = 1.98028e-05, No Iterations 4
GAMG: Solving for p, Initial residual = 0.000754537, Final residual = 4.36505e-06, No Iterations 3
time step continuity errors : sum local = 7.75789e-09, global = 4.332e-10, cumulative = 7.30219e-07
GAMG: Solving for p, Initial residual = 0.00374923, Final residual = 2.00379e-05, No Iterations 4
GAMG: Solving for p, Initial residual = 0.000705467, Final residual = 6.62802e-07, No Iterations 7
time step continuity errors : sum local = 1.17908e-09, global = -3.78833e-10, cumulative = 7.2984e-07
smoothSolver: Solving for omega, Initial residual = 2.24531e-05, Final residual = 1.25385e-06, No Iterations 1
smoothSolver: Solving for k, Initial residual = 6.10787e-05, Final residual = 4.03104e-06, No Iterations 1
PIMPLE: Iteration 5
smoothSolver: Solving for Ux, Initial residual = 7.17316e-05, Final residual = 4.35357e-06, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 0.000222978, Final residual = 1.23042e-05, No Iterations 1
smoothSolver: Solving for Uz, Initial residual = 7.08052e-05, Final residual = 4.56745e-06, No Iterations 1
GAMG: Solving for p, Initial residual = 0.00390302, Final residual = 3.13145e-05, No Iterations 3
GAMG: Solving for p, Initial residual = 0.000409693, Final residual = 8.55303e-06, No Iterations 3
time step continuity errors : sum local = 1.53258e-08, global = 3.93791e-09, cumulative = 7.33778e-07
GAMG: Solving for p, Initial residual = 0.00171579, Final residual = 7.02324e-06, No Iterations 4
GAMG: Solving for p, Initial residual = 0.000181272, Final residual = 8.61135e-06, No Iterations 1
time step continuity errors : sum local = 1.54803e-08, global = 6.64287e-09, cumulative = 7.40421e-07
GAMG: Solving for p, Initial residual = 0.00144378, Final residual = 6.86926e-06, No Iterations 4
GAMG: Solving for p, Initial residual = 0.000162935, Final residual = 5.52017e-07, No Iterations 7
time step continuity errors : sum local = 9.92553e-10, global = 4.32425e-10, cumulative = 7.40853e-07
smoothSolver: Solving for omega, Initial residual = 1.02472e-05, Final residual = 5.91612e-07, No Iterations 1
smoothSolver: Solving for k, Initial residual = 2.72085e-05, Final residual = 1.85781e-06, No Iterations 1
PIMPLE: Iteration 6
smoothSolver: Solving for Ux, Initial residual = 2.65473e-05, Final residual = 1.69321e-06, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 6.88e-05, Final residual = 4.54147e-06, No Iterations 1
smoothSolver: Solving for Uz, Initial residual = 2.62085e-05, Final residual = 1.77625e-06, No Iterations 1
GAMG: Solving for p, Initial residual = 0.0017311, Final residual = 8.74477e-06, No Iterations 5
GAMG: Solving for p, Initial residual = 0.000357266, Final residual = 4.89035e-06, No Iterations 3
time step continuity errors : sum local = 8.82443e-09, global = 3.3515e-09, cumulative = 7.44205e-07
GAMG: Solving for p, Initial residual = 0.000761454, Final residual = 9.82832e-06, No Iterations 4
GAMG: Solving for p, Initial residual = 0.000133049, Final residual = 7.91594e-06, No Iterations 1
time step continuity errors : sum local = 1.42964e-08, global = 4.17428e-09, cumulative = 7.48379e-07
GAMG: Solving for p, Initial residual = 0.000644368, Final residual = 8.84289e-06, No Iterations 4
GAMG: Solving for p, Initial residual = 0.000126339, Final residual = 9.66247e-07, No Iterations 5
time step continuity errors : sum local = 1.74511e-09, global = 7.31198e-10, cumulative = 7.4911e-07
smoothSolver: Solving for omega, Initial residual = 4.97623e-06, Final residual = 2.86261e-07, No Iterations 1
smoothSolver: Solving for k, Initial residual = 1.24484e-05, Final residual = 8.72288e-07, No Iterations 1
PIMPLE: Iteration 7
smoothSolver: Solving for Ux, Initial residual = 8.38714e-06, Final residual = 5.55043e-07, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 3.02269e-05, Final residual = 1.62476e-06, No Iterations 1
smoothSolver: Solving for Uz, Initial residual = 8.45018e-06, Final residual = 6.00375e-07, No Iterations 1
GAMG: Solving for p, Initial residual = 0.000741519, Final residual = 3.80987e-06, No Iterations 5
GAMG: Solving for p, Initial residual = 0.000143734, Final residual = 8.36365e-06, No Iterations 1
time step continuity errors : sum local = 1.51125e-08, global = 2.89613e-09, cumulative = 7.52006e-07
GAMG: Solving for p, Initial residual = 0.000328853, Final residual = 7.51672e-06, No Iterations 2
GAMG: Solving for p, Initial residual = 5.23058e-05, Final residual = 6.16932e-06, No Iterations 1
time step continuity errors : sum local = 1.11445e-08, global = 1.51966e-09, cumulative = 7.53526e-07
GAMG: Solving for p, Initial residual = 0.000281089, Final residual = 7.02204e-06, No Iterations 2
GAMG: Solving for p, Initial residual = 4.92908e-05, Final residual = 4.39403e-07, No Iterations 7
time step continuity errors : sum local = 7.93722e-10, global = 1.68259e-10, cumulative = 7.53694e-07
smoothSolver: Solving for omega, Initial residual = 2.82637e-06, Final residual = 1.47576e-07, No Iterations 1
smoothSolver: Solving for k, Initial residual = 5.99283e-06, Final residual = 4.22433e-07, No Iterations 1
PIMPLE: Iteration 8
smoothSolver: Solving for Ux, Initial residual = 6.61349e-06, Final residual = 7.97449e-07, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 2.229e-05, Final residual = 3.01211e-07, No Iterations 2
smoothSolver: Solving for Uz, Initial residual = 6.64923e-06, Final residual = 8.21615e-07, No Iterations 1
GAMG: Solving for p, Initial residual = 0.155916, Final residual = 0.00105258, No Iterations 2
GAMG: Solving for p, Initial residual = 0.00890627, Final residual = 7.44728e-05, No Iterations 3
time step continuity errors : sum local = 1.63909e-07, global = -2.67794e-08, cumulative = 7.26915e-07
GAMG: Solving for p, Initial residual = 0.0401865, Final residual = 0.000183517, No Iterations 2
GAMG: Solving for p, Initial residual = 0.00172845, Final residual = 1.08675e-05, No Iterations 4
time step continuity errors : sum local = 2.24388e-08, global = -9.16245e-09, cumulative = 7.17752e-07
GAMG: Solving for p, Initial residual = 0.0135472, Final residual = 4.87362e-05, No Iterations 2
GAMG: Solving for p, Initial residual = 0.000630845, Final residual = 7.14904e-07, No Iterations 9
time step continuity errors : sum local = 1.44197e-09, global = -4.89227e-10, cumulative = 7.17263e-07
smoothSolver: Solving for omega, Initial residual = 1.91752e-05, Final residual = 1.05424e-07, No Iterations 2
smoothSolver: Solving for k, Initial residual = 2.97993e-05, Final residual = 2.37212e-07, No Iterations 2
ExecutionTime = 141.79 s ClockTime = 145 s

forces forces write:
sum of forces:
pressure : (-0.0147785 2.259 -0.0435249)
viscous : (-0.00239986 0.000924774 0.00203156)
porous : (0 0 0)
sum of moments:
pressure : (-0.000257647 -0.0349593 -0.00154703)
viscous : (3.86898e-05 -0.00114474 -5.58975e-05)
porous : (0 0 0)

Courant Number mean: 0.13151 max: 4.97833
deltaT = 0.000143081
Time = 0.2255645

PIMPLE: Iteration 1
AMI: Creating addressing and weights between 23228 source faces and 17352 target faces
AMI: Patch source sum(weights) min/max/average = 0.652674, 2.13182, 1.00375
AMI: Patch target sum(weights) min/max/average = 0, 1.68866, 1.00261
smoothSolver: Solving for Ux, Initial residual = 0.00314458, Final residual = 0.000161758, No Iterations 1
smoothSolver: Solving for Uy, Initial residual = 0.00324429, Final residual = 0.000149037, No Iterations 1
smoothSolver: Solving for Uz, Initial residual = 0.00310572, Final residual = 0.000151366, No Iterations 1
[11] #0 Foam::error:rintStack(Foam::Ostream&) at ??:?
[11] #1 Foam::sigFpe::sigHandler(int) at ??:?
[11] #2 ? in "/usr/lib64/libc.so.6"
[11] #3 Foam::divide(Foam::Field<double>&, double const&, Foam::UList<double> const&) at ??:?
[11] #4 Foam::tmp<Foam::GeometricField<double, Foam::volMesh> > Foam:perator/<Foam::fvPatchField, Foam::volMesh>(Foam::dimensioned<double> const&, Foam::tmp<Foam::GeometricField<double, Foam::volMesh> > const&) at ??:?
[11] #5 Foam::tmp<Foam::GeometricField<double, Foam::volMesh> > Foam:perator/<Foam::fvPatchField, Foam::volMesh>(double const&, Foam::tmp<Foam::GeometricField<double, Foam::volMesh> > const&) at ??:?
[11] #6 ? at ??:?
[11] #7 __libc_start_main in "/usr/lib64/libc.so.6"
[11] #8 ? at ??:?

================================================== =================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 9823 RUNNING AT s12p29
= EXIT CODE: 8
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
================================================== =================================
Intel(R) MPI Library troubleshooting guide:
https://software.intel.com/node/561764
================================================== =================================



The mesh is Ok. I ran checkMesh with no errors.
arionfard is offline   Reply With Quote

Old   December 10, 2018, 05:17
Default
  #2
Member
 
Pekka Pasanen
Join Date: Feb 2012
Location: Finland
Posts: 87
Rep Power: 11
zordiack is on a distinguished road
I have two suggestions. First I would try to improve the match between rotating and non-rotating surfaces, because this could be better:

Quote:
AMI: Creating addressing and weights between 23228 source faces and 17352 target faces
AMI: Patch source sum(weights) min/max/average = 0.640126, 2.02482, 1.00383
AMI: Patch target sum(weights) min/max/average = 0.00764601, 1.60293, 1.0027
AMI simulations I have done before had problems with such low minimum and high maximum weights, although average values seem ok. So try to improve interface matching (surfaces and/or mesh).

Secondly, there is an option to keep AMI patches intact when running decomposePar. You will need to create faceSets out of cyclicAMI patches to do this and then use the created faceSets with singleProcessorFaceSets in decomposeParDict. See example in caseDicts file:

https://github.com/OpenFOAM/OpenFOAM...composeParDict

I don't exactly why the case is running with less decomposition, but I suspect that when you split it further, some of the bad AMI weights gets distributed in a numerically unstable way and it crashes.
zordiack is offline   Reply With Quote

Old   December 10, 2018, 06:49
Default
  #3
New Member
 
behzad
Join Date: Dec 2010
Posts: 15
Rep Power: 12
arionfard is on a distinguished road
how can i improve the interface matching?!
arionfard is offline   Reply With Quote

Old   December 10, 2018, 09:36
Default
  #4
Member
 
Pekka Pasanen
Join Date: Feb 2012
Location: Finland
Posts: 87
Rep Power: 11
zordiack is on a distinguished road
Quote:
Originally Posted by arionfard View Post
how can i improve the interface matching?!
Well that is hard to say when I don't know anything about your case or mesh..

In general, make sure that the rotational surfaces match as closely as possibly in geometry (use good quality .stl etc). Then try increasing mesh density on the interface, I also found out that using one boundary layer in snappyHexMesh (even with same cell size as inner mesh) makes the mesh more conforming because it extrudes from the rotational surfaces. Also make sure that edges are properly snapped.
zordiack is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Steady state solution as an initial condition for a transient problem adnanakhtar FLUENT 7 November 25, 2016 05:16
grid dependancy gueynard a. Main CFD Forum 19 June 27, 2014 21:22
boundary conditions dependent by the numerical solution c120613 CFX 3 September 22, 2011 15:31
reactingFoam solution is strongly dependent on time step size andras OpenFOAM Bugs 3 July 13, 2011 10:28
Wall functions Abhijit Tilak Main CFD Forum 6 February 5, 1999 01:16


All times are GMT -4. The time now is 13:43.