For my approx 1.2 million cell
For my approx 1.2 million cells (singlephase) icoFoam vortex shedding case, here is the output after the first three time steps and the corresponding fvSolution pressure solver entries:
ICCG: (p ICCG 1e06 0;) /**\  =========    \ / F ield  OpenFOAM: The Open Source CFD Toolbox   \ / O peration  Version: 1.3   \ / A nd  Web: http://www.openfoam.org   \/ M anipulation   \**/ Exec : /home/madhavan/OpenFOAM/OpenFOAM1.3/applications/bin/linuxAMD64Gcc4DPOpt/icoFoa m /home/madhavan/OpenFOAM/madhavan1.3/run/tutorials/icoFoam/. one_sq_cyl_3d_unsteady Date : Nov 07 2006 Time : 12:27:17 Host : masternode PID : 21882 Root : /home/madhavan/OpenFOAM/madhavan1.3/run/tutorials/icoFoam/. Case : one_sq_cyl_3d_unsteady Nprocs : 1 Create time Create mesh for time = 0 Reading transportProperties Reading field p Reading field U Reading/calculating face flux field phi Reading probeLocations Constructing probes Starting time loop Time = 0.1 Mean and max Courant Numbers = 0 0.0741732 BICCG: Solving for Ux, Initial residual = 1, Final residual = 7.60531e08, No Iterations 3 BICCG: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0 BICCG: Solving for Uz, Initial residual = 0, Final residual = 0, No Iterations 0 ICCG: Solving for p, Initial residual = 1, Final residual = 9.73006e07, No Iterations 825 time step continuity errors : sum local = 4.55914e10, global = 5.75299e13, cumulative = 5.75299e13 ICCG: Solving for p, Initial residual = 0.00108292, Final residual = 9.32198e07, No Iterations 685 time step continuity errors : sum local = 3.75695e07, global = 7.83026e10, cumulative = 7.83601e10 Wall patch = 0 Wall patch name = ChannelWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 2.01178e05 pressureDragCoefficient = 0 viscDragCoefficient = 2.01178e05 LiftCoefficient = 1.41566e11 Wall patch = 1 Wall patch name = ObstacleWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 4.79068e06 pressureDragCoefficient = 4.62468e06 viscDragCoefficient = 1.66004e07 LiftCoefficient = 8.15603e14 ExecutionTime = 347.02 s ClockTime = 347 s Time = 0.2 Mean and max Courant Numbers = 0.108516 4.04533 BICCG: Solving for Ux, Initial residual = 0.823434, Final residual = 4.45773e06, No Iterations 3 BICCG: Solving for Uy, Initial residual = 0.329552, Final residual = 3.46137e06, No Iterations 4 BICCG: Solving for Uz, Initial residual = 0.362595, Final residual = 4.31759e06, No Iterations 3 ICCG: Solving for p, Initial residual = 0.00371997, Final residual = 9.84174e07, No Iterations 719 time step continuity errors : sum local = 1.53182e07, global = 3.88971e10, cumulative = 3.9463e10 ICCG: Solving for p, Initial residual = 0.00419475, Final residual = 9.92882e07, No Iterations 709 time step continuity errors : sum local = 7.36119e08, global = 2.94289e10, cumulative = 6.88919e10 Wall patch = 0 Wall patch name = ChannelWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 2.11014e05 pressureDragCoefficient = 0 viscDragCoefficient = 2.11014e05 LiftCoefficient = 2.18023e11 Wall patch = 1 Wall patch name = ObstacleWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 9.70849e07 pressureDragCoefficient = 1.11824e06 viscDragCoefficient = 1.47396e07 LiftCoefficient = 2.70848e11 ExecutionTime = 639.75 s ClockTime = 640 s   AMG (p AMG 1e06 0 1000;): /**\  =========    \ / F ield  OpenFOAM: The Open Source CFD Toolbox   \ / O peration  Version: 1.3   \ / A nd  Web: http://www.openfoam.org   \/ M anipulation   \**/ Exec : /home/madhavan/OpenFOAM/OpenFOAM1.3/applications/bin/linuxAMD64Gcc4DPOpt/icoFoa m /home/madhavan/OpenFOAM/madhavan1.3/run/tutorials/icoFoam/. one_sq_cyl_3d_unsteady Date : Nov 07 2006 Time : 11:29:35 Host : masternode PID : 21242 Root : /home/madhavan/OpenFOAM/madhavan1.3/run/tutorials/icoFoam/. Case : one_sq_cyl_3d_unsteady Nprocs : 1 Create time Create mesh for time = 0 Reading transportProperties Reading field p Reading field U Reading/calculating face flux field phi Reading probeLocations Constructing probes Starting time loop Time = 0.1 Mean and max Courant Numbers = 0 0.0741732 BICCG: Solving for Ux, Initial residual = 1, Final residual = 7.60531e08, No Iterations 3 BICCG: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0 BICCG: Solving for Uz, Initial residual = 0, Final residual = 0, No Iterations 0 AMG: Solving for p, Initial residual = 1, Final residual = 9.80164e07, No Iterations 105 time step continuity errors : sum local = 4.59248e10, global = 2.47297e14, cumulative = 2.47297e14 AMG: Solving for p, Initial residual = 0.00108292, Final residual = 9.69421e07, No Iterations 48 time step continuity errors : sum local = 3.90696e07, global = 1.15826e11, cumulative = 1.15579e11 Wall patch = 0 Wall patch name = ChannelWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 2.01179e05 pressureDragCoefficient = 0 viscDragCoefficient = 2.01179e05 LiftCoefficient = 5.64623e09 Wall patch = 1 Wall patch name = ObstacleWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 4.79158e06 pressureDragCoefficient = 4.62556e06 viscDragCoefficient = 1.6602e07 LiftCoefficient = 1.45852e10 ExecutionTime = 146.7 s ClockTime = 147 s Time = 0.2 Mean and max Courant Numbers = 0.108519 4.04592 BICCG: Solving for Ux, Initial residual = 0.823441, Final residual = 4.46106e06, No Iterations 3 BICCG: Solving for Uy, Initial residual = 0.329408, Final residual = 3.4621e06, No Iterations 4 BICCG: Solving for Uz, Initial residual = 0.362669, Final residual = 4.339e06, No Iterations 3 AMG: Solving for p, Initial residual = 0.0037213, Final residual = 9.14087e07, No Iterations 82 time step continuity errors : sum local = 1.42274e07, global = 4.78435e13, cumulative = 1.20363e11 AMG: Solving for p, Initial residual = 0.00419589, Final residual = 9.94429e07, No Iterations 75 time step continuity errors : sum local = 7.37282e08, global = 1.93804e13, cumulative = 1.18425e11 Wall patch = 0 Wall patch name = ChannelWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 2.11013e05 pressureDragCoefficient = 0 viscDragCoefficient = 2.11013e05 LiftCoefficient = 8.69834e09 Wall patch = 1 Wall patch name = ObstacleWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 9.72347e07 pressureDragCoefficient = 1.11973e06 viscDragCoefficient = 1.47384e07 LiftCoefficient = 2.83295e10 ExecutionTime = 294.14 s ClockTime = 295 s   AMG (p AMG 1e06 0 100;): /**\  =========    \ / F ield  OpenFOAM: The Open Source CFD Toolbox   \ / O peration  Version: 1.3   \ / A nd  Web: http://www.openfoam.org   \/ M anipulation   \**/ Exec : /home/madhavan/OpenFOAM/OpenFOAM1.3/applications/bin/linuxAMD64Gcc4DPOpt/icoFoa m /home/madhavan/OpenFOAM/madhavan1.3/run/tutorials/icoFoam/. one_sq_cyl_3d_unsteady Date : Nov 07 2006 Time : 12:20:05 Host : masternode PID : 21848 Root : /home/madhavan/OpenFOAM/madhavan1.3/run/tutorials/icoFoam/. Case : one_sq_cyl_3d_unsteady Nprocs : 1 Create time Create mesh for time = 0 Reading transportProperties Reading field p Reading field U Reading/calculating face flux field phi Reading probeLocations Constructing probes Starting time loop Time = 0.1 Mean and max Courant Numbers = 0 0.0741732 BICCG: Solving for Ux, Initial residual = 1, Final residual = 7.60531e08, No Iterations 3 BICCG: Solving for Uy, Initial residual = 0, Final residual = 0, No Iterations 0 BICCG: Solving for Uz, Initial residual = 0, Final residual = 0, No Iterations 0 AMG: Solving for p, Initial residual = 1, Final residual = 8.07098e07, No Iterations 102 time step continuity errors : sum local = 3.78169e10, global = 1.46199e14, cumulative = 1.46199e14 AMG: Solving for p, Initial residual = 0.00108292, Final residual = 8.70596e07, No Iterations 48 time step continuity errors : sum local = 3.50868e07, global = 3.39341e10, cumulative = 3.39356e10 Wall patch = 0 Wall patch name = ChannelWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 2.01179e05 pressureDragCoefficient = 0 viscDragCoefficient = 2.01179e05 LiftCoefficient = 4.35326e09 Wall patch = 1 Wall patch name = ObstacleWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 4.79156e06 pressureDragCoefficient = 4.62554e06 viscDragCoefficient = 1.6602e07 LiftCoefficient = 8.678e11 ExecutionTime = 108.75 s ClockTime = 109 s Time = 0.2 Mean and max Courant Numbers = 0.10852 4.04587 BICCG: Solving for Ux, Initial residual = 0.823445, Final residual = 4.46093e06, No Iterations 3 BICCG: Solving for Uy, Initial residual = 0.329439, Final residual = 3.4626e06, No Iterations 4 BICCG: Solving for Uz, Initial residual = 0.362502, Final residual = 4.33196e06, No Iterations 3 AMG: Solving for p, Initial residual = 0.00372114, Final residual = 7.40435e07, No Iterations 68 time step continuity errors : sum local = 1.15246e07, global = 2.12212e12, cumulative = 3.37234e10 AMG: Solving for p, Initial residual = 0.00419577, Final residual = 9.12739e07, No Iterations 60 time step continuity errors : sum local = 6.76724e08, global = 1.6342e11, cumulative = 3.53576e10 Wall patch = 0 Wall patch name = ChannelWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 2.11013e05 pressureDragCoefficient = 0 viscDragCoefficient = 2.11013e05 LiftCoefficient = 7.399e09 Wall patch = 1 Wall patch name = ObstacleWalls Uav = (1 0 0) Aref = 1 nu = nu [0 2 1 0 0 0 0] 1.004e06 DragCoefficient = 9.72242e07 pressureDragCoefficient = 1.11963e06 viscDragCoefficient = 1.47384e07 LiftCoefficient = 2.05608e10 ExecutionTime = 191.77 s ClockTime = 192 s Clearly, the AMG solver is better than ICCG for this case. Are there any guidelines on what number to use at the coarsest level for the AMG solver? 
On the top level of AMG you ha
On the top level of AMG you have a system of equations that needs to be solved to a tight tolerance and the question is how much work to leave for the top level. In serial runs, anything between a dozen and a couple of hundred cells will give you the same result (1000 is too much). In parallel, it is the communications that matter, which means you need to be more careful: make sure there's a sufficient number of cells for each CPU. You also need to worry about the case where you have multiple separate domains. In general, 2030 cells per CPU in parallel will do you fine (until I do some more tricks to speed this up). :)
Hrv 
Thanks for that insightful rep
Thanks for that insightful reply Hrv. Much appreciated.

All times are GMT 4. The time now is 20:46. 