[itsAdoozy]

I'm trying to run some unsteady airfoil simulations (2D) using the overPimpleDyMFoam solver, with OpenFOAM v2012. I previously have had it running, but it was running slowly and I wanted to speed up my computation time. Specifically I was having issues where it was running faster on a single core than when I split it up between multiple cores.

I did some searching and read in this post that I should use the GAMG solver for pressure (was using PBiCGStab) since it's less resistant to parallelization. But, now that I've switched over to the GAMG solver I was running into this error on my first solution step:

PIMPLE: iteration 1
[1] 
[1] 
[1] --> FOAM FATAL ERROR: (openfoam-2012)
[1] field does not correspond to level 0 sizes: field = 100549 level = 101003
[1] 
[1]     From void Foam::GAMGAgglomeration::restrictFaceField(Foam::Field<Type>&, const Foam::Field<Type>&, Foam::label) const [with Type = double; Foam::label = int]
[1]     in file lnInclude/GAMGAgglomerationTemplates.C at line 151.
[1] 
FOAM parallel run aborting

After more searching the only relevant post I could find was from 2010 where they said to turn off my cacheAgglomeration, but I still got the same error.

fvSolution:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2012                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      fvSolution;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

solvers
{
    cellDisplacement
    {
        solver          PCG;
        preconditioner  DIC;
        tolerance       1e-06;
        relTol          0;
        maxIter         100;
    }

    p
    {
        solver          GAMG; //PBiCGStab
        // preconditioner  DILU;
        smoother GaussSeidel;
        cacheAgglomeration off;
        tolerance       1e-6;
        relTol          0;
    }

    pcorr
    {
        $p;
        // solver          PCG;
        // preconditioner  DIC;
    }

    pFinal
    {
        $p;
    }

    pcorrFinal
    {
        $pcorr;
        relTol          0;
    }

    "(U|k|omega|epsilon)"
    {
        solver          smoothSolver;
        smoother        symGaussSeidel;
        tolerance       1e-6;
        relTol          0;
    }

    "(U|k|omega|epsilon)Final"
    {
        $U;
        tolerance       1e-6;
        relTol          0;
    }
}

PIMPLE
{
    momentumPredictor           no;
    correctPhi                  no;
    nOuterCorrectors            1;
    nCorrectors                 2;
    nNonOrthogonalCorrectors    0;

    ddtCorr                     false;
}

relaxationFactors
{
    fields
    {
        p               0.3;
    }
    equations
    {
        U                           0.7;
        k                           0.7;
        omega                       0.7;
        epsilon                     0.7;
        "(U|k|omega|epsilon)Final"  1.0;
    }
}

// ************************************************************************* //

controlDict:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2012                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

libs            (overset fvMotionSolvers);

application     overPimpleDyMFoam;

startFrom       startTime;

startTime       0;

stopAt          endTime;

endTime         1;

deltaT          0.01;

writeControl    runTime;

writeInterval   0.1;

purgeWrite      0;

writeFormat     ascii;

writePrecision  10;

writeCompression off;

timeFormat      general;

timePrecision   6;

runTimeModifiable true;

adjustTimeStep  yes;

maxCo   1;

functions
{
    liftCoeff
    {
        type            forceCoeffs;
        libs            ("libforces.so");
        writeControl    timeStep;
        writeInterval   10;

        // Field names
        p               p;
        U               U;
        rho             rho;

        patches         (walls);
        rho             rhoInf;
        rhoInf          1.225;
        CofR            (0 0 0);
        liftDir         (0 1 0);
        dragDir         (1 0 0);
        pitchAxis       (0 0 1);
        magUInf         10;
        lRef            1;
        Aref            0.05;
    }

decomposeParDict:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2012                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 3;

method          scotch;

coeffs
{
    n           (4 2 1);
}

// ************************************************************************* //

Note: I'm using 3 processors to keep my cells/processor above 50,000.

I'm pretty stuck at this point, I'd like to get it working with the GAMG solver to see if that improves my parallel run time, but if you have other ideas on how to speed up the comps I'm happy to try them, thanks in advance!