[letzel]

Dear FOAMers,

my pisoFoam solver modified to produce postprocessing output for LES quality control causes intermittent floating point errors (FPE). Code and error message will be posted below.

It is possible to restart the crashed job from latestTime, which is up to several dozen time steps in the past, because writeInterval >> deltaT. To my surprise I noticed that this procedure has so far always been successful to overcome the time when the FPE initially occurred in the crashed job, but only to produce the same FPE at some time later in the course of the simulation. So far, the FPE has never occurred twice at the same time, although it recurs it is strictly speaking not reproducible. I see no indication to predict when the FPE will occur.

Code added to createFields.H:

    volVectorField Uav
    (
        IOobject
        (
            "Uav",
            runTime.timeName(),
            mesh,
            IOobject::NO_READ,
            IOobject::AUTO_WRITE
        ),
        mesh,
        dimensionedVector("Uav", dimensionSet(0,1,-1,0,0,0,0), Foam::vector(0,0,0))
    );

    volVectorField U2av
    (
        IOobject
        (
            "U2av",
            runTime.timeName(),
            mesh,
            IOobject::NO_READ,
            IOobject::AUTO_WRITE
        ),
        mesh,
        dimensionedVector("U2av", dimensionSet(0,2,-2,0,0,0,0), Foam::vector(0,0,0))
    );

Code added to pisoFoam.C, renamed to XC15Heise2.C:

        scalar dt = runTime.deltaT().value();

        if ( time > startAveraging + dt )
        {
          Uav  = ( Uav*(time-startAveraging) + U*dt ) / (time-startAveraging + dt);
          for ( int i = 1; i <= 3; ++i)
          {
            U2av.component(i) = ( U2av.component(i)*(time-startAveraging) + U.component(i)*U.component(i)*dt ) / (time-startAveraging + dt);
          }
        }

The FPE occurs when the term U.component(i)*U.component(i) is calculated. Replacing that term by sqr(U.component(i)causes almost the same error message. Here is an exemplary excerpt from the log:

Time = 2107.95

Courant Number mean: 0.0923695 max: 0.869684
Ubar_g = 1.7558
fMC = 1.00107
DILUPBiCG:  Solving for Ux, Initial residual = 0.00387829, Final residual = 2.36e-08, No Iterations 2
DILUPBiCG:  Solving for Uy, Initial residual = 0.0124318, Final residual = 2.63485e-07, No Iterations 2
DILUPBiCG:  Solving for Uz, Initial residual = 0.0130876, Final residual = 9.39095e-08, No Iterations 2
GAMG:  Solving for p, Initial residual = 0.118412, Final residual = 0.00851802, No Iterations 8
time step continuity errors : sum local = 9.43142e-07, global = -2.8715e-07, cumulative = -3.47101e-06
GAMG:  Solving for p, Initial residual = 0.0124153, Final residual = 0.00732398, No Iterations 1
time step continuity errors : sum local = 8.17403e-07, global = 2.50387e-07, cumulative = -3.22063e-06
DILUPBiCG:  Solving for s1, Initial residual = 0.000401176, Final residual = 8.4247e-08, No Iterations 2
DILUPBiCG:  Solving for s2, Initial residual = 0.000371733, Final residual = 9.41308e-07, No Iterations 1
ExecutionTime = 14039.3 s  ClockTime = 14080 s

Time = 2108

Courant Number mean: 0.0923709 max: 0.873343
Ubar_g = 1.7558
fMC = 0.999099
DILUPBiCG:  Solving for Ux, Initial residual = 0.00355117, Final residual = 2.59767e-08, No Iterations 2
DILUPBiCG:  Solving for Uy, Initial residual = 0.0125795, Final residual = 3.48008e-07, No Iterations 2
DILUPBiCG:  Solving for Uz, Initial residual = 0.0116423, Final residual = 1.20912e-07, No Iterations 2
GAMG:  Solving for p, Initial residual = 0.123565, Final residual = 0.00945173, No Iterations 10
time step continuity errors : sum local = 1.05124e-06, global = 2.93489e-07, cumulative = -2.92714e-06
GAMG:  Solving for p, Initial residual = 0.0139839, Final residual = 0.00829818, No Iterations 1
time step continuity errors : sum local = 9.2276e-07, global = -2.56682e-07, cumulative = -3.18382e-06
DILUPBiCG:  Solving for s1, Initial residual = 0.000401261, Final residual = 9.34449e-08, No Iterations 2
DILUPBiCG:  Solving for s2, Initial residual = 0.000371337, Final residual = 9.54196e-07, No Iterations 1
[0] #0  Foam::error::printStack(Foam::Ostream&) in "/opt/openfoam222/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #1  Foam::sigFpe::sigHandler(int) in "/opt/openfoam222/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #2   in "/lib/x86_64-linux-gnu/libc.so.6"
[0] #3  Foam::multiply(Foam::Field<double>&, Foam::UList<double> const&, Foam::UList<double> const&) in "/opt/openfoam222/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #4  void Foam::multiply<Foam::fvPatchField, Foam::volMesh>(Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh>&, Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> const&, Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> const&) in "/home/deineruser/OpenFOAM/deineruser-2.2.2/platforms/linux64GccDPOpt/bin/XC15Heise2"
[0] #5  Foam::tmp<Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> > Foam::operator*<Foam::fvPatchField, Foam::volMesh>(Foam::tmp<Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> > const&, Foam::tmp<Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> > const&) in "/home/deineruser/OpenFOAM/deineruser-2.2.2/platforms/linux64GccDPOpt/bin/XC15Heise2"
[0] #6  
[0]  in "/home/deineruser/OpenFOAM/deineruser-2.2.2/platforms/linux64GccDPOpt/bin/XC15Heise2"
[0] #7  __libc_start_main in "/lib/x86_64-linux-gnu/libc.so.6"
[0] #8  
[0]  in "/home/deineruser/OpenFOAM/deineruser-2.2.2/platforms/linux64GccDPOpt/bin/XC15Heise2"
[deiner:01410] *** Process received signal ***
[deiner:01410] Signal: Floating point exception (8)
[deiner:01410] Signal code:  (-6)
[deiner:01410] Failing at address: 0x3e800000582
[deiner:01410] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x2b2135d2c4a0]
[deiner:01410] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x2b2135d2c425]
[deiner:01410] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x2b2135d2c4a0]
[deiner:01410] [ 3] /opt/openfoam222/platforms/linux64GccDPOpt/lib/libOpenFOAM.so(_ZN4Foam8multiplyERNS_5FieldIdEERKNS_5UListIdEES6_+0xe6) [0x2b2134ef4746]
[deiner:01410] [ 4] XC15Heise2(_ZN4Foam8multiplyINS_12fvPatchFieldENS_7volMeshEEEvRNS_14GeometricFieldIdT_T0_EERKS6_S9_+0xb7) [0x43b047]
[deiner:01410] [ 5] XC15Heise2(_ZN4FoammlINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_SA_+0x24b) [0x45982b]
[deiner:01410] [ 6] XC15Heise2() [0x422f92]
[deiner:01410] [ 7] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x2b2135d1776d]
[deiner:01410] [ 8] XC15Heise2() [0x424e19]
[deiner:01410] *** End of error message ***
--------------------------------------------------------------------------
mpirun.openmpi noticed that process rank 0 with PID 1410 on node deiner exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------

My hypothesis is that the FPE is related to unnatural values of U.component(i) during the course of the simulation.
The fact that the FPE does not recur in restart runs at exactly the same time may simply be due to the chosen settings writeFormat ascii; writePrecision 6;, i.e. the restart is not an exact restart due to limited numerical accuracy of the restart data.

Two requests to the community:

• If anyone knows the source of the FPE and how to avoid it, please let me know.
• I am also looking for advice on how to further debug this, e.g. how to identidy the individual cell, how to obtain its location and values of other variables at that cell.

Best wishes,
Marcus