[syavash]

Dear mates,

I have tried to install both OF1812 and the recently released OF1906 on a cluster. However, I cannot manage to use GAMG solver in either versions, not even in the tutorial cases. It gives me error that I cannot understand.

Now, my question here, is there a connection between CGAL/boost installation and proper compilation of GAMG solver? I ask because I could not manage to install CGAL/boost. However, OpenFOAM still has remained functional and I can use PCG algorithm and other stuff.

Kind Regards,
Syavash

[wyldckat]

Quick answer: GAMG should not depend on CGAL nor Boost. And if it did, you would likely never been able to run the solver, since it would not have compiled either.

Without being able to see the error message you are seeing, there isn't much we can suggest. My best guess is that perhaps it fails if you try to decompose a 100000 cell case with 200 cores, which results in 500 cells per subdomain, which can easily result in the default GAMG settings for minimum cells in the sub-matrix being larger then the number of faces on a patch.

[syavash]

Originally Posted by wyldckat

Quick answer: GAMG should not depend on CGAL nor Boost. And if it did, you would likely never been able to run the solver, since it would not have compiled either.

Without being able to see the error message you are seeing, there isn't much we can suggest. My best guess is that perhaps it fails if you try to decompose a 100000 cell case with 200 cores, which results in 500 cells per subdomain, which can easily result in the default GAMG settings for minimum cells in the sub-matrix being larger then the number of faces on a patch.

Thanks for your reply Bruno,

I agree that I did not provided enough info. Here is the error message I get from executing ./Allrun in tutorial "airfoil2D" from simpleFoam:

Reading field p

Reading field U

Reading/calculating face flux field phi

Selecting incompressible transport model Newtonian
Selecting turbulence model type RAS
Selecting RAS turbulence model SpalartAllmaras
Selecting patchDistMethod meshWave
RAS
{
    RASModel        SpalartAllmaras;
    turbulence      on;
    printCoeffs     on;
    sigmaNut        0.66666;
    kappa           0.41;
    Cb1             0.1355;
    Cb2             0.622;
    Cw2             0.3;
    Cw3             2;
    Cv1             7.1;
    Cs              0.3;
}

No MRF models present

No finite volume options present

Starting time loop

Time = 1

smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 0.0611362, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 1, Final residual = 0.0585177, No Iterations 2
#0  Foam::error::printStack(Foam::Ostream&) at ??:?
#1  Foam::sigFpe::sigHandler(int) at ??:?
#2  ? in /lib64/libpthread.so.0
#3  Foam::GAMGSolver::scale(Foam::Field<double>&, Foam::Field<double>&, Foam::lduMatrix const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, Foam::Field<double> const&, unsigned char) const at ??:?
#4  Foam::GAMGSolver::Vcycle(Foam::PtrList<Foam::lduMatrix::smoother> const&, Foam::Field<double>&, Foam::Field<double> const&, Foam::Field<double>&, Foam::Field<double>&, Foam::Field<double>&, Foam::Field<double>&, Foam::Field<double>&, Foam::PtrList<Foam::Field<double> >&, Foam::PtrList<Foam::Field<double> >&, unsigned char) const at ??:?
#5  Foam::GAMGSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
#6  Foam::fvMatrix<double>::solveSegregated(Foam::dictionary const&) at ??:?
#7  Foam::fvMatrix<double>::solveSegregatedOrCoupled(Foam::dictionary const&) at ??:?
#8  ? at ??:?
#9  __libc_start_main in /lib64/libc.so.6
#10  ? at ??:?

Please note that the execution was in serial and I did not change anything.

Regards,
Syavash

[wyldckat]

Quick answer: I remember seeing this problem in the past and I have a very vague memory that the issue was due to a compiler version problem...

Questions:

1. Which Linux Distribution are you using?
2. Which compiler and compiler version are you using?
3. Did you do any changes in the 'wmake/rules' folder?
4. Do you have any special shell environment variables that have compiler options loaded into them?

The last two questions is because I vaguely remember that the issue could have been due to an optimization flag that didn't work properly with OpenFOAM.

[syavash]

Originally Posted by wyldckat

Quick answer: I remember seeing this problem in the past and I have a very vague memory that the issue was due to a compiler version problem...

Questions:

1. Which Linux Distribution are you using?
2. Which compiler and compiler version are you using?
3. Did you do any changes in the 'wmake/rules' folder?
4. Do you have any special shell environment variables that have compiler options loaded into them?

The last two questions is because I vaguely remember that the issue could have been due to an optimization flag that didn't work properly with OpenFOAM.

Thanks Bruno, I guess we are in the right track.

In response to your questions:

1-The distribution is CentOS 7.

2-I have used Intel compiler/2018a-eb to compile the foam.

3-Yes! I have added the following flags to the "c++" file under 'wmake/rules/linux64Icc':

-Nmpi -Nmkl -xCORE-AVX512

The first flag we had discussed before, it was essential to run foam in parallel, see:

Non-root installation of OpenFOAM 2.4.x, parallel issue

The other flags I added were recommended by HPC webpage. I had included them also in foam-2.3.1 installation but it was fine there!

4-Well, I have provided the alias I use to call foam 1906 below. There are some other modules but I cannot say that they use compiler options (honestly not sure!).

alias OF19='export FOAM_INST_DIR=/.../.../.../.../Ehsan ; module load buildenv-intel/2018a-eb; module load ParaView/5.4.1-nsc1-gcc-2018a-eb; module load HDF5/1.8.19-nsc1-intel-2018a-eb; export MPI_ROOT=$I_MPI_ROOT; source /.../.../.../.../Ehsan/OpenFOAM-v1906/etc/bashrc WM_NCOMPPROCS=4 WM_COMPILER=Icc WM_MPLIB=INTELMPI'

I hope it can help to resolve my issue!

Regards,
Syavash

[wyldckat]

Quick answers:

1. I don't remember if "mkl" can introduce problems or not.
2. AVX is useless with OpenFOAM and could be causing problems.
3. If after removing those two options and rebuilding OpenFOAM, it still gives this problem, then open a report here: https://develop.openfoam.com/Develop...AM-plus/issues

[syavash]

Originally Posted by wyldckat

Quick answers:

1. I don't remember if "mkl" can introduce problems or not.
2. AVX is useless with OpenFOAM and could be causing problems.
3. If after removing those two options and rebuilding OpenFOAM, it still gives this problem, then open a report here: https://develop.openfoam.com/Develop...AM-plus/issues

Thanks Bruno,

I will recompile foam without those two flags and then report back!

Regards,
Syavash

[syavash]

Originally Posted by wyldckat

Quick answers:

1. I don't remember if "mkl" can introduce problems or not.
2. AVX is useless with OpenFOAM and could be causing problems.
3. If after removing those two options and rebuilding OpenFOAM, it still gives this problem, then open a report here: https://develop.openfoam.com/Develop...AM-plus/issues

Bruno,

Thanks to your help, the issue resolved! I removed those flags and compiled foam from scratch. Now, GAMG works.

Regards,
Syavash

[wyldckat]

🥳 I'm glad that solved out the issue! (Nice, emoji characters seem to work!)