[hazem1]

Hello everyone,

I have a problem in running a parallel simulation with OpenFOAM on HPC.

I have a case that I used to run it smoothly on 40 processors. The same case now keeps failing without changing any of the input files.

I doubted that the problem is the version of openMPI, so I re-compliled OpenFOAM with a newer version of OpenMPI, but this didn't solve the problem.

Besides, I tried to run the case on a fewer number of processors. It ran without any problems on 4, 8 and 12 processors, and it failed on 16 and higher. The node of our HPC has 20 processors, so it failed even if it should be on the same node.

I tried also to remove the current version of OpenFOAM, and I installed the latest version, but this also doesn't help.

Has anyone encountered a similar problem? And how should I fix this?

Attached, screenshots of the log files at the line at which the simulation continues or fails. (after "Create time")

Screenshot 2024-03-03 172335.png

Screenshot 2024-03-03 173215.png

[AdrianaPinto2]

Originally Posted by hazem1

Hello everyone,

I have a problem in running a parallel simulation with OpenFOAM on HPC.

I have a case that I used to run it smoothly on 40 processors. The same case now keeps failing without changing any of the input files.

I doubted that the problem is the version of openMPI, so I re-compliled OpenFOAM with a newer version of OpenMPI, but this didn't solve the problem.

Besides, I tried to run the case on a fewer number of processors. It ran without any problems on 4, 8 and 12 processors, and it failed on 16 and higher. The node of our HPC has 20 processors, so it failed even if it should be on the same node.

I tried also to remove the current version of OpenFOAM, and I installed the latest version, but this also doesn't help.

Has anyone encountered a similar problem? And how should I fix this?

Attached, screenshots of the log files at the line at which the simulation continues or fails. (after "Create time")

Attachment 98749

Attachment 98750

Hello Hazem. Did you find a solution to this problem? I think I am having a similar problem.

[kol1978]

Originally Posted by AdrianaPinto2

Hello Hazem. Did you find a solution to this problem? I think I am having a similar problem.

I'm also interested in this topic...
What commands do you use to run multi-threading and multi-core?

[hazem1]

Yes, I have found a temporary solution to this problem by commenting these lines in etc/bashrc file of openfoam

#- Floating-point signal handling:
# set or unset
#export FOAM_SIGFPE=

#- memory initialisation:
# set or unset
#export FOAM_SETNAN=

[kol1978]

Originally Posted by hazem1

Yes, I have found a temporary solution to this problem by commenting these lines in etc/bashrc file of openfoam

#- Floating-point signal handling:
# set or unset
#export FOAM_SIGFPE=

#- memory initialisation:
# set or unset
#export FOAM_SETNAN=

How can you explain the “way” to achieve the result? does this mean that messages are just ignored...or is it something else? PS sorry for the curiosity.

[kol1978]

Originally Posted by hazem1

Yes, I have found a temporary solution to this problem by commenting these lines in etc/bashrc file of openfoam

#- Floating-point signal handling:
# set or unset
#export FOAM_SIGFPE=

#- memory initialisation:
# set or unset
#export FOAM_SETNAN=

I have version 12... and in it export FOAM_SETNAN= is commented out by default... - FYI

Time = 499s

DILUPBiCGStab: Solving for Ux, Initial residual = 0.99850453006779127385, Final residual = 0.00010498198180043912865, No Iterations 1
DILUPBiCGStab: Solving for Uy, Initial residual = 0.99998959877070658919, Final residual = 0.031512857985444418796, No Iterations 2
DILUPBiCGStab: Solving for h, Initial residual = 1, Final residual = 0.019798126998869420429, No Iterations 2
GAMG: Solving for p, Initial residual = 1, Final residual = 2.0185126205988228598, No Iterations 1000
time step continuity errors : sum local = 5.146150313434651517e+33, global = -9.0143693102528848905e+31, cumulative = -9.0143693044835395807e+31
limitPressure: min -2.9014427041733308485e+67
limitPressure: max 1.171646142087871299e+67
DILUPBiCGStab: Solving for omega, Initial residual = 0.16904390887232240148, Final residual = 0.0033324748957708474548, No Iterations 1
DILUPBiCGStab: Solving for k, Initial residual = 0.16350213330906379627, Final residual = 0.010349788792748771532, No Iterations 1
ExecutionTime = 20.375533999999998258 s ClockTime = 23 s

Time = 500s

DILUPBiCGStab: Solving for Ux, Initial residual = 0.82662523492615680443, Final residual = 0.026385557104654375898, No Iterations 208
DILUPBiCGStab: Solving for Uy, Initial residual = 0.72158252182498183736, Final residual = 0.010988437993210267707, No Iterations 203
[3] #0 [1] [2] #0 Foam::error:rintStack(Foam::Ostream&)[0] #0 Foam::error:rintStack(Foam::Ostream&)Foam::error :rintStack(Foam::Ostream&)#0 Foam::error:rintStack(Foam::Ostream&) at ??:?
[0] #1 Foam::sigFpe::sigHandler(int) at ??:?
[1] #1 Foam::sigFpe::sigHandler(int) at ??:?
at ??:?
[3] #1 Foam::sigFpe::sigHandler(int)[2] #1 Foam::sigFpe::sigHandler(int) at ??:?
[2] #2 ? at ??:?
[1] #2 ? at ??:?
[0] #2 ? at ??:?
[3] #2 ? in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[2] #3 double Foam::sumProd<double>(Foam::UList<double> const&, Foam::UList<double> const&) in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[1] #3 double Foam::sumProd<double>(Foam::UList<double> const&, Foam::UList<double> const&) in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[0] #3 double Foam::sumProd<double>(Foam::UList<double> const&, Foam::UList<double> const&) in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[3] #3 double Foam::sumProd<double>(Foam::UList<double> const&, Foam::UList<double> const&) at ??:?
[2] #4 Foam::PBiCGStab::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[1] #4 Foam::PBiCGStab::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[0] #4 Foam::PBiCGStab::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[1] #5 Foam::fvMatrix<double>::solveSegregated(Foam::dict ionary const&) at ??:?
[2] #5 Foam::fvMatrix<double>::solveSegregated(Foam::dict ionary const&) at ??:?
[3] #4 Foam::PBiCGStab::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[0] #5 Foam::fvMatrix<double>::solveSegregated(Foam::dict ionary const&) at ??:?
[3] #5 Foam::fvMatrix<double>::solveSegregated(Foam::dict ionary const&) at ??:?
[2] #6 Foam::fvMatrix<double>::solve(Foam::dictionary const&) at ??:?
[1] #6 Foam::fvMatrix<double>::solve(Foam::dictionary const&) at ??:?
[0] #6 Foam::fvMatrix<double>::solve(Foam::dictionary const&) at ??:?
[3] #6 Foam::fvMatrix<double>::solve(Foam::dictionary const&) at ??:?
[2] #7 Foam::fvMatrix<double>::solve() at ??:?
[1] #7 Foam::fvMatrix<double>::solve() at ??:?
[0] #7 Foam::fvMatrix<double>::solve() at ??:?
[3] #7 Foam::fvMatrix<double>::solve() at ??:?
[2] #8 Foam::solvers::fluid::thermophysicalPredictor() at ??:?
[2] #9 at ??:?
[1] #8 Foam::solvers::fluid::thermophysicalPredictor() at ??:?
[1] #9 ?? in "/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/bin/foamRun"
[1] #10 ? in "/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/bin/foamRun"
[2] #10 ? at ??:?
[0] #8 Foam::solvers::fluid::thermophysicalPredictor() in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[1] #11 __libc_start_main in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[2] #11 __libc_start_main in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[1] #12 at ??:?
[3] #8 Foam::solvers::fluid::thermophysicalPredictor() at ??:?
[0] #9 in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[2] #12 at ??:?
[3] #9 ???? in "/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/bin/foamRun"
[kol-VirtualBox:165236] *** Process received signal ***
[kol-VirtualBox:165236] Signal: Floating point exception (8)
[kol-VirtualBox:165236] Signal code: (-6)
[kol-VirtualBox:165236] Failing at address: 0x3e800028574
[kol-VirtualBox:165236] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x706210445320]
[kol-VirtualBox:165236] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x70621049eb1c]
[kol-VirtualBox:165236] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x70621044526e]
[kol-VirtualBox:165236] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x706210445320]
[kol-VirtualBox:165236] [ 4] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam7sumProdIdEEdRKNS_5UListIT_ EES5_+0x3e)[0x706211406ede]
[kol-VirtualBox:165236] [ 5] in "/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/bin/foamRun"
[kol-VirtualBox:165235] *** Process received signal ***
[kol-VirtualBox:165235] Signal: Floating point exception (8)
[kol-VirtualBox:165235] Signal code: (-6)
[kol-VirtualBox:165235] Failing at address: 0x3e800028573
[kol-VirtualBox:165235] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x7b1f40a45320]
[kol-VirtualBox:165235] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7b1f40a9eb1c]
[kol-VirtualBox:165235] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7b1f40a4526e]
[kol-VirtualBox:165235] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x7b1f40a45320]
[kol-VirtualBox:165235] [ 4] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam7sumProdIdEEdRKNS_5UListIT_ EES5_+0x3e)[0x7b1f41a06ede]
[kol-VirtualBox:165235] [ 5] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam9PBiCGStab5solveERNS_5Fiel dIdEERKS2_h+0x7a1)[0x7062111cc561]
[kol-VirtualBox:165236] [ 6] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE15solveSegr egatedERKNS_10dictionaryE+0x24f)[0x7062126584ff]
[kol-VirtualBox:165236] [ 7] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam9PBiCGStab5solveERNS_5Fiel dIdEERKS2_h+0x7a1)[0x7b1f417cc561]
[kol-VirtualBox:165235] [ 6] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS _10dictionaryE+0x302)[0x706212134852]
[kol-VirtualBox:165236] [ 8] in "/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/bin/foamRun"
[0] #10 ?/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveEv+0x 8d)[0x706212147f3d]
[kol-VirtualBox:165236] [ 9] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfluid.so(_ZN4Foam7solvers5fluid23thermophysical PredictorEv+0x53a)[0x70620c183f1a]
/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE15solveSegr egatedERKNS_10dictionaryE+0x24f)[0x7b1f42c584ff]
[kol-VirtualBox:165235] [ 7] [kol-VirtualBox:165236] [10] foamRun(+0x3df2)[0x5deb10210df2]
[kol-VirtualBox:165236] [11] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x70621042a1ca]
[kol-VirtualBox:165236] [12] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x70621042a28b]
[kol-VirtualBox:165236] [13] foamRun(+0x4565)[0x5deb10211565]
[kol-VirtualBox:165236] *** End of error message ***
/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS _10dictionaryE+0x302)[0x7b1f42734852]
[kol-VirtualBox:165235] [ 8] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveEv+0x 8d)[0x7b1f42747f3d]
[kol-VirtualBox:165235] [ 9] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfluid.so(_ZN4Foam7solvers5fluid23thermophysical PredictorEv+0x53a)[0x7b1f3c363f1a]
[kol-VirtualBox:165235] [10] foamRun(+0x3df2)[0x5bb987be5df2]
[kol-VirtualBox:165235] [11] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7b1f40a2a1ca]
[kol-VirtualBox:165235] [12] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7b1f40a2a28b]
[kol-VirtualBox:165235] [13] foamRun(+0x4565)[0x5bb987be6565]
[kol-VirtualBox:165235] *** End of error message ***
in "/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/bin/foamRun"
[3] #10 ? in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[0] #11 __libc_start_main in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[3] #11 __libc_start_main in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[0] #12 in "/usr/lib/x86_64-linux-gnu/libc.so.6"
[3] #12 ? in ?"/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/bin/foamRun"
[kol-VirtualBox:165237] *** Process received signal ***
[kol-VirtualBox:165237] Signal: Floating point exception (8)
[kol-VirtualBox:165237] Signal code: (-6)
[kol-VirtualBox:165237] Failing at address: 0x3e800028575
[kol-VirtualBox:165237] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x765179c45320]
[kol-VirtualBox:165237] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x765179c9eb1c]
[kol-VirtualBox:165237] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x765179c4526e]
[kol-VirtualBox:165237] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x765179c45320]
[kol-VirtualBox:165237] [ 4] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam7sumProdIdEEdRKNS_5UListIT_ EES5_+0x3e)[0x76517ac06ede]
[kol-VirtualBox:165237] [ 5] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam9PBiCGStab5solveERNS_5Fiel dIdEERKS2_h+0x7a1)[0x76517a9cc561]
[kol-VirtualBox:165237] [ 6] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE15solveSegr egatedERKNS_10dictionaryE+0x24f)[0x76517be584ff]
[kol-VirtualBox:165237] [ 7] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS _10dictionaryE+0x302)[0x76517b934852]
[kol-VirtualBox:165237] [ 8] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveEv+0x 8d)[0x76517b947f3d]
[kol-VirtualBox:165237] [ 9] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfluid.so(_ZN4Foam7solvers5fluid23thermophysical PredictorEv+0x53a)[0x76517806bf1a]
[kol-VirtualBox:165237] [10] foamRun(+0x3df2)[0x5c013c4e7df2]
[kol-VirtualBox:165237] [11] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x765179c2a1ca]
[kol-VirtualBox:165237] [12] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x765179c2a28b]
[kol-VirtualBox:165237] [13] foamRun(+0x4565)[0x5c013c4e8565]
[kol-VirtualBox:165237] *** End of error message ***
in "/home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/bin/foamRun"
[kol-VirtualBox:165234] *** Process received signal ***
[kol-VirtualBox:165234] Signal: Floating point exception (8)
[kol-VirtualBox:165234] Signal code: (-6)
[kol-VirtualBox:165234] Failing at address: 0x3e800028572
[kol-VirtualBox:165234] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x7551db245320]
[kol-VirtualBox:165234] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7551db29eb1c]
[kol-VirtualBox:165234] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7551db24526e]
[kol-VirtualBox:165234] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x7551db245320]
[kol-VirtualBox:165234] [ 4] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam7sumProdIdEEdRKNS_5UListIT_ EES5_+0x3e)[0x7551dc206ede]
[kol-VirtualBox:165234] [ 5] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam9PBiCGStab5solveERNS_5Fiel dIdEERKS2_h+0x7a1)[0x7551dbfcc561]
[kol-VirtualBox:165234] [ 6] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE15solveSegr egatedERKNS_10dictionaryE+0x24f)[0x7551dd4584ff]
[kol-VirtualBox:165234] [ 7] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS _10dictionaryE+0x302)[0x7551dcf34852]
[kol-VirtualBox:165234] [ 8] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveEv+0x 8d)[0x7551dcf47f3d]
[kol-VirtualBox:165234] [ 9] /home/kol/foam/OpenFOAM-12/platforms/linux64GccDPInt32Opt/lib/libfluid.so(_ZN4Foam7solvers5fluid23thermophysical PredictorEv+0x53a)[0x7551d8253f1a]
[kol-VirtualBox:165234] [10] foamRun(+0x3df2)[0x61fae9ecfdf2]
[kol-VirtualBox:165234] [11] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7551db22a1ca]
[kol-VirtualBox:165234] [12] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7551db22a28b]
[kol-VirtualBox:165234] [13] foamRun(+0x4565)[0x61fae9ed0565]
[kol-VirtualBox:165234] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 0 on node kol-VirtualBox exited on signal 8 (Floating point exception).

didn't help me...