[sidsahu]

Hello all,


I've written a custom solver on openfoam that calculates the Lyapunov exponents of a given flow and prints them into a text file. I wish to parallelise my run and I'm using the domain decomposition method is inbuilt into OpenFOAM.


I call my solver in the run directory using the following commands;


decomposePar


mpirun -np 4 SC_LE_Analysis -parallel


This command starts the calculation. But after completing a few time steps, I get the following error:


Checkpoint 4 (Finished computing all 10 perturbations' growth)
Starting Reorthonormalisation
[0] #0 Foam::error:rintStack(Foam::Ostream&)[2] #0 Foam::error:rintStack(Foam::Ostream&) at ??:?
[0] #1 Foam::sigFpe::sigHandler(int) at ??:?
[2] #1 Foam::sigFpe::sigHandler(int) at ??:?
at ??:?
[2] #2 ?[0] #2 ? in "/lib/x86_64-linux-gnu/libc.so.6"
[2] #3 in "/lib/x86_64-linux-gnu/libc.so.6"
[0] #3 ?? in "/home/sid in "/home/sidsahu/sahu/OpeOpenFOAM/sidnFOAM/sidsahu-sahu-10/p10/platformlatforms/lis/linux64Gccnux64GccDPDPInt32Opt/bInt32Opt/binin/SC_LE_Analy/SC_LE_Analsis"
ysis"
[0] #4 [2] #4 ?? in "/home/sidsahu/OpenFOAM/sidsahu-10/platforms/linux64GccDPInt32Opt/bin/SC_LE_Analysis"
[0] #5 in "/home/sidsahu/OpenFOAM/sidsahu-10/platforms/linux64GccDPInt32Opt/bin/SC_LE_Analysis"
[2] #5 ?? in "/home/sidsahu/OpenFOAM/sidsahu-10/platforms/l in "inux64GccDPIn/home/sidsahut32Opt/bin/OpenFOAM//SC_LE_Anasidsahu-10/lysis"
platforms/linux64Gc[0] #cDPInt32Op6 ?t/bin/SC_LE_Analysis"
[2] #6 ? in "/lib/x86_64-linux-gnu/libc.so.6"
[0] #7 __libc_start_main in "/lib/x86_64-linux-gnu/libc.so.6"
[2] #7 __libc_start_main in "/lib/x86_64-linux-gnu/libc.so.6"
[0] #8 in "/lib/x86_64-linux-gnu/libc.so.6"
[2] #8 ?? in "/home/sidsahu/OpenFOAM/sidsahu-10/platforms/linux64GccDPInt32Opt/bin/SC_LE_Analysis"
[sidsahu-HP-ZBook-15-G4:16740] *** Process received signal ***
[sidsahu-HP-ZBook-15-G4:16740] Signal: Floating point exception (8)
[sidsahu-HP-ZBook-15-G4:16740] Signal code: (-6)
[sidsahu-HP-ZBook-15-G4:16740] Failing at address: 0x3e800004164
[sidsahu-HP-ZBook-15-G4:16740] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f3768442520]
[sidsahu-HP-ZBook-15-G4:16740] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f37684969fc]
[sidsahu-HP-ZBook-15-G4:16740] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f3768442476]
[sidsahu-HP-ZBook-15-G4:16740] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f3768442520]
[sidsahu-HP-ZBook-15-G4:16740] [ 4] SC_LE_Analysis(+0x448d0)[0x55b8e26b98d0]
[sidsahu-HP-ZBook-15-G4:16740] [ 5] SC_LE_Analysis(+0x45db2)[0x55b8e26badb2]
[sidsahu-HP-ZBook-15-G4:16740] [ 6] SC_LE_Analysis(+0x3d75d)[0x55b8e26b275d]
[sidsahu-HP-ZBook-15-G4:16740] [ 7] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f3768429d90]
[sidsahu-HP-ZBook-15-G4:16740] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f3768429e40]
[sidsahu-HP-ZBook-15-G4:16740] [ 9] SC_LE_Analysis(+0x40915)[0x55b8e26b5915]
[sidsahu-HP-ZBook-15-G4:16740] *** End of error message ***
in "/home/sidsahu/OpenFOAM/sidsahu-10/platforms/linux64GccDPInt32Opt/bin/SC_LE_Analysis"
[sidsahu-HP-ZBook-15-G4:16742] *** Process received signal ***
[sidsahu-HP-ZBook-15-G4:16742] Signal: Floating point exception (8)
[sidsahu-HP-ZBook-15-G4:16742] Signal code: (-6)
[sidsahu-HP-ZBook-15-G4:16742] Failing at address: 0x3e800004166
[sidsahu-HP-ZBook-15-G4:16742] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f9ee7642520]
[sidsahu-HP-ZBook-15-G4:16742] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f9ee76969fc]
[sidsahu-HP-ZBook-15-G4:16742] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f9ee7642476]
[sidsahu-HP-ZBook-15-G4:16742] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f9ee7642520]
[sidsahu-HP-ZBook-15-G4:16742] [ 4] SC_LE_Analysis(+0x4493a)[0x563e7e0b093a]
[sidsahu-HP-ZBook-15-G4:16742] [ 5] SC_LE_Analysis(+0x45db2)[0x563e7e0b1db2]
[sidsahu-HP-ZBook-15-G4:16742] [ 6] SC_LE_Analysis(+0x3d75d)[0x563e7e0a975d]
[sidsahu-HP-ZBook-15-G4:16742] [ 7] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f9ee7629d90]
[sidsahu-HP-ZBook-15-G4:16742] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f9ee7629e40]
[sidsahu-HP-ZBook-15-G4:16742] [ 9] SC_LE_Analysis(+0x40915)[0x563e7e0ac915]
[sidsahu-HP-ZBook-15-G4:16742] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node sidsahu-HP-ZBook-15-G4 exited on signal 8 (Floating point exception).




%%%%%%%%%%%%


The section of SC_LE_Analsysis.C that prints the LEs into a text file is shown in SC_LE_AnalysisC.png



I believe I'm doing the decomposition properly, I've also attached the decomposeParDict file in the attachements. I'm running this code on a machine with 4 cores.


Sometimes I've noticed, the values printed into the text file on a parallel run are very different from that of a serial run. I'm not very well-versed with Parallelisation of code and would be grateful for your help

[sidsahu]

Dear members,
I'd be grateful for your insights

[geth03]

Division by zero? Check your code if it might be pissible to divide by zero. If yes, then you should write something like 1/(max(yourFormula, 1E-19)

[sidsahu]

Hi Geth03,


Thanks for your reply.


I figured the main source of error is the fact that I don't gather the details of my velocity field for ex. "U_d" from all the processors, before I perform a norm() operation or QR decomposition on U_d.



Do you know how I can use Pstream::gather() or Pstream::gatherList() to gather U_d within my main code.



P.S. This is almost like performing reconstructPar, but I need to do it within my main code.


I'd really appreciate any help.


Thank you.

[sidsahu]

I'd be grateful for your inputs