[sponiar]

Hello FOAMing user,
I am working on adjoint problem in internal and external aerodynamic. I take simpleFoam source code and append new one vector field 'V' and one scalar field 'q'.
There is also new boundary condition for vector 'V', which is implement for walls.
I have simple case with external aerodynamic:
- with cylinder like bluff body,
- Re ~ 2.5 mil.,
- Spalart Allmaras,
- total sum of finite volumes is 300e+03.

Simulation is succesfully runnig in single.
When I decompose domain with METIS into the:
- 2 subdomain, OK
- 3 subdomains, OK
- 4 subdomains, BAD
- more then 4, BAD

For the single process, or parallel run upto the 3 CPUs is sumalation OK. But when is domain decompose into the 4 and more CPUs, some errors occur:

------------------------------------------------

[cld:01519] *** Process received signal ***
[cld:01519] Signal: Floating point exception (8)
[cld:01519] Signal code: (-6)
[cld:01519] Failing at address: 0xbc1000005ef
[cld:01519] [ 0] /lib64/libc.so.6 [0x2ac5a1fcac10]
[cld:01519] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x2ac5a1fcab95]
[cld:01519] [ 2] /lib64/libc.so.6 [0x2ac5a1fcac10]
[cld:01519] [ 3] /home/sponiar/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN K4Foam21processorLduInterface14compressedSendINS_6 VectorIdEEEEvRKNS_5UListIT_EEb +0x92) [0x2ac5a058e072]
[cld:01519] [ 4] /home/sponiar/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN 4Foam21processorFvPatchFieldINS_6VectorIdEEE12init EvaluateEb+0x65) [0x2ac5a0591055]
[cld:01519] [ 5] simpleFoam_Ev04_2008-06-07(_ZN4Foam14GeometricFieldINS_6VectorIdEENS_12fvP atchFi eldENS_7volMeshEE22GeometricBoundaryField8evaluate Ev+0x87) [0x425f57]
[cld:01519] [ 6] /home/sponiar/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN 4Foam3fvc16surfaceIntegrateINS_6VectorIdEEEENS_3tm pINS_14GeometricFieldIT_NS_12f vPatchFieldENS_7volMeshEEEEERKNS5_IS6_NS_13fvsPatc hFieldENS_11surfaceMeshEEE+0x3 b0) [0x2ac5a0a56610]
[cld:01519] [ 7] /home/sponiar/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN 4Foam3fvc16surfaceIntegrateINS_6VectorIdEEEENS_3tm pINS_14GeometricFieldIT_NS_12f vPatchFieldENS_7volMeshEEEEERKNS4_INS5_IS6_NS_13fv sPatchFieldENS_11surfaceMeshEE EEE+0x2f) [0x2ac5a0a567bf]
[cld:01519] [ 8] /home/sponiar/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN 4Foam2fv14gaussDivSchemeINS_6TensorIdEEE6fvcDivERK NS_14GeometricFieldIS3_NS_12fv PatchFieldENS_7volMeshEEE+0x69) [0x2ac5a0a577c9]
[cld:01519] [ 9] /home/sponiar/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/libincompressibleTurbu lenceModels.so(_ZN4Foam3fvc3divINS_6TensorIdEEEENS _3tmpINS_14GeometricFieldINS_1 2innerProductINS_6VectorIdEET_E4typeENS_12fvPatchF ieldENS_7volMeshEEEEERKNS5_IS9 _SC_SD_EERKNS_4wordE+0x68) [0x2ac59fc35da8]
[cld:01519] [10] /home/sponiar/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/libincompressibleTurbu lenceModels.so(_ZN4Foam3fvc3divINS_6TensorIdEEEENS _3tmpINS_14GeometricFieldINS_1 2innerProductINS_6VectorIdEET_E4typeENS_12fvPatchF ieldENS_7volMeshEEEEERKNS5_IS9 _SC_SD_EE+0x1d2) [0x2ac59fc37e22]
[cld:01519] [11] /home/sponiar/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/libincompressibleTurbu lenceModels.so(_ZN4Foam3fvc3divINS_6TensorIdEEEENS _3tmpINS_14GeometricFieldINS_1 2innerProductINS_6VectorIdEET_E4typeENS_12fvPatchF ieldENS_7volMeshEEEEERKNS4_INS 5_IS9_SC_SD_EEEE+0x2f) [0x2ac59fc37eff]
[cld:01519] [12] /home/sponiar/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/libincompressibleTurbu lenceModels.so(_ZNK4Foam16turbulenceModels15Spalar tAllmaras4divRERNS_14Geometric FieldINS_6VectorIdEENS_12fvPatchFieldENS_7volMeshE EE+0xe8) [0x2ac59fd34f08]
[cld:01519] [13] simpleFoam_Ev04_2008-06-07 [0x4161e9]
[cld:01519] [14] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2ac5a1fb8154]
[cld:01519] [15] simpleFoam_Ev04_2008-06-07(__gxx_personality_v0+0xd9) [0x412f69]
[cld:01519] *** End of error message ***


------------------------------------------------

Maybe, it seems like:
when interface of subdomains is lies on patch, where is implement new boundary condition, something is wrong.

I don't know where could be a mistake.
Could you give some hint?
David
_____

[sponiar]

Hi FOAMing users,
with new OpenFOAM 1.5 and new option in decomposition domain: 'preservePatches', which take a patch and give it with sourrounding cells to one proccessor.

I rebuild the solver and BC for OopenFOAM 1.5 and use this option during decomposition.


It doesn't work, when the case is running on the more then 4 CPUs. So I play with set-up fvSolutinon and changed GAMG solver to PCG with GAMG preconditier.
It looks more stable when I use for pressure and adjoint pressure PCGGAMG, when simulation is runnig on 2 or 4 CPUs. When I use more then 4CPUS, GAMG is more stable.



Strange things happend: when I stop simulation (CTRL+C) and again strart the same simulation, same error occurs. So I wait for example 1 minute and re-run same simulation with same set-up and it goes correct.
Same things occurs in OF 1.4.1 and OF 1.5.

I try the aplication 'statifier', which build solver static. I though, when solver is static, it could be more stable. But same error occurs.

Should I rewrite boundary condition for parallel runnig? Solver is stable in single proccess in every case, which I tested.
May help rebuild OpenFOAM library to static form for stable runnig?

Please, give me some hint.
David