CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

mpi problem with tutorial bed2

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   June 16, 2010, 08:59
Default mpi problem with tutorial bed2
  #1
New Member
 
Join Date: Jan 2010
Posts: 7
Rep Power: 7
piterw82 is on a distinguished road
Dear OpenFOAM users,

I am trying to run "bed2" tutorial (twoPhaseEulerFoam) on 2 CPU's using OpenFOAM-1.6. It works on single CPU ( i reach 2s of the flow) but fails on more than one CPU (it crashes at 0.114s of the flow). I run the case on local machine that has 4CPU's. Did anybody hve the same problems?
My operating system is OpenSuse 11.1 (x86_64)

The error message I get is:

[0] #0 Foam::error:rintStack(Foam::Ostream&) in "/home/OpenFOAM /OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #1 Foam::sigFpe::sigFpeHandler(int) in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #2 ?? in "/lib64/libc.so.6"
[0] #3 Foam::GAMGSolver::scalingFactor(Foam::Field<double >&, Foam::Field<double> const&, Foam::Field<double> const&, Foam::Field<double> const&) const in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #4 Foam::GAMGSolver::scalingFactor(Foam::Field<double >&, Foam::lduMatrix const&, Foam::Field<double>&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, Foam::Field<double> const&, unsigned char) const in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #5 Foam::GAMGSolver::Vcycle(Foam::PtrList<Foam::lduMa trix::smoother> const&, Foam::Field<double>&, Foam::Field<double> const&, Foam::Field<double>&, Foam::Field<double>&, Foam::Field<double>&, Foam::PtrList<Foam::Field<double> >&, Foam::PtrList<Foam::Field<double> >&, unsigned char) const in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #6 Foam::GAMGSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so"
[0] #7 Foam::fvMatrix<double>::solve(Foam::dictionary const&) in "/home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libfiniteVolume.so"
[0] #8 main in "/root/OpenFOAM/root-1.6/applications/bin/linux64GccDPOpt/twoPhaseEulerFoam"
[0] #9 __libc_start_main in "/lib64/libc.so.6"
[0] #10 _start at /usr/src/packages/BUILD/glibc-2.9/csu/../sysdeps/x86_64/elf/start.S:116
[peter:15350] *** Process received signal ***
[peter:15350] Signal: Floating point exception (8)
[peter:15350] Signal code: (-6)
[peter:15350] Failing at address: 0x3bf6
[peter:15350] [ 0] /lib64/libc.so.6 [0x7f388a8196e0]
[peter:15350] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7f388a819645]
[peter:15350] [ 2] /lib64/libc.so.6 [0x7f388a8196e0]
[peter:15350] [ 3] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam10GAMGSolver13scalingFacto rERNS_5FieldIdEERKS2_S5_S5_+0x6d) [0x7f388b69391d]
[peter:15350] [ 4] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam10GAMGSolver13scalingFacto rERNS_5FieldIdEERKNS_9lduMatrixES3_RKNS_10FieldFie ldIS1_dEERKNS_8UPtrListIKNS_17lduInterfaceFieldEEE RKS2_h+0xaa) [0x7f388b693d2a]
[peter:15350] [ 5] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam10GAMGSolver6VcycleERKNS_7 PtrListINS_9lduMatrix8smootherEEERNS_5FieldIdEERKS 8_S9_S9_S9_RNS1_IS8_EESD_h+0x1294) [0x7f388b696b34]
[peter:15350] [ 6] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam10GAMGSolver5solveERNS_5Fi eldIdEERKS2_h+0x46f) [0x7f388b697d1f]
[peter:15350] [ 7] /home/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN4Foam8fvMatrixIdE5solveERKNS _10dictionaryE+0x158) [0x7f388cc28498]
[peter:15350] [ 8] twoPhaseEulerFoam [0x432158]
[peter:15350] [ 9] /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f388a805586]
[peter:15350] [10] twoPhaseEulerFoam [0x42ae29]
[peter:15350] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 15350 on node peter exited on signal 8 (Floating point exception).
--------------------------------------------------------------------------

Regards,
Peter

Last edited by piterw82; June 16, 2010 at 09:54.
piterw82 is offline   Reply With Quote

Old   June 19, 2010, 12:23
Default
  #2
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,511
Blog Entries: 34
Rep Power: 86
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Peter,

I've just tested the same tutorial case you've reported about, and I ran with a recent (pulled today) OpenFOAM 1.6.x with 2 cores under KUbuntu 10.04 64bit using gcc 4.4.3 and it crashed later on during the run at "Time = 1.04"s, instead of 0.114s.
Also ran the same tutorial case using the pre-built OpenFOAM 1.6 in the same machine and it crashed the very same way, but at "Time = 1.0105"s.
edit: a minor note: with OpenFOAM 1.6, both mpi launched processes crashed with the same sigFpe error!

I've used the simple decomposition method with 2 subdomains and split it in 2 over the 1st coordinate.

I thought this could be a bug already fixed, but apparently it is yet to be fixed. Please report it on the bug report section of this forum, since it was you who found about it : OpenFOAM Bugs
I think you can post a short description there and put a link to this thread, in order to minimize entropy

Best regards,
Bruno

Last edited by wyldckat; June 19, 2010 at 12:26. Reason: minor note about the OpenFOAM 1.6...
wyldckat is offline   Reply With Quote

Old   June 21, 2010, 20:33
Default
  #3
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,894
Rep Power: 26
alberto will become famous soon enoughalberto will become famous soon enough
Simply use a conjugate gradient method instead than GAMG. In my experience it is more robust in multiphase simulations.

Best,
__________________
Alberto Passalacqua

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
OpenQBMM - An open-source implementation of quadrature-based moment methods
alberto is offline   Reply With Quote

Old   June 22, 2010, 05:07
Default
  #4
New Member
 
Join Date: Jan 2010
Posts: 7
Rep Power: 7
piterw82 is on a distinguished road
Hi Alberto,

I have switched GAMG to PCG solver for the pressure but the only result was to get further time step. I also set the tolerances for pressure and alpha to 1e-15 and it also did not help. Maybe you have some other sugestions?

Is this tutorial working for you in parallel?

Regards,
Peter
piterw82 is offline   Reply With Quote

Old   June 22, 2010, 10:27
Default
  #5
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,894
Rep Power: 26
alberto will become famous soon enoughalberto will become famous soon enough
Quote:
Originally Posted by piterw82 View Post
Hi Alberto,

I have switched GAMG to PCG solver for the pressure but the only result was to get further time step. I also set the tolerances for pressure and alpha to 1e-15 and it also did not help. Maybe you have some other sugestions?
Is the code crashing for alpha being too high?

Best,
__________________
Alberto Passalacqua

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
OpenQBMM - An open-source implementation of quadrature-based moment methods
alberto is offline   Reply With Quote

Old   June 22, 2010, 11:52
Default
  #6
New Member
 
Join Date: Jan 2010
Posts: 7
Rep Power: 7
piterw82 is on a distinguished road
Quote:
Originally Posted by alberto View Post
Is the code crashing for alpha being too high?

Best,
yes, the value for alpha reach very high level (sometime min and max are -1e-5 and 1e+5). below is the printout from my log file (here alpha goes only from -0.68 up to 1.1). The strange thing is that this case is working fine for single processor. Do you have any idea why this is happening for multi-processor?

Courant Number mean: 0.0280557 max: 0.36132
Max Ur Courant Number = 0.461666
Calculating averages

Time = 0.9655

DILUPBiCG: Solving for alpha, Initial residual = 0.00161798, Final residual = 3.43713e-11, No Iterations 4
Dispersed phase volume fraction = 0.275303 Min(alpha) = -2.12575e-18 Max(alpha) = 0.802811
DILUPBiCG: Solving for alpha, Initial residual = 0.000113185, Final residual = 5.23969e-11, No Iterations 3
Dispersed phase volume fraction = 0.275303 Min(alpha) = -2.11835e-18 Max(alpha) = 0.802806
GAMG: Solving for p, Initial residual = 0.000399759, Final residual = 8.44465e-09, No Iterations 12
time step continuity errors : sum local = 3.62591e-09, global = -7.10158e-10, cumulative = 2.70152e-07
DILUPBiCG: Solving for alpha, Initial residual = 8.22487e-05, Final residual = 6.99567e-11, No Iterations 3
Dispersed phase volume fraction = 0.275303 Min(alpha) = -2.11584e-18 Max(alpha) = 0.802824
DILUPBiCG: Solving for alpha, Initial residual = 8.09589e-06, Final residual = 0.00755801, No Iterations 1001
Dispersed phase volume fraction = 0.275245 Min(alpha) = -0.684932 Max(alpha) = 1.10975
GAMG: Solving for p, Initial residual = 0.00143021, Final residual = 7.14171e-09, No Iterations 19
time step continuity errors : sum local = 3.07254e-09, global = 5.55504e-10, cumulative = 2.70707e-07
DILUPBiCG: Solving for epsilon, Initial residual = 0.00673291, Final residual = 2.54617e-06, No Iterations 5
DILUPBiCG: Solving for k, Initial residual = 0.00751057, Final residual = 3.37159e-06, No Iterations 5

Regards,
Peter
piterw82 is offline   Reply With Quote

Old   June 22, 2010, 12:13
Default
  #7
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,894
Rep Power: 26
alberto will become famous soon enoughalberto will become famous soon enough
Quote:
Originally Posted by piterw82 View Post
yes, the value for alpha reach very high level (sometime min and max are -1e-5 and 1e+5). below is the printout from my log file (here alpha goes only from -0.68 up to 1.1). The strange thing is that this case is working fine for single processor. Do you have any idea why this is happening for multi-processor?
The problem with the particle pressure in twoPhaseEulerFoam is due to how the particle pressure is treated, and to the structure of the solution algorithm, which should be more robust.

I don't know why this happens in parallel. Probably it has something to do with how the domain is decomposed. For example, decompose with the simple method, and try to keep the particle phase in the same processor patch (I know it's not convenient, but it might help understanding).

Best,
__________________
Alberto Passalacqua

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
OpenQBMM - An open-source implementation of quadrature-based moment methods
alberto is offline   Reply With Quote

Old   June 23, 2010, 04:46
Default
  #8
New Member
 
Join Date: Jan 2010
Posts: 7
Rep Power: 7
piterw82 is on a distinguished road
Quote:
Originally Posted by alberto View Post
The problem with the particle pressure in twoPhaseEulerFoam is due to how the particle pressure is treated, and to the structure of the solution algorithm, which should be more robust.

I don't know why this happens in parallel. Probably it has something to do with how the domain is decomposed. For example, decompose with the simple method, and try to keep the particle phase in the same processor patch (I know it's not convenient, but it might help understanding).

Best,
Hello Alberto,

As the domain is rectangular I do the simple method for decomposition. Here is the printout after the decomposeParDic:

/*---------------------------------------------------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 1.6 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
Build : 1.6-f802ff2d6c5a
Exec : decomposePar
Date : Jun 23 2010
Time : 10:02:14
Host : root
PID : 32387
Case : /root/OpenFOAM/root-1.6/run/test_4_bed2
nProcs : 1
SigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Time = 0
Create mesh

Calculating distribution of cells
Selecting decompositionMethod simple

Finished decomposition in 0.01 s

Calculating original mesh data

Distributing cells to processors

Distributing faces to processors

Calculating processor boundary addressing

Distributing points to processors

Constructing processor meshes

Processor 0
Number of cells = 3000
Number of faces shared with processor 1 = 30
Number of processor patches = 1
Number of processor faces = 30
Number of boundary faces = 6230

Processor 1
Number of cells = 3000
Number of faces shared with processor 0 = 30
Number of processor patches = 1
Number of processor faces = 30
Number of boundary faces = 6230

Number of processor faces = 30
Max number of processor patches = 1
Max number of faces between processors = 30

Processor 0: field transfer
Processor 1: field transfer

End.



I have run the test (as you suggested) with alpha phase being always in one processor (I have extended the hight of the test case to 2m instead of 1m). This simulation was running without crushing. Maybe there is a problem with sharing the information of alpha between the processors? If this is the case how can you run parallel simulation with Cirulating Fluidized Bed. Do you have any suggestion what can be done?

Regards,
Peter
piterw82 is offline   Reply With Quote

Old   June 23, 2010, 13:09
Default
  #9
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,894
Rep Power: 26
alberto will become famous soon enoughalberto will become famous soon enough
There is either some missing "correctBoundaryConditions()", which I doubt, or something changes between serial and parallel runs (just guessing: interpolation of ppMagf?).

I run CFB's in the past in parallel, but it was long ago (OF 1.4). Stability problems are extremely common with twoPhaseEulerFoam. As I told you, the algorithm should be revised to make the management of the particle pressure more robust.

Best,
__________________
Alberto Passalacqua

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
OpenQBMM - An open-source implementation of quadrature-based moment methods
alberto is offline   Reply With Quote

Old   June 23, 2010, 14:59
Default
  #10
New Member
 
Join Date: Jan 2010
Posts: 7
Rep Power: 7
piterw82 is on a distinguished road
Quote:
Originally Posted by alberto View Post
There is either some missing "correctBoundaryConditions()", which I doubt, or something changes between serial and parallel runs (just guessing: interpolation of ppMagf?).

I run CFB's in the past in parallel, but it was long ago (OF 1.4). Stability problems are extremely common with twoPhaseEulerFoam. As I told you, the algorithm should be revised to make the management of the particle pressure more robust.

Best,

I have reported the problem with parallel run for "bed2" tutorial in OpenFOAM Bugs, as Bruno advise me to do. I guess there is a change when decomposing the mesh which generates the problem. That could explain why the solver is running on single CPU and crashes with multi processor.
piterw82 is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem on Fluent Tutorial: Horizontal Film Boilig Feng FLUENT 2 April 13, 2013 05:34
Building OpenFOAM on IRIX lakeat OpenFOAM Installation 7 July 16, 2008 07:27
Error using LaunderGibsonRSTM on SGI ALTIX 4700 jaswi OpenFOAM 2 April 29, 2008 10:54
Adiabatic and Rotating wall (Convection problem) ParodDav CFX 5 April 29, 2007 19:13
Problem with icoDyMFoam tutorial matlie OpenFOAM Bugs 10 April 26, 2007 04:51


All times are GMT -4. The time now is 18:51.