CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Programming & Development

Moving mesh crashes in parallel (1.5-dev)

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   September 30, 2009, 13:29
Default Moving mesh crashes in parallel (1.5-dev)
  #1
Member
 
Jean-Peer Lorenz
Join Date: Mar 2009
Location: Rostock, Germany
Posts: 33
Rep Power: 8
jploz is on a distinguished road
Hi Foamers,

I've created a fvDynamicMesh class for moving the mesh based on flow forces in 1.5-dev. When running the case on a single processor everything works as expected. However, if I run the case in parallel it does not work.
For testing purposes the points are set to the (fixed) original coordinates - but i cannot move the points in parallel.

Does not work means:

if I use tetFaceDecomposition the simulation runs (i.e. it does not crash) but never finishes the fvMesh::movePoints step-the processors run with 100% but nothing happens for a really long time (strange)

if I use fv based motion the first mesh update is done but the simulation crashes when the fvMesh::movePoints step is reached for the second time.

Here is the traceback:

Code:
Getting new points
PCG:  Solving for cellMotionUx, Initial residual = 0, Final residual = 0, No Iterations 0
PCG:  Solving for cellMotionUy, Initial residual = 0, Final residual = 0, No Iterations 0
PCG:  Solving for cellMotionUz, Initial residual = 0.00115045894, Final residual = 9.72441498e-06, No Iterations 26
Getting new points (fix)
Calling for move points
[host:08332] *** Process received signal ***
[host:08333] *** Process received signal ***
[host:08334] *** Process received signal ***
[host:08334] Signal: Segmentation fault (11)
[host:08334] Signal code:  (-6)
[host:08334] Failing at address: 0x1f80000208e
[host:08333] Signal: Segmentation fault (11)
[host:08333] Signal code:  (-6)
[host:08333] Failing at address: 0x1f80000208d
[host:08332] Signal: Segmentation fault (11)
[host:08332] Signal code:  (-6)
[host:08332] Failing at address: 0x1f80000208c
[host:08334] [ 0] /lib64/libc.so.6 [0x7f6d6a0c16e0]
[host:08334] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7f6d6a0c1645]
[host:08334] [ 2] /lib64/libc.so.6 [0x7f6d6a0c16e0]
[host:08334] [ 3] /lib64/libc.so.6 [0x7f6d6a105bcb]
[host:08334] [ 4] /lib64/libc.so.6(__libc_malloc+0xa1) [0x7f6d6a107891]
[host:08334] [ 5] /usr/lib64/libstdc++.so.6(_Znwm+0x1d) [0x7f6d6a91cf4d]
[host:08334] [ 6] /usr/lib64/libstdc++.so.6(_Znam+0x9) [0x7f6d6a91d089]
[host:08334] [ 7] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam4ListIcE7setSizeEi+0x33) [0x7f6d6af27593]
[host:08332] [ 0] /lib64/libc.so.6 [0x7f2bffc546e0]
[host:08332] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7f2bffc54645]
[host:08332] [ 2] /lib64/libc.so.6 [0x7f2bffc546e0]
[host:08332] [ 3] /lib64/libc.so.6 [0x7f2bffc98a9a]
[host:08332] [ 4] /lib64/libc.so.6(__libc_malloc+0xa1) [0x7f2bffc9a891]
[host:08332] [ 5] /usr/lib64/libstdc++.so.6(_Znwm+0x1d) [0x7f2c004aff4d]
[host:08332] [ 6] /usr/lib64/libstdc++.so.6(_Znam+0x9) [0x7f2c004b0089]
[host:08332] [ 7] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libinterfaceProperties.so(_ZN4Foam5FieldINS_6VectorIdEEEC1Ei+0x67) [0x7f2c036ee667]
[host:08332] [ 8] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam13primitiveMesh22calcCellCentresAndVolsEv+0xad) [0x7f2c00c0077d]
[host:08332] [ 9] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam13primitiveMesh11cellCentresEv+0x1d) [0x7f2c00c009bd]
[host:08332] [10] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam9polyPatch15faceCellCentresEv+0x58) [0x7f2c00b6a5a8]
[host:08332] [11] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam18processorPolyPatch12initGeometryEv+0xca) [0x7f2c00b8952a]
[host:08332] [12] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam16polyBoundaryMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0xd1) [0x7f2c00bb53e1]
[host:08332] [13] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8polyMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x262) [0x7f2c00be3c12]
[host:08332] [14] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN4Foam6fvMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x119) [0x7f2c0255da89]
[host:08332] [15] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libmyMotionFvMesh.so(_ZN4Foam17translation3DOFfv6updateEv+0x693) [0x7f2c01000fb3]
[host:08332] [16] myInterDyMFoam-1.5-dev [0x4226d9]
[host:08332] [17] /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f2bffc40586]
[host:08332] [18] myInterDyMFoam-1.5-dev [0x41fba9]
[host:08332] *** End of error message ***
[host:08334] [ 8] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8OPstreamC1ENS_7Pstream10commsTypesEiiNS_8IOstream12streamFormatENS3_13versionNumberE+0xcc) [0x7f6d6af2689c]
[host:08334] [ 9] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam18processorPolyPatch12initGeometryEv+0xbf) [0x7f6d6aff651f]
[host:08334] [10] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam16polyBoundaryMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0xd1) [0x7f6d6b0223e1]
[host:08334] [11] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8polyMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x262) [0x7f6d6b050c12]
[host:08334] [12] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN4Foam6fvMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x119) [0x7f6d6c9caa89]
[host:08334] [13] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libmyMotionFvMesh.so(_ZN4Foam17translation3DOFfv6updateEv+0x693) [0x7f6d6b46dfb3]
[host:08334] [14] myInterDyMFoam-1.5-dev [0x4226d9]
[host:08334] [15] /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f6d6a0ad586]
[host:08334] [16] myInterDyMFoam-1.5-dev [0x41fba9]
[host:08334] *** End of error message ***
[host:08333] [ 0] /lib64/libc.so.6 [0x7f6cda8d66e0]
[host:08333] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7f6cda8d6645]
[host:08333] [ 2] /lib64/libc.so.6 [0x7f6cda8d66e0]
[host:08333] [ 3] /lib64/libc.so.6 [0x7f6cda91aa9a]
[host:08333] [ 4] /lib64/libc.so.6(__libc_malloc+0xa1) [0x7f6cda91c891]
[host:08333] [ 5] /usr/lib64/libstdc++.so.6(_Znwm+0x1d) [0x7f6cdb131f4d]
[host:08333] [ 6] /usr/lib64/libstdc++.so.6(_Znam+0x9) [0x7f6cdb132089]
[host:08333] [ 7]
 /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libinterfaceProperties.so(_ZN4Foam5FieldINS_6VectorIdEEEC1Ei+0x67) [0x7f6cde370667]
[host:08333] [ 8] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZNK4Foam9polyPatch15faceCellCentresEv+0x2a) [0x7f6cdb7ec57a]
[host:08333] [ 9] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam18processorPolyPatch12initGeometryEv+0xca) [0x7f6cdb80b52a]
[host:08333] [10] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam16polyBoundaryMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0xd1) [0x7f6cdb8373e1]
[host:08333] [11] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libOpenFOAM.so(_ZN4Foam8polyMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x262) [0x7f6cdb865c12]
[host:08333] [12] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libfiniteVolume.so(_ZN4Foam6fvMesh10movePointsERKNS_5FieldINS_6VectorIdEEEE+0x119) [0x7f6cdd1dfa89]
[host:08333] [13] /home/acuser3/OpenFOAM/OpenFOAM-1.5-dev/lib/linux64GccDPOpt/libmyMotionFvMesh.so(_ZN4Foam17translation3DOFfv6updateEv+0x693) [0x7f6cdbc82fb3]
[host:08333] [14] myInterDyMFoam-1.5-dev [0x4226d9]
[host:08333] [15] /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f6cda8c2586]
[host:08333] [16] myInterDyMFoam-1.5-dev [0x41fba9]
[host:08333] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 8332 on node host exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
Do I need special decomposition in order to run dynamic meshes in parallel or did I miss something else? I suppose dynamic meshes work in parallel, don't they?

Any help or suggestions are welcome!

Last but not least here is a code snippet from my dynamic mesh class using the fvMotion solver. Can I move the mesh in this way?

Code:
    pointVectorField& pointMotionU =
        const_cast<pointVectorField&>
        (
            this->objectRegistry::lookupObject<pointVectorField>("pointMotionU")
        );

    pointMotionU.boundaryField()[bodyPatchID_] == ode_.U().value();

    Info<< "Translation: " << ode_.X().value().x() << tab
                            << ode_.X().value().y() << tab
                            << ode_.X().value().z() << endl;

    Info << "Getting new points" << endl;
//    pointField newPoints = motionPtr_->newPoints();
// this works

    Info << "Getting new points (fix)" << endl;
    pointField newPoints = stationaryPoints_ * 1.0;

    Info << "Calling for move points" << endl;
    movePoints(newPoints);          // <-- this fails
    Info << "Move points done." << endl;

    volVectorField& U = 
        const_cast<volVectorField&>(lookupObject<volVectorField>("U"));
    U.correctBoundaryConditions();

Thank you so much in advance.

Best regards.
Jean-Peer
jploz is offline   Reply With Quote

Old   October 1, 2009, 02:26
Default
  #2
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,758
Rep Power: 21
hjasak will become famous soon enough
You need to run blocking comms: everyting will be fine with that.

I'll have a closer look in 1.6-dev merge...

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   October 1, 2009, 05:56
Default
  #3
Member
 
Jean-Peer Lorenz
Join Date: Mar 2009
Location: Rostock, Germany
Posts: 33
Rep Power: 8
jploz is on a distinguished road
Ah...ok, it works flawless now.
Thank you.

Jean-Peer
jploz is offline   Reply With Quote

Old   October 13, 2009, 15:54
Default Parallel Mesh Motion Error 1.5-dev
  #4
New Member
 
Rob Campbell
Join Date: Oct 2009
Posts: 12
Rep Power: 7
rlc138 is on a distinguished road
Hi.

I am having trouble moving my dynamic mesh in parallel. It seems that vertices shared by more than two processors are not moved correctly. It works fine in serial mode. I have the commsType set to blocking and I am using the laplaceFaceDecomposition solver.

I have a simple test case with a fin in a rectangular flow field. I specify a velocity of the fin's patch only in the y-direction and solve using moveDynamicMesh. The figure below shows vertices on two processors in blue and vertices shared by three processors in red. The vertices in red are not being moved correctly. Can anyone help???

Rob

rlc138 is offline   Reply With Quote

Old   October 21, 2009, 05:37
Default
  #5
Senior Member
 
Mark Couwenberg
Join Date: Mar 2009
Location: Netherlands
Posts: 130
Rep Power: 8
markc is on a distinguished road
Hello All,

Just to share my experience with you: I experienced similar problems with crashes when running moving-mesh cases in parallel. I suspect that the original mesh needs to have an even number of cells. So far I have seen now everything runs fine in that case.

Brgds,

Mark
markc is offline   Reply With Quote

Old   October 21, 2009, 08:55
Default
  #6
New Member
 
Rob Campbell
Join Date: Oct 2009
Posts: 12
Rep Power: 7
rlc138 is on a distinguished road
Mark:

Thanks for the input.

I checked my case and it has 6000 cells (all hexahedral cells with 7005 points).

The decomposed mesh, however, does not have an even number for all processors. Do you think this is a problem?

Rob

Processor 0
Number of cells = 738
Processor 1
Number of cells = 772
Processor 2
Number of cells = 738
Processor 3
Number of cells = 769
Processor 4
Number of cells = 760
Processor 5
Number of cells = 728
Processor 6
Number of cells = 760
Processor 7
Number of cells = 735
rlc138 is offline   Reply With Quote

Old   October 21, 2009, 09:01
Default
  #7
Senior Member
 
Mark Couwenberg
Join Date: Mar 2009
Location: Netherlands
Posts: 130
Rep Power: 8
markc is on a distinguished road
Hello Rob,

I have no idea whether this is a problem. I only noted for a few cases I have done
so far that this may be a clue.

Brgds,

Mark
markc is offline   Reply With Quote

Old   October 21, 2009, 14:31
Default
  #8
Senior Member
 
Mark Couwenberg
Join Date: Mar 2009
Location: Netherlands
Posts: 130
Rep Power: 8
markc is on a distinguished road
Btw, Hrv mentiones to use "blocking comms". Uh, what's that?

Brgds,

Mark
markc is offline   Reply With Quote

Old   October 21, 2009, 17:22
Default
  #9
New Member
 
Rob Campbell
Join Date: Oct 2009
Posts: 12
Rep Power: 7
rlc138 is on a distinguished road
Mark:

I already have the comms set to blocking. This controls how the interprocessor messages are sent/received and is set in foam/etc/controlDict and the options are: blocking, scheduled, or nonblocking. BTW, I have posted this issue as an OpenFOAM bug, to which Hrv has responded: Parallel Mesh Motion Error 1.5-dev .

Rob
rlc138 is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Moving mesh part 2 lr103476 OpenFOAM Running, Solving & CFD 49 December 14, 2010 07:20
salome, openfoam and moving mesh prhlava OpenFOAM Running, Solving & CFD 8 November 9, 2009 09:59
Moving Mesh with Ansa File Paul1985 OpenFOAM 1 August 20, 2009 02:13
moving mesh in parallel mode Karteek CD-adapco 4 June 16, 2008 04:12
CavitatingFoam with a moving mesh mass conservation error idosil OpenFOAM Running, Solving & CFD 3 November 27, 2007 18:27


All times are GMT -4. The time now is 23:01.