CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

Moving mesh still crashes in parallel in 1.6-ext

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   February 18, 2011, 11:59
Default Moving mesh still crashes in parallel in 1.6-ext
  #1
New Member
 
Paul Bomke
Join Date: Mar 2010
Posts: 14
Rep Power: 7
paul b is on a distinguished road
Dear forum,

trying to run moving mesh cases in parallel using the laplaceFaceDecomposition motion solver I encounter two problems:

1) In 1.5-dev there has been a bug related to the points on processor boundaries not being moved correctly during parallel runs. See this post for reference:

Parallel Mesh Motion Error 1.5-dev

This is supposed to be corrected in the new 1.6-ext version, however I still encounter that error. Comms are set to blocking, relaxing matching tolerances just delays the problem. To illustrate this I set up a 2D case using dynamicBodyFvMesh (colours show processors):



2) Another 3d case, again using dynamicBodyFvMesh runs fine first, but then suddenly the motion solver diverges (1000 iterations) and after several timesteps crashes. The resulting mesh is completely distorted.

Before (colored patches = processor boundaries):


After:



I tried this using the moveDynamicMesh utility and ICCG as well as PCG solvers for motionU (AMG solver does not seem to work any more for solving the motionU field).

What am I doing wrong or whats still wrong with the code?

If anybody had the time to take a look at my cases I'd be really glad. I'm trying to du parallel runs with that motion solver for about half a year now and thought the issue would be solved with the new ext release.

Hoping for your help!

Sincerely, Paul

ps. I can email the cases, also some videos, the board won't let me upload files of that size...
paul b is offline   Reply With Quote

Old   February 18, 2011, 12:52
Default
  #2
Senior Member
 
Sandeep Menon
Join Date: Mar 2009
Location: Amherst, MA
Posts: 386
Rep Power: 15
deepsterblue will become famous soon enough
Hrv suggested using GMRES instead, but I still had problems similar to what you're seeing. You could try this (might need to tweak it a bit):

Code:
    
    motionU
    {
        solver          GMRES;
        nDirections     8;
        preconditioner  Cholesky;
        tolerance       1e-09;
        relTol          0.0001;
    }
Otherwise, if you don't mind restricting yourself to tri-prism / tet meshes, you can look at mesquiteMotionSolver instead, which I've managed to hack to work in parallel (see attached file)

These options tend to work okay for me (might need some tweaking too):

Code:
mesquiteOptions
{
    // Optimization metric
    optMetric               AspectRatioGamma;

    // Objective function
    objFunction             LPtoP;

    // Optimization algorithm
    optAlgorithm            FeasibleNewton;

    // Termination criteria sub-dictionary (takes default values if not specified)
    // Specifying an empty sub-dictionary terminates with available options
    tcInner
    {
       relGradL2            1e-2;
       cpuTime              0.5;
    }
    // tcOuter
    // {}

    // For composite functions, two objectives need to be specified
    firstFunction           LPtoP;
    secondFunction          LInf;

    // For scaled functions, scale and objective needs to be specified
    // scaleFunction        PMeanP;
    // scale                1.5;

    // Power value for the LPtoP objective function
    pValue                  2;
    power                   2;

    // Specify a tolerance for the CG solver
    tolerance               1e-4;

    // Specify number of CG sweeps
    nSweeps                 1;

    // Specify slip patches for the motionSolver
    slipPatches
    {
        sideWall;
        topWall;
    }

    /*
    cylindricalConstraints
    {
        // Specify options per slip patch
        sideWall
        {
            axisPoint      (0.0 0.0 0.0);
            axisVector     (0.0 0.0 1.0);
            radius          1.0;
        }
    }
    */

    // Specify fixedValue patches for the motionSolver
    fixedValuePatches
    {
        topWall
        {
            //type          angularOscillatingDisplacement;
            //amplitude     -0.0125;
            type          oscillatingDisplacement;
            amplitude     (0 0 -0.01);
            axis          (1 0 0);
            origin        (0 0 3);
            angle0        0.0;
            omega         0.15;
            value         uniform (0 0 0);
        }
    }

    // Specify interval for surface smoothing
    surfInterval            1;
}
Attached Files
File Type: gz mesquiteMotionSolver.gz (20.4 KB, 48 views)
__________________
Sandeep Menon
University of Massachusetts Amherst
https://github.com/smenon
deepsterblue is offline   Reply With Quote

Old   February 18, 2011, 16:01
Default
  #3
New Member
 
Paul Bomke
Join Date: Mar 2010
Posts: 14
Rep Power: 7
paul b is on a distinguished road
Hey Sandeep,

thanks a lot for the advice!
I will try this when I'm back at work on monday and report how far I get.

Regards,
Paul
paul b is offline   Reply With Quote

Old   February 21, 2011, 05:36
Default
  #4
New Member
 
Paul Bomke
Join Date: Mar 2010
Posts: 14
Rep Power: 7
paul b is on a distinguished road
Hi Sandeep,

first of all: this is exactly what I was looking for! I'm working on simulating animal locomotion, flying birds and so on, and we often have to deal with large mesh deformation. At the moment I'm using a mesh class which I build using a axisymmetric mesh which encloses the object (bird wing etc.) and which can rotate and translate in the surrounding mesh. The deformation of the latter is solved by a motion solver and coupling between meshes is done by ggi.
However, now I'd like to use the mesquite motion solver for the surrounding mesh and I'm a little stuck at this point.
1) the solver you posted compiled fine.
2) In my dynamicFvMesh class I used the tetDecompositionMotionSolver. The only thing I needed to do in order to define the motion of my moving patches was get the pointer to the motion solver object:

Code:
tetDecompositionMotionSolver& mSolver =
                dynamic_cast<tetDecompositionMotionSolver&>
                (
                motionPtr_()
                );
Then I could access the mesh motion velocity field motionU:

Code:
tetPointVectorField& motionU = mSolver.motionU();
and use this pointer to set the desired mesh motion on my moving patch.

But now with the mesquite solver by looking at the code there only seems to be a way of specifying the motion of the moving patches via the method:
Code:
applyFixedValuePatches()
which reads from the dynamicMeshDict.

To make a long story short: is there a way to specify the motion of my moving patch from within my dynamicFvMesh class or do I have to implement such a feature?
paul b is offline   Reply With Quote

Old   February 21, 2011, 06:01
Default
  #5
New Member
 
Paul Bomke
Join Date: Mar 2010
Posts: 14
Rep Power: 7
paul b is on a distinguished road
Ok, I had a further look at that applyFixedValuePatches() method of the mesquiteMotionSolver. Seems like the movement of the fixedValuePatches is specified via moving boundary conditions which are simply read from the dynamicMeshDict. Thus the part

Code:
pField().updateCoeffs();
where the motion for the point field pField happens has to be replaced by my own motion specification which I might pass to the mesquiteMotionSolver as an argument of the applyFixedValuePatches() method.

Please correct me if this is nonsense!

I'll try...
paul b is offline   Reply With Quote

Old   February 21, 2011, 22:29
Default
  #6
Senior Member
 
Sandeep Menon
Join Date: Mar 2009
Location: Amherst, MA
Posts: 386
Rep Power: 15
deepsterblue will become famous soon enough
I would advise against source-code modification. It's possible to look-up the "refPoints" pointField from the objectRegistry and assign boundary displacement values.

Code:
        
        // Boundary motion specified for the motionSolver
        pointField& refPoints = const_cast<pointField&>
        (
            mesh.lookupObject<pointField>("refPoints")
        );

        // Assign boundary conditions to the motion solver at patch 'pI'
        const labelList& meshPts = mesh.boundaryMesh()[pI].meshPoints();

        // Apply displacement
        forAll(meshPts,pointI)
        {
            refPoints[meshPts[pointI]] += dispField[pointI];
        }
__________________
Sandeep Menon
University of Massachusetts Amherst
https://github.com/smenon
deepsterblue is offline   Reply With Quote

Old   February 22, 2011, 04:11
Default
  #7
New Member
 
Paul Bomke
Join Date: Mar 2010
Posts: 14
Rep Power: 7
paul b is on a distinguished road
Sandeep,

thanks a lot for the advice, this seems to be a more elegant way!

I'll post again if I succeed...
paul b is offline   Reply With Quote

Old   February 22, 2011, 05:17
Default
  #8
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,758
Rep Power: 21
hjasak will become famous soon enough
Hi Paul,

Can I please have this case to look at - this MUST WORK and if it does not I need to know what happened.

We are running lots of parallel mesh motion and there are no open issues.

As for parallel AMG for FEM, you will appreciate they agglomerative AMG like the one used in the FVM cannot parallelise, so we need Selective AMG coarsening. This currently works in serial and I know who will parallelise this

Please let me know - I guess you can easily find my E-mail.

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   February 22, 2011, 09:36
Default
  #9
Senior Member
 
Karl-Johan Nogenmyr
Join Date: Mar 2009
Location: Linköping
Posts: 252
Rep Power: 12
kalle is on a distinguished road
Hi Sandeep,

cool to see you did a parallel hack! I compiled it and tried it on the simpleEngine tutorial you made last month. Just running a decomposPar with method simple and splitting in two domains gave me the same error as before: (at the first time step)

Code:
[1] Cannot find point in pts1 matching point 2 coord:(-2.11212 -0.726195 0.247738) in pts0 when using tolerance 4.38945e-05
[1] Searching started from:0 in pts1
[0] Cannot find point in pts1 matching point 4 coord:(-7.88788 -2.27381 -0.247738) in pts0 when using tolerance 4.38945e-05
[1] Compared coord:(-2.11212 -0.726195 -0.247738) with difference to point 0.495476
[0] Searching started from:2 in pts1
[0] Compared coord:(-2.21917 -0.326676 0.247738) with difference to point 6.01424
[0] Compared coord:(-7.88788 -2.27381 0.247738) with difference to point 0.495476
[0] 
[0] 
[0] --> FOAM FATAL ERROR: 
[0]  Failed point match: 
 My proc: 0
 neiProcNo: 1
 size(bufPoints): 20
 size(recvField): 20
...
Would there be some additional settings to get the case running in parallel?

Regards,
Kalle
kalle is offline   Reply With Quote

Old   February 22, 2011, 09:46
Default
  #10
Senior Member
 
Sandeep Menon
Join Date: Mar 2009
Location: Amherst, MA
Posts: 386
Rep Power: 15
deepsterblue will become famous soon enough
Kalle,
Could you post the case so that I can take a look at it?
__________________
Sandeep Menon
University of Massachusetts Amherst
https://github.com/smenon
deepsterblue is offline   Reply With Quote

Old   February 22, 2011, 10:09
Default
  #11
Senior Member
 
Karl-Johan Nogenmyr
Join Date: Mar 2009
Location: Linköping
Posts: 252
Rep Power: 12
kalle is on a distinguished road
Sure, of course. I've emailed it to you... (why having a limit of 97kB year 2011??) The log file is found below.

It is almost identical to the case you posted here. I only added a decomposParDict and decomposed the case. I've also uploaded the log file.

Regards,
Kalle
Attached Files
File Type: txt log.txt (3.5 KB, 17 views)
kalle is offline   Reply With Quote

Old   February 22, 2011, 10:26
Default
  #12
Senior Member
 
Sandeep Menon
Join Date: Mar 2009
Location: Amherst, MA
Posts: 386
Rep Power: 15
deepsterblue will become famous soon enough
Yup, there are still bugs that need to be fixed. The trouble with this particular case is that the 'valveSide' patch has points on the parallel patch on edge-boundaries as well. I suppose you could use the modified configuration that is attached.

Meanwhile, I'll look into a fix. Also, please be aware that the dynamicTopoFvMesh class is not fully parallelized for topology changes (yet), so you'll be stuck with mesh-motion only at the moment.
Attached Files
File Type: txt dynamicMeshDict.txt (5.0 KB, 83 views)
__________________
Sandeep Menon
University of Massachusetts Amherst
https://github.com/smenon
deepsterblue is offline   Reply With Quote

Old   February 23, 2011, 03:29
Default
  #13
Senior Member
 
Karl-Johan Nogenmyr
Join Date: Mar 2009
Location: Linköping
Posts: 252
Rep Power: 12
kalle is on a distinguished road
Thanks! Now it runs (as long as I don't decompose in the axial direction). Speed up seems to be good too!

Cool work this library!


Regards,
Kalle
kalle is offline   Reply With Quote

Old   February 28, 2011, 04:23
Default
  #14
New Member
 
Paul Bomke
Join Date: Mar 2010
Posts: 14
Rep Power: 7
paul b is on a distinguished road
Dear all,

the problem that I described in the first post is solved. Hrv was so kind as to have a look at my cases and he found that due to the metis decomposition which I used some strange edge decomposition occured which was not accounted for in the tetDecompMotionSolver (maybe Hrv could elaborate a bit further...).

He provided a fix for the problem, you can find it in the attachement.
Extract the stuff into OpenFOAM-1.6-ext/scr/tetDecompositionFiniteElement, rebuild and all will be well.


Thanks to Hrv for fixing the issue, and also thanks to Sandeep for pointing me towards the mesquite motion solver!

Regards, Paul
Attached Files
File Type: gz motionSolverFix.tar.gz (18.6 KB, 59 views)
paul b is offline   Reply With Quote

Old   April 14, 2011, 09:20
Default
  #15
Senior Member
 
Join Date: Apr 2010
Posts: 151
Rep Power: 7
flowris is on a distinguished road
Dear all,


I have also been trying to run dynamicFvMesh in parallel, as well in 1.5-dev as 1.6-ext. i try to make a cylinder move inside a mesh. In serial, all goes well, but in parallel, some cells deform (see screenshot) until they become negative and move uncontrolled. These cells are on and near the processor boundaries.

I installed the moitionSolverFix, but this did not fix the problem. I also tried several solver for motionU. If anybody is willing to check my case, I will send the necessary files to her/him.


Kind regards,


Joris
Attached Images
File Type: jpg Screenshot.jpg (99.7 KB, 90 views)
flowris is offline   Reply With Quote

Old   April 14, 2011, 09:32
Default
  #16
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,758
Rep Power: 21
hjasak will become famous soon enough
In the test you talk about we were doing simple mesh motion and this works without a problem.

In the test you have set up, there is a topological change, which is different. You need updates in the dynamic mesh support + a change in the dynamic mesh class + updated domain decomposition and reconstruction tool. I would recommend getting some support if you need this asap, or wait for the updated examples.

The 1.5-dev and 1.6-ext available to you has got a part of the features, but the new stuff makes it easier, faster and more robust.

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   April 15, 2011, 02:59
Default
  #17
Senior Member
 
Join Date: Apr 2010
Posts: 151
Rep Power: 7
flowris is on a distinguished road
Thank you, though this is quite discouraging. While waiting for updates (how long will this take?), I was wondering if I can do this trick:
- first move the mesh in serial and save it at each time step, which I think is possible
- use it for a parallel simulation, with fixed time step
flowris is offline   Reply With Quote

Old   April 15, 2011, 06:03
Default
  #18
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,758
Rep Power: 21
hjasak will become famous soon enough
I am sorry, I do not understand: what do you mean "though this is quite discouraging"? Could you please explain.

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   April 15, 2011, 07:52
Default
  #19
Senior Member
 
Join Date: Apr 2010
Posts: 151
Rep Power: 7
flowris is on a distinguished road
Hello,

Never mind about that, I was just hoping to get my cases running in a couple of days, but it will take some more effort. Do you think the trick I propose is possible?
flowris is offline   Reply With Quote

Old   April 18, 2011, 08:11
Default
  #20
Member
 
Oliver Borm
Join Date: Mar 2009
Posts: 59
Rep Power: 8
deepblue17 is on a distinguished road
The laplaceFaceDecomposition and displacementLaplacian motion solver will only work in parallel if commsType is set scheduled or blocking. If commsType is set to nonBlocking they will crash.
deepblue17 is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
3D Hybrid Mesh Errors DarrenC ANSYS Meshing & Geometry 11 August 5, 2013 06:42
2D Mesh Generation Tutorial for GMSH aeroslacker Open Source Meshers: Gmsh, Netgen, CGNS, ... 12 January 19, 2012 04:52
Create moving mesh without simulating (CFX) spatialtime ANSYS 2 July 22, 2010 10:30
salome, openfoam and moving mesh prhlava OpenFOAM Running, Solving & CFD 8 November 9, 2009 09:59
moving mesh in parallel mode Karteek CD-adapco 4 June 16, 2008 04:12


All times are GMT -4. The time now is 14:45.