CFD Online Discussion Forums - Case running in parallel gives error whereby running in serial works

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM Running, Solving & CFD (https://www.cfd-online.com/Forums/openfoam-solving/)

- - Case running in parallel gives error whereby running in serial works (https://www.cfd-online.com/Forums/openfoam-solving/157302-case-running-parallel-gives-error-whereby-running-serial-works.html)

Case running in parallel gives error whereby running in serial works

Hello,

I want to run a case in parallel with 2 processors. Therefore I decomposed my mesh with the scotch method and with 2 subdomains. My case is about a 3D flow simulation hitting a beam and flowing around it.

I've already run it in serial and it just worked fine but when I tried in parallel following error occurs:

Code:

Time = 0.00045, iteration: 3

Current fsi under-relaxation factor (Aitken): 1.0185847

Maximal accumulated displacement of interface points: 0.0073361798

[0] 

[0] 

[0] --> FOAM FATAL ERROR: 

[0] face 84 area does not match neighbour by 0.0122327% -- possible face ordering problem.

patch: procBoundary0to1 my area: 4.54816e-08 neighbour area: 4.54761e-08 matching tolerance: 0.0001

Mesh face: 10126 vertices: 4((0.00506497 0.00556421 0.000275998) (0.005268 0.0055643 0.000275994) (0.00526801 0.0055643 0.000499999) (0.00506497 0.00556421 0.0005))

Rerun with processor debug flag set for more information.[1] 

[1] 

[1] --> FOAM FATAL ERROR: 

[1] face 84 area does not match neighbour by 0.0122327% -- possible face ordering problem.

patch: procBoundary1to0 my area: 4.54761e-08 neighbour area: 4.54816e-08 matching tolerance: 0.0001

Mesh face: 9704 vertices: 4((0.00506497 0.00556421 0.000275998) (0.00506497 0.00556421 0.0005) (0.00526798 0.0055643 0.000499999) (0.00526798 0.00556431 0.000275994))

Rerun with processor debug flag set for more information.

[1] 

[1]     From function processorPolyPatch::calcGeometry()

[1]     in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 

[0] 

[0]     From function processorPolyPatch::calcGeometry()

[0]     in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 217.

[0] 

FOAM parallel run exiting

[0] 

217.

[1] 

FOAM parallel run exiting

[1] 

--------------------------------------------------------------------------

MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 

with errorcode 1.



NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.

You may or may not see output from other processes, depending on

exactly when Open MPI kills them.

--------------------------------------------------------------------------

Here's my decomposeParDict:

Code:

/*---------------------------------------------------------------------------*\

| =========                 |                                                 |

| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |

|  \\    /   O peration     | Version:  3.0                                |

|   \\  /    A nd           | Web:      http://www.openfoam.org               |

|    \\/     M anipulation  |                                                 |

\*---------------------------------------------------------------------------*/

 

FoamFile

{

    version         2.0;

    format          ascii;

 

     

    class           dictionary;

    object          decomposeParDict;

}

 

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

 

 

numberOfSubdomains 2;

 

method          scotch;

 

simpleCoeffs

{

    n               (4 1 1);

    delta           0.0001;

}



globalFaceZones ( InterfaceZone );





 

hierarchicalCoeffs

{

    n               (1 1 1);

    delta           0.001;

    order           xyz;

}

 

metisCoeffs

{

    processorWeights 

    (

        1

        1

        1

    );

}

 

manualCoeffs

{

    dataFile        "";

}

 

distributed     no;

 



 

 

// ************************************************************************* //

Is it about the matching tolerance? If yes, how can I increase this tolerance or should I even do it? Or is there any other opportunity to solve this kind of problem? :confused:

Thank you all in advance :)

Quote:

Originally Posted by wyldckat (Post 556969)

Then it will unlikely work for your case :(

I've added it to my to-do list, but I have no idea when I'll be able to look into it, because it's not a straight forward answer. In addition, it's best that you indicate which OpenFOAM version/variant you're using.

You're just awesome, Bruno! Many thanks :)
I'm using OpenFOAM 2.3.0 and foam-extend 3.1. In this particularly case I'm using foam-extend because my study is about a fluid-structure-problem. The solver is called icoFsiElasticNonLinULSolidFoam. But I think my problem is independent of whether I use OpenFOAM or foam-extend…

I really appreciate your help! :D

Greetings Harak,

OK, time flew by so fast :(
Anyway I've finally managed to take a look into this today and... I'm confused.
I've tested just now the tutorial "solidMechanics/icoFsiElasticNonLinULSolidFoam/HronTurekFsi" in both foam-extend 3.0 and 3.1, by running the script:

Code:

./AllrunPar

and I didn't have any problems.

Without at least an example case, I can't reproduce the same error message you're getting. And there aren't enough details for me to diagnose how this could be fixed.

Best regards,
Bruno

Quote:

Originally Posted by wyldckat (Post 564871)

Code:

./AllrunPar

Hi Bruno,

you wouldn't believe how much I was waiting for your response :D

You'll find my case here:
https://www.dropbox.com/s/hl5e8bfy09...se.tar.gz?dl=0

Please read the README first, because it gives you a small overview about my case and explains how the different folders are connected to each other.
I forgot to write there that you should only run the FSI-case (icoFsiElasticNonLinULSolidFoam) with foam-extend 3.1! All other "preparation-cases" should be run with OpenFOAM 2.3.0 because there might be problems with my symmetry patch as foam-extend wants it named 'symmetryPlane' whereas OpenFOAM handles it with just 'symmetry'.

I'm urgely waiting for your answer! :)

Hi Harak,

I've finally managed to look into your case... I think I did the steps correctly for using your case. And I have to say that you're approaching this problem from a very wrong perspective.

Let's see if I'm able to explain myself. Attached is the image "surface_meshes.png" which shows:

In grey with the edges in blue is the patch "consoleFluid". It's the surface mesh on the fluid side.
In white is the mesh for the solid side, namely the patch "solid/consoleSolid".

Now, if it doesn't seem clear to you right away, the problem is that the two meshes don't match very well. The problems are:

The corners, where there are faces that are inclined between the two main surfaces, which means that even before they deform, it's already not able to interface between the fluid and the solid surfaces on those faces.
Around the curved surface of the "roving", the faces on the two surface meshes have problems matching between them as well.
Last but not least, even if you fix the two previous mesh errors, I'm not sure that you will be able to run an FSI simulation with this kind of mesh interface.

It would be preferable if you have a 1:1 face match for each face pair on the two surface meshes, because in this mesh you have several situations of partial correspondences between faces, along with 4:1 or 8:1 matches of faces, which can lead to some very complicated calculations. Worse even is that I suspect that you will end up with non-matching deflections, if the face matching is not uniform; what I mean by this is that the solid surface will deform in such a way that the fluid surface will go into the solid region, which should not happen in this kind of simulation.

As for taking the wrong approach, this is very common in beginners: when people get started with CFD, they tend to go head first into the final problem that they are trying to solve. This is like jumping off a 150m cliff in a pool with unknown water depth, which will likely result in the jumper getting hurt or worse, if the water depth is not enough for absorbing the jumper's momentum.

My advice: take several steps back. You need to use a much simpler geometry, using a much less refined mesh, namely something that you can first learn how things work. Then gradually add complexity and check how things are working after each change. Otherwise, you're trying to solve 100 or more problems by mere luck. I wrote about this very same issue 11h ago for someone else who is making the similar errors in how to approach the resolution of their problem, which you can find here: http://www.cfd-online.com/Forums/ope...tml#post567548 - post 14.

The attached image was gotten by relying on the instructions given at the following locations:

http://openfoamwiki.net/index.php/FA...is_in_ParaView

http://www.cfd-online.com/Forums/ope...tml#post563782 - post #7

Quote:

Originally Posted by wyldckat (Post 563782)

You can see the two meshes (fluid and solid regions) by running paraFoam like this:

Code:

paraFoam -nativeReader

I did try running your case in serial mode first, to see if the problem also happens when running in serial mode, but my machine doesn't have enough RAM for running such a heavy case. Therefore I was not able to reproduce the same problem, but my description above about the surface mesh is the most probable reason for the crash you're witnessing.

Trying to solve the original error message with a mesh that has 1.6 million cells is what I would refer to "jumping head first into a pool without first knowing the water depth it has"...

And as I mentioned to the other person, if you're up to reading on how to deal with OpenFOAM and foam-extend: http://openfoamwiki.net/index.php/Tu...etting_Started

Best regards,
Bruno

Hi Bruno,

I really appreciate your help and your effort!

Quote:

As for taking the wrong approach, this is very common in beginners: when people get started with CFD, they tend to go head first into the final problem that they are trying to solve. This is like jumping off a 150m cliff in a pool with unknown water depth, which will likely result in the jumper getting hurt or worse, if the water depth is not enough for absorbing the jumper's momentum.

My advice: take several steps back. You need to use a much simpler geometry, using a much less refined mesh, namely something that you can first learn how things work. Then gradually add complexity and check how things are working after each change. Otherwise, you're trying to solve 100 or more problems by mere luck.

Actually, this geometry is not my final geometry. As you said, I've already taken some steps back and modified the geometry to make it easier. In fact, the rovings have an elliptical profile and my geometry contains not only two of them. Besides, my inlet have a quarter (because of symmetry) circular cross section. But I understand what you mean and because of that, I've made them much easier by setting up some two dimensional simulations and created both geometries with blockMeshDict and they are actually working very well!

Quote:

Now, if it doesn't seem clear to you right away, the problem is that the two meshes don't match very well. The problems are:

The corners, where there are faces that are inclined between the two main surfaces, which means that even before they deform, it's already not able to interface between the fluid and the solid surfaces on those faces.
Around the curved surface of the "roving", the faces on the two surface meshes have problems matching between them as well.
Last but not least, even if you fix the two previous mesh errors, I'm not sure that you will be able to run an FSI simulation with this kind of mesh interface.

This is indeed something I've never thought about and absolutely reasonable, too. Actually it suits very well with working and not working cases I did besides to find the reason for this problem. I'm still checking it...

Quote:

I ran the exact same case in serial again. In comparison to parallel computing it is running...No error occurred until now.

Quote:

The attached image was gotten by relying on the instructions given at the following locations:

http://openfoamwiki.net/index.php/FA...is_in_ParaView
HronTurekFsi3 foam-extend3.1 - post #7
Quote:
Originally Posted by wyldckat http://www.cfd-online.com/Forums/ima...s/viewpost.gif
You can see the two meshes (fluid and solid regions) by running paraFoam like this:
Code:
paraFoam -nativeReader

Unfortunately, I wasn't able to get the same image as you. I executed paraFoam -nativeReader in the fluid directory but it didn't worked... could you please explain it further?

Thank your very much!!!

Best regards,
Harak

Quote:

Unfortunately, I wasn't able to get the same image as you. I executed paraFoam -nativeReader in the fluid directory but it didn't worked... could you please explain it further?

Nevermind, I could reproduce it now. My mistake was that I haven't run the case at all before executing paraFoam -nativeReader :D

Hey Bruno,

In the last three days I did some research concerning the non matching meshes.

Here's what I've found:

Strangely, the case is running in serial without any problem. It's been almost 69 hours by now and it didn't abort. Quite the contrary, I get some time directories which I haven't checked yet.

However, my main focus was on the problem that the two meshes don't match very well. So, I took another 3D-case, namely "OneToOne_FaceMatch", where I created both meshes with blockMeshDict aiming a 1:1 face match as you suggested. See both images below;

The blue-grey one is the patch "consoleFluid" on the fluid side.
The other image shows the mesh for the solid side (white mesh) lying on top of each other.

My discoveries are:

The case is now running in parallel BUT only with 2 processors.
Running with more than 2 processors leads to the same problem with the face ordering problem, again. I tried it with 4, 8, 16 and 32 processors.

See attached the log-files for running the case "OneToOne_FaceMatch" once with 2 processors
https://www.dropbox.com/s/zhltha7pdm...rocessors?dl=0
and one with 4 processors
https://www.dropbox.com/s/ft0a60s1ll...rocessors?dl=0.
And yes, the case with 2 processors crashes after a while due to high Courant numbers. I'm aware of that :)

I've also attached this case if you want to have a look into that. You can just run it, there aren't any "preparation-cases" for this one. . :)

I'm looking forward to hear from you! ;)

Best regards,
Harak.

Quick answer: I don't have time right to test this, but I suspect that you are stepping into a limitation for this solver. You're using the "scotch" decomposition, which tries to achieve the best balance in mesh distribution between processors.

But this kind of simulation needs consistency in how each sub-domain can interact between fluid and solid. Try using either the "hierarchical" or "simple" decomposition and make sure that you decompose the mesh in a way that either the whole solid and surrounding fluid is part of a single sub-domain, or make sure that each section of the solid has its associated fluid region in the same sub-domain.