CFD Online Discussion Forums - Case running in serial, but Parallel run gives error

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM Running, Solving & CFD (https://www.cfd-online.com/Forums/openfoam-solving/)

- - Case running in serial, but Parallel run gives error (https://www.cfd-online.com/Forums/openfoam-solving/109128-case-running-serial-but-parallel-run-gives-error.html)

Case running in serial, but Parallel run gives error

Dear all,

Im using OpenFOAM 2.1 and am implementing the cyclic boundary condition for my case in both the streamwise and spanwise directions (its a kind of LES simulation of channel flow).

I use the following in my boundary file(and likewise for spanwise patches):
inlet
{
type cyclic;
nFaces 23808;
startFace 13950208;
matchTolerance 0.01;
neighbourPatch outlet;
}
outlet
{
type cyclic;
nFaces 23808;
startFace 13974016;
matchTolerance 0.01;
neighbourPatch inlet;
}

This works fine when I run the code in serial. However, in parallel, I get this error:

--> FOAM FATAL ERROR:
[6] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.
patch:procBoundary6to11throughinlet my area:0.000103534 neighbour area:0.00010351 matching tolerance:1.49818e-08
Mesh face:1174459 vertices:4((-3.74192e-08 1.05217 2.2654) (-3.74192e-08 1.05217 2.2895) (-3.74192e-08 1.05647 2.2895) (-3.74192e-08 1.05647 2.2654))
If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.
Rerun with processor debug flag set for more information.
[6]
[6] From function processorPolyPatch::calcGeometry()
[6] in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.
[6]
FOAM parallel run exiting

No matter how much I relax the tolerance I get the same error. I think this is due to the processor boundary-neighbour mismatch. Has anyone encountered this before and solved this???

Also, does the "processor" boundary condition help in any way? I couldn't find much documentation for it.

Thank you!!

Case running in serial, but Parallel run gives error - Please help! urgent !

Hello all,

I had posted a similar query some days back but couldn't get any replies. I am being more detail now... I hope I get some feedback.
I am running an LES case using channelFoam. the boundary conditions are CYCLIC in streamwise and spanwise directions.

I can run my case in serial, but when I try to run it in parallel, I get the following error.

Code:

 

[3] 

[3] 

[3] --> FOAM FATAL ERROR: 

[3] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.

patch:procBoundary3to5throughinlet my area:0.000103534 neighbour area:0.00010351 matching tolerance:1.49818e-08

Mesh face:2347842 vertices:4((-3.74192e-08 1.05217 2.2654) (-3.74192e-08 1.05217 2.2895) (-3.74192e-08 1.05647 2.2895) (-3.74192e-08 1.05647 2.2654))

If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.

Rerun with processor debug flag set for more information.

[3] 

[3]     From function processorPolyPatch::calcGeometry()

[3]     in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.

[3] 

FOAM parallel run exiting

[3] 

[2] 

[2] 

[2] --> FOAM FATAL ERROR: 

[2] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.

patch:procBoundary2to0throughoutlet my area:0.00010351 neighbour area:0.000103534 matching tolerance:1.49814e-08

Mesh face:2318738 vertices:4((9 1.05217 0.0241) (9 1.05646 0.0241) (9 1.05646 0.0482) (9 1.05217 0.0482))

If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.

Rerun with processor debug flag set for more information.

[2] 

[2]     From function processorPolyPatch::calcGeometry()

[2]     in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.

[2] 

FOAM parallel run exiting

[2] 

[0] 

[0] 

[0] --> FOAM FATAL ERROR: 

[0] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.

patch:procBoundary0to2throughinlet my area:0.000103534 neighbour area:0.00010351 matching tolerance:1.49816e-08

Mesh face:2331016 vertices:4((-3.74192e-08 1.05217 0.0241) (-3.74192e-08 1.05217 0.0482) (-3.74192e-08 1.05647 0.0482) (-3.74192e-08 1.05647 0.0241))

If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.

Rerun with processor debug flag set for more information.

[0] 

[0]     From function processorPolyPatch::calcGeometry()

[0]     in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.

[0] 

FOAM parallel run exiting

[0] 

[5] 

[5] 

[5] --> FOAM FATAL ERROR: 

[5] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.

patch:procBoundary5to3throughoutlet my area:0.00010351 neighbour area:0.000103534 matching tolerance:1.49815e-08

Mesh face:2335568 vertices:4((9 1.05217 2.2654) (9 1.05646 2.2654) (9 1.05646 2.2895) (9 1.05217 2.2895))

If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.

Rerun with processor debug flag set for more information.

[5] 

[5]     From function processorPolyPatch::calcGeometry()

[5]     in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.

[5] 

FOAM parallel run exiting

[5] 

--------------------------------------------------------------------------

MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD 

with errorcode 1.



NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.

You may or may not see output from other processes, depending on

exactly when Open MPI kills them.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun has exited due to process rank 3 with PID 4498 on

node bo332-chen04.mecheng.osu.edu exiting improperly. There are two reasons this could occur:



1. this process did not call "init" before exiting, but others in

the job did. This can cause a job to hang indefinitely while it waits

for all processes to call "init". By rule, if one process calls "init",

then ALL processes must call "init" prior to termination.



2. this process called "init", but exited without calling "finalize".

By rule, all processes that call "init" MUST call "finalize" prior to

exiting or it will be considered an "abnormal termination"



This may have caused other processes in the application to be

terminated by signals sent by mpirun (as reported here).

--------------------------------------------------------------------------

I have played around with the tolerances in the boundary file, and no matter how much I relax it, I get the same error.

Following is the link for my "boundary" file.

http://dl.dropbox.com/u/37012034/boundary

Additional notes:
This is the geometry of the case (in 2D view), but I'm running a 3D case extruded uniformly, such that "front" and "back" walls are of same size, and "inlet" and "outlet" surfaces are of same size.

http://dl.dropbox.com/u/37012034/hill_streamlines.jpg

here is the checkMesh output:
http://dl.dropbox.com/u/37012034/checkmesh_output

I would very much appreciate if someone could help with this. The case runs good in serial, but it is terribly slow. I REALLY need to get it running in parallel to move ahead, since it has a very high run time.:confused:
I guess the problem is with the cyclic b.c . I have run parallel cases in OpenFOAM in the past w/o any issues, when they weren't using cyclic b.c.

Thank you in advance :)

Greetings atmcfd,

I'll recycle a recent post of mine:

Quote:

Originally Posted by wyldckat (Post 391337)

Edit the file "system/decomposeParDict".
Add this line:

Code:

preservePatches (fan_half0 fan_half1);
Now decompose the case and run the solver!

That line in the dictionary does what it says: preserves the patches in a single processor!

As for your previous thread, you could have updated your question there with a new post! ;)

Best regards,
Bruno

PS: I've moved that other post into this thread instead.

Dear Bruno,

Thank you very much for your feedback. My case now works in parallel! I never knew such a "preservePatches" command existed. Is there some sort of command sheet etc. for OF you know of? So that I can try them myself before asking help?

A couple of things:

1) The case would run on 12 processors, but would start diverging after about 2500 iters ( which takes about 13 hours :eek:)
I use hierarchichal (4 1 3)

2) The same case gives a floating point error and aborts in 100 iters, when I run it with 24 processors !!!
I tried hierarchichal (4 2 3) and (4 1 6) both.

more interestingly , my checkMesh (which I already posted earlier) gives

Code:

Checking topology...

 ****Problem with boundary patch 0 named inlet of type cyclic. The patch should start on face no 13913752 and the patch specifies 13950208.

Possibly consecutive patches have this same problem. Suppressing future warnings.

 ***Boundary definition is in error.

The boundary file is in the previous post. - I am trying hard to find out how this is happening. I created the boundary patches and faces using the setSet and faceSet utility in OF, and cross checked if the faces were in the right position using paraview. Is there something I am missing to see here?I am looking for a way to see those "suppressed warnings"

I would appreciate any suggestions/pointers.

Thank you...

I found out about "preservePatches" here on the forum, probably when I was searching for stuff about "cyclic BCs" and "parallel" in OpenFOAM.

The "suppressed warnings" message indicates that there would be a metric ton of identical error messages, varying only in the two numbers you can see you that 1st message. You can try the full check:

Code:

checkMesh -allTopology -allGeometry

As for the error it's giving you, you haven't given enough information to deduce where the wrong or missing step is :(

Dear Bruno,

Thanks for the suggestion.

Here is the additional info:

Image of the Mesh, with patch names:

http://dl.dropbox.com/u/37012034/mesh.png

The "0" folder:

http://dl.dropbox.com/u/37012034/0.zip

The "system" folder:

http://dl.dropbox.com/u/37012034/system.zip

the boundary file:

http://dl.dropbox.com/u/37012034/boundary

the complete checkMesh output:

http://dl.dropbox.com/u/37012034/checkmesh_output

I am trying to figure out why "The patch should start on face no 13913752...." is showing up. I built the mesh from extruding the 2D mesh directly into 3D, and I don't see why the patch should start from a different face, since I defined all of them myself :confused: . this boundary error must be the reason the simulation is diverging too.

Also, here is the link for the complete 3D mesh for the standard periodic hill test case ,133mb, if people ever need it in future. Would save them a lot of time building a new one. :p

http://dl.dropbox.com/u/37012034/constant.zip

I'm pretty sure about the consistency of the 2D mesh, since its from a official test case. I dont foresee any problems in the 3D version either, since its just plain extrusion in z direction. atleast thats what the checkMesh appears to say so :rolleyes:.

The B.C is cyclic in streamwise and spanwise direction, and Im 100% sure its physically correct....

Thanks again Bruno!!! :)

P.S.: I really wish OF had some sort of "cheat sheet" of all the commands it has. would do a world of good to newbies.:(

Hi Atm,

There might be a cheat sheet available, but I haven't found it yet. There are several sources of information about OpenFOAM, so it's a matter of looking for it...
Either way, usually "cheat sheets" are made by people in any community, not usually distributed by the software developers.

As for the case, I forgot to mention that it's the procedure itself that I would like to know about. And a simpler test case would make it a lot easier to diagnose where the problem is :(

Best regards,
Bruno

Quote:

Originally Posted by wyldckat (Post 391783)

Bruno,

I will just explain the procedure I have followed.

1) I have a mesh in plot3d format, which I converted to FOAM using plot3dtoFoam utility.

2) then I used the interactive "setSet" utility to set the boundaries inlet,outlet,top wall, bottom wall, front and back, by using a bounding box to specify the faces within each patch.
eg. the command I used for "inlet" was like this

faceSet inlet add boxToFace (-1 1 0.0241) (0 3.035 4.5067)

3) Once I did step 2 for all the boundaries , I do createPatch and a "polyMesh" folder is created, having the boundary file, and the remaning files in the polyMesh directory, for which I posted a link in my earlier post.

4) Then , I apply "cyclic" b.c. for inlet - outlet and the front-back pairs.

Code:

inlet

{

type cyclic;

nFaces 23808;

startFace 13950208;

matchTolerance 0.01;

neighbourPatch outlet;

}

outlet

{

type cyclic;

nFaces 23808;

startFace 13974016;

matchTolerance 0.01;

neighbourPatch inlet;

}



 front

 {

        type            cyclic;

        nFaces          25088;

        startFace       14034280;

        matchTolerance  0.01;

        neighbourPatch  back;



 }

    back

 {

        type            cyclic;

        nFaces          25088;

        startFace       14059368;

        matchTolerance  0.01;

        neighbourPatch  front;

}

5) From here on, I am not sure what to do. I checkMesh , and I get the error as stated previously. I think I should use createPatch, and I have tried it multiple times , as given here

Code:

oamFile

{

    version     2.0;

    format      ascii;

    class       dictionary;

    object      createPatchDict;

}



// This application/dictionary controls:

// - optional: create new patches from boundary faces (either given as

//   a set of patches or as a faceSet)

// - always: order faces on coupled patches such that they are opposite. This

//   is done for all coupled faces, not just for any patches created.

// - optional: synchronise points on coupled patches.



// 1. Create cyclic:

// - specify where the faces should come from

// - specify the type of cyclic. If a rotational specify the rotationAxis

//   and centre to make matching easier

// - always create both halves in one invocation with correct 'neighbourPatch'

//   setting.

// - optionally pointSync true to guarantee points to line up. 



// 2. Correct incorrect cyclic:

// This will usually fail upon loading:

//  "face 0 area does not match neighbour 2 by 0.0100005%"

//  " -- possible face ordering problem."

// - in polyMesh/boundary file:

//      - loosen matchTolerance of all cyclics to get case to load

//      - or change patch type from 'cyclic' to 'patch'

//        and regenerate cyclic as above



// Do a synchronisation of coupled points after creation of any patches.

// Note: this does not work with points that are on multiple coupled patches

//       with transformations (i.e. cyclics).

pointSync false;



// Patches to create.

patches

(

    {   

        // Name of new patch

        name inlet;



        // Dictionary to construct new patch from

        patchInfo

        {   

            type cyclic;

            neighbourPatch outlet;



            // Optional: explicitly set transformation tensor.

            // Used when matching and synchronising points.

            // transform rotational;

            // rotationAxis (1 0 0);

            // rotationCentre (0 0 0);

            // transform translational;

            // separationVector (1 0 0);



            // Optional non-default tolerance to be able to define cyclics

            // on bad meshes

              matchTolerance 1E-2;

        }



        // How to construct: either from 'patches' or 'set'

        constructFrom patches;



        // If constructFrom = patches : names of patches. Wildcards allowed.

        patches (inlet);



        // If constructFrom = set : name of faceSet

        // set f0;

    }

    {

        // Name of new patch

        name outlet;



        // Dictionary to construct new patch from

        patchInfo

        {

            type cyclic;

            neighbourPatch inlet;



            // Optional: explicitly set transformation tensor.

            // Used when matching and synchronising points.

            // transform rotational;

            // rotationAxis    ( 0 0 1 );

            // rotationCentre  ( 0.3 0 0 );

        }



        // How to construct: either from 'patches' or 'set'

        constructFrom patches;



        // If constructFrom = patches : names of patches. Wildcards allowed.

        patches (outlet);



        // If constructFrom = set : name of faceSet

        // set f0;

    }

);

And this doesnt work either. The problem is I dont know exactly how to use this createPatchDict to create cyclic patches from two existing patches. Or even if I need createPatchDict at all.

I looked up tutorials using cyclic patches like "boxTurb16" and "channel395" and they have not been very helpful, since they create faces in blockMeshDict itself (whereas I am importing a mesh).

Is my procedure correct?
Meanwhile I will also be trying some different approaches, using commercial mesh gen software, different,simpler cases etc. to understand whats happening here.

One thing i observe is that the checkMesh doesnt give the "boundary definition error" for the "front" and "back" planes, even though I have defined them using exactly the same way as the "inlet" and "outlet", and have not even used CreatePatchDict for them. This confuses me even more!! :confused:
Please let me know if you need any other info...I appreciate your time and help.:)

Thank you!

Hi Atm,

Attached is the tutorial "DNS/dnsFoam/boxTurb16" modified to use topoSet and createPatch.

Modified/added files:

Allrun
constant/polyMesh/blockMeshDict
system/topoSetDict
system/changePatchDict

Best regards,
Bruno

Bruno,

That was truly awesome of you to post that tutorial!!!!! Helped me understand the concept very clearly!

And also, I figured out the reason what the actual error in the checkMesh:

So as it turns out, it was a hilariously simple solution. This was my old boundary file

Code:

/*--------------------------------*- C++ -*----------------------------------*\

| =========                 |                                                 |

| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |

|  \\    /   O peration     | Version:  2.1.1                                 |

|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |

|    \\/     M anipulation  |                                                 |

\*---------------------------------------------------------------------------*/

FoamFile

{

    version     2.0;

    format      ascii;

    class       polyBoundaryMesh;

    location    "polyMesh";

    object      boundary;

}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //



6

(

    inlet

    {

        type            cyclic;

        nFaces          23808;

        startFace       13950208;

       matchTolerance  0.001;

       neighbourPatch  outlet; 

    }

    outlet

    {

        type            cyclic;

        nFaces          23808;

        startFace       13974016;

       matchTolerance  0.001;

       neighbourPatch  inlet; 

    }

    bottom

    {

        type            wall;

        nFaces          36456;

        startFace       13913752;

    }

    top

    {

        type            wall;

        nFaces          36456;

        startFace       13997824;

    }

    front

    {

        type            cyclic;

        nFaces          25088;

        startFace       14034280;

        matchTolerance  0.001;

        neighbourPatch  back;



    }

    back

    {

        type            cyclic;

        nFaces          25088;

        startFace       14059368;

        matchTolerance  0.001;

        neighbourPatch  front;



    }

)



// ************************************************************************* //

Now if I do this, the error disappears :-P

Code:

/*--------------------------------*- C++ -*----------------------------------*\

| =========                 |                                                 |

| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |

|  \\    /   O peration     | Version:  2.1.1                                 |

|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |

|    \\/     M anipulation  |                                                 |

\*---------------------------------------------------------------------------*/

FoamFile

{

    version     2.0;

    format      ascii;

    class       polyBoundaryMesh;

    location    "polyMesh";

    object      boundary;

}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //



6

(



    bottom

    {

        type            wall;

        nFaces          36456;

        startFace       13913752;

    }



    inlet

    {

        type            cyclic;

        nFaces          23808;

        startFace       13950208;

       matchTolerance  0.001;

       neighbourPatch  outlet; 

    }

    outlet

    {

        type            cyclic;

        nFaces          23808;

        startFace       13974016;

       matchTolerance  0.001;

       neighbourPatch  inlet; 

    }



    top

    {

        type            wall;

        nFaces          36456;

        startFace       13997824;

    }

    front

    {

        type            cyclic;

        nFaces          25088;

        startFace       14034280;

        matchTolerance  0.001;

        neighbourPatch  back;



    }

    back

    {

        type            cyclic;

        nFaces          25088;

        startFace       14059368;

        matchTolerance  0.001;

        neighbourPatch  front;



    }

)



// ************************************************************************* //

So basically I just put the "bottom" patch definition in the beginning of the file. Looks like OF reads the startFace in an incremental manner,
startFace + nFaces = startFace of next patch, and so on. In my earlier case since I had mixed up the position of the "bottom" patch, this wasnt happening.
Might be a really basic thing, but easily overlooked.:p

Thank you again for your help :)

Quote:

Originally Posted by wyldckat (Post 391408)

Greetings atmcfd,

I'll recycle a recent post of mine:

As for your previous thread, you could have updated your question there with a new post! ;)

Best regards,
Bruno

PS: I've moved that other post into this thread instead.

THANKS BRUNO! Amazing! It really solved my problem with parallel run!

Quote:

Originally Posted by wyldckat (Post 391408)

Greetings atmcfd,

I'll recycle a recent post of mine:

Code:

preservePatches (fan_half0 fan_half1);

As for your previous thread, you could have updated your question there with a new post! ;)

Best regards,
Bruno

PS: I've moved that other post into this thread instead.

Hello Bruno,

can you please explain what I've to put into the brackets of preservePatches?

Thank you!

Quote:

Originally Posted by Harak (Post 556962)

can you please explain what I've to put into the brackets of preservePatches?

Quick answer:

It's parenthesis "()", not brackets "{}" ;)
You should place the name of the patches that are of type cyclic. In that example, the patches were named "fan_half0" and "fan_half1".

Quote:

Originally Posted by wyldckat (Post 556966)

Quick answer:

It's parenthesis "()", not brackets "{}" ;)
You should place the name of the patches that are of type cyclic. In that example, the patches were named "fan_half0" and "fan_half1".

Thanks for your really fast reply :)
Ok, now I understood it but in my case, I don't have any cyclic patch :confused:
It's about a 3D flow around a beam placed in a square duct.

In the meanwhile I've opened this thread where you can find further information:
http://www.cfd-online.com/Forums/ope...ial-works.html

I would really appreciate if you could have a look on it :D

Greetings
Harak

Quote:

Originally Posted by Harak (Post 556967)

Ok, now I understood it but in my case, I don't have any cyclic patch :confused:

Then it will unlikely work for your case :(

Quote:

Originally Posted by Harak (Post 556967)

In the meanwhile I've opened this thread where you can find further information:
http://www.cfd-online.com/Forums/ope...ial-works.html

I've added it to my to-do list, but I have no idea when I'll be able to look into it, because it's not a straight forward answer. In addition, it's best that you indicate which OpenFOAM version/variant you're using.

hi
I have this problem too

Checking topology...
****Problem with boundary patch 0 named inlet of type cyclic. The patch should start on face no 13913752 and the patch specifies 13950208.
Possibly consecutive patches have this same problem. Suppressing future warnings.
***Boundary definition is in error.

can anyone help me?

best regards

Quote:

Originally Posted by masoudsh (Post 591766)

can anyone help me?

Quick answer: Possibly anyone can help you, as long as you follow the steps detailed on this thread: http://www.cfd-online.com/Forums/ope...-get-help.html

hi Bruno

thank you for quick reply.
I solve it ,now.
with atmcfd comment.

can you help me with other question?

I have this problem in my run:

--> FOAM FATAL ERROR:
Maximum number of iterations exceeded

From function thermo<Thermo, Type>::T(scalar f, scalar T0, scalar (thermo<Thermo, Type>::*F)(const scalar) const, scalar (thermo<Thermo, Type>::*dFdT)(const scalar) const, scalar (thermo<Thermo, Type>::*limit)(const scalar) const) const
in file /home/opencfd/OpenFOAM/OpenFOAM-2.3.0/src/thermophysicalModels/specie/lnInclude/thermoI.H at line 76.

best regards

masoudsh, please follow the steps detailed on this thread: How to give enough info to get help