CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Running, Solving & CFD (http://www.cfd-online.com/Forums/openfoam-solving/)
-   -   Case running in serial, but Parallel run gives error - Please help! urgent ! (http://www.cfd-online.com/Forums/openfoam-solving/109128-case-running-serial-but-parallel-run-gives-error-please-help-urgent.html)

atmcfd November 7, 2012 07:22

Case running in serial, but Parallel run gives error - Please help! urgent !
 
Dear all,

Im using OpenFOAM 2.1 and am implementing the cyclic boundary condition for my case in both the streamwise and spanwise directions (its a kind of LES simulation of channel flow).

I use the following in my boundary file(and likewise for spanwise patches):
inlet
{
type cyclic;
nFaces 23808;
startFace 13950208;
matchTolerance 0.01;
neighbourPatch outlet;
}
outlet
{
type cyclic;
nFaces 23808;
startFace 13974016;
matchTolerance 0.01;
neighbourPatch inlet;
}


This works fine when I run the code in serial. However, in parallel, I get this error:


--> FOAM FATAL ERROR:
[6] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.
patch:procBoundary6to11throughinlet my area:0.000103534 neighbour area:0.00010351 matching tolerance:1.49818e-08
Mesh face:1174459 vertices:4((-3.74192e-08 1.05217 2.2654) (-3.74192e-08 1.05217 2.2895) (-3.74192e-08 1.05647 2.2895) (-3.74192e-08 1.05647 2.2654))
If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.
Rerun with processor debug flag set for more information.
[6]
[6] From function processorPolyPatch::calcGeometry()
[6] in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.
[6]
FOAM parallel run exiting


No matter how much I relax the tolerance I get the same error. I think this is due to the processor boundary-neighbour mismatch. Has anyone encountered this before and solved this???

Also, does the "processor" boundary condition help in any way? I couldn't find much documentation for it.

Thank you!!

atmcfd November 9, 2012 18:55

Case running in serial, but Parallel run gives error - Please help! urgent !
 
Hello all,

I had posted a similar query some days back but couldn't get any replies. I am being more detail now... I hope I get some feedback.
I am running an LES case using channelFoam. the boundary conditions are CYCLIC in streamwise and spanwise directions.

I can run my case in serial, but when I try to run it in parallel, I get the following error.

Code:


[3]
[3]
[3] --> FOAM FATAL ERROR:
[3] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.
patch:procBoundary3to5throughinlet my area:0.000103534 neighbour area:0.00010351 matching tolerance:1.49818e-08
Mesh face:2347842 vertices:4((-3.74192e-08 1.05217 2.2654) (-3.74192e-08 1.05217 2.2895) (-3.74192e-08 1.05647 2.2895) (-3.74192e-08 1.05647 2.2654))
If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.
Rerun with processor debug flag set for more information.
[3]
[3]    From function processorPolyPatch::calcGeometry()
[3]    in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.
[3]
FOAM parallel run exiting
[3]
[2]
[2]
[2] --> FOAM FATAL ERROR:
[2] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.
patch:procBoundary2to0throughoutlet my area:0.00010351 neighbour area:0.000103534 matching tolerance:1.49814e-08
Mesh face:2318738 vertices:4((9 1.05217 0.0241) (9 1.05646 0.0241) (9 1.05646 0.0482) (9 1.05217 0.0482))
If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.
Rerun with processor debug flag set for more information.
[2]
[2]    From function processorPolyPatch::calcGeometry()
[2]    in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.
[2]
FOAM parallel run exiting
[2]
[0]
[0]
[0] --> FOAM FATAL ERROR:
[0] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.
patch:procBoundary0to2throughinlet my area:0.000103534 neighbour area:0.00010351 matching tolerance:1.49816e-08
Mesh face:2331016 vertices:4((-3.74192e-08 1.05217 0.0241) (-3.74192e-08 1.05217 0.0482) (-3.74192e-08 1.05647 0.0482) (-3.74192e-08 1.05647 0.0241))
If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.
Rerun with processor debug flag set for more information.
[0]
[0]    From function processorPolyPatch::calcGeometry()
[0]    in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.
[0]
FOAM parallel run exiting
[0]
[5]
[5]
[5] --> FOAM FATAL ERROR:
[5] face 14 area does not match neighbour by 0.0232802% -- possible face ordering problem.
patch:procBoundary5to3throughoutlet my area:0.00010351 neighbour area:0.000103534 matching tolerance:1.49815e-08
Mesh face:2335568 vertices:4((9 1.05217 2.2654) (9 1.05646 2.2654) (9 1.05646 2.2895) (9 1.05217 2.2895))
If you are certain your matching is correct you can increase the 'matchTolerance' setting in the patch dictionary in the boundary file.
Rerun with processor debug flag set for more information.
[5]
[5]    From function processorPolyPatch::calcGeometry()
[5]    in file meshes/polyMesh/polyPatches/constraint/processor/processorPolyPatch.C at line 239.
[5]
FOAM parallel run exiting
[5]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 3 with PID 4498 on
node bo332-chen04.mecheng.osu.edu exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

I have played around with the tolerances in the boundary file, and no matter how much I relax it, I get the same error.

Following is the link for my "boundary" file.

http://dl.dropbox.com/u/37012034/boundary

Additional notes:
This is the geometry of the case (in 2D view), but I'm running a 3D case extruded uniformly, such that "front" and "back" walls are of same size, and "inlet" and "outlet" surfaces are of same size.

http://dl.dropbox.com/u/37012034/hill_streamlines.jpg

here is the checkMesh output:
http://dl.dropbox.com/u/37012034/checkmesh_output

I would very much appreciate if someone could help with this. The case runs good in serial, but it is terribly slow. I REALLY need to get it running in parallel to move ahead, since it has a very high run time.:confused:
I guess the problem is with the cyclic b.c . I have run parallel cases in OpenFOAM in the past w/o any issues, when they weren't using cyclic b.c.

Thank you in advance :)

wyldckat November 10, 2012 14:18

Greetings atmcfd,

I'll recycle a recent post of mine:
Quote:

Originally Posted by wyldckat (Post 391337)
  1. Edit the file "system/decomposeParDict".
  2. Add this line:
    Code:

    preservePatches (fan_half0 fan_half1);
  3. Now decompose the case and run the solver!
That line in the dictionary does what it says: preserves the patches in a single processor!

As for your previous thread, you could have updated your question there with a new post! ;)

Best regards,
Bruno

PS: I've moved that other post into this thread instead.

atmcfd November 11, 2012 17:44

Dear Bruno,

Thank you very much for your feedback. My case now works in parallel! I never knew such a "preservePatches" command existed. Is there some sort of command sheet etc. for OF you know of? So that I can try them myself before asking help?

A couple of things:

1) The case would run on 12 processors, but would start diverging after about 2500 iters ( which takes about 13 hours :eek:)
I use hierarchichal (4 1 3)

2) The same case gives a floating point error and aborts in 100 iters, when I run it with 24 processors !!!
I tried hierarchichal (4 2 3) and (4 1 6) both.

more interestingly , my checkMesh (which I already posted earlier) gives

Code:

Checking topology...
 ****Problem with boundary patch 0 named inlet of type cyclic. The patch should start on face no 13913752 and the patch specifies 13950208.
Possibly consecutive patches have this same problem. Suppressing future warnings.
 ***Boundary definition is in error.

The boundary file is in the previous post. - I am trying hard to find out how this is happening. I created the boundary patches and faces using the setSet and faceSet utility in OF, and cross checked if the faces were in the right position using paraview. Is there something I am missing to see here?I am looking for a way to see those "suppressed warnings"

I would appreciate any suggestions/pointers.

Thank you...

wyldckat November 12, 2012 18:17

I found out about "preservePatches" here on the forum, probably when I was searching for stuff about "cyclic BCs" and "parallel" in OpenFOAM.

The "suppressed warnings" message indicates that there would be a metric ton of identical error messages, varying only in the two numbers you can see you that 1st message. You can try the full check:
Code:

checkMesh -allTopology -allGeometry
As for the error it's giving you, you haven't given enough information to deduce where the wrong or missing step is :(

atmcfd November 13, 2012 04:04

Dear Bruno,

Thanks for the suggestion.

Here is the additional info:

Image of the Mesh, with patch names:

http://dl.dropbox.com/u/37012034/mesh.png

The "0" folder:

http://dl.dropbox.com/u/37012034/0.zip

The "system" folder:

http://dl.dropbox.com/u/37012034/system.zip

the boundary file:

http://dl.dropbox.com/u/37012034/boundary

the complete checkMesh output:

http://dl.dropbox.com/u/37012034/checkmesh_output

I am trying to figure out why "The patch should start on face no 13913752...." is showing up. I built the mesh from extruding the 2D mesh directly into 3D, and I don't see why the patch should start from a different face, since I defined all of them myself :confused: . this boundary error must be the reason the simulation is diverging too.

Also, here is the link for the complete 3D mesh for the standard periodic hill test case ,133mb, if people ever need it in future. Would save them a lot of time building a new one. :p

http://dl.dropbox.com/u/37012034/constant.zip

I'm pretty sure about the consistency of the 2D mesh, since its from a official test case. I dont foresee any problems in the 3D version either, since its just plain extrusion in z direction. atleast thats what the checkMesh appears to say so :rolleyes:.

The B.C is cyclic in streamwise and spanwise direction, and Im 100% sure its physically correct....

Thanks again Bruno!!! :)

P.S.: I really wish OF had some sort of "cheat sheet" of all the commands it has. would do a world of good to newbies.:(

wyldckat November 13, 2012 05:02

Hi Atm,

There might be a cheat sheet available, but I haven't found it yet. There are several sources of information about OpenFOAM, so it's a matter of looking for it...
Either way, usually "cheat sheets" are made by people in any community, not usually distributed by the software developers.

As for the case, I forgot to mention that it's the procedure itself that I would like to know about. And a simpler test case would make it a lot easier to diagnose where the problem is :(

Best regards,
Bruno

atmcfd November 16, 2012 20:41

Quote:

Originally Posted by wyldckat (Post 391783)
Hi Atm,

There might be a cheat sheet available, but I haven't found it yet. There are several sources of information about OpenFOAM, so it's a matter of looking for it...
Either way, usually "cheat sheets" are made by people in any community, not usually distributed by the software developers.

As for the case, I forgot to mention that it's the procedure itself that I would like to know about. And a simpler test case would make it a lot easier to diagnose where the problem is :(

Best regards,
Bruno

Bruno,

I will just explain the procedure I have followed.

1) I have a mesh in plot3d format, which I converted to FOAM using plot3dtoFoam utility.

2) then I used the interactive "setSet" utility to set the boundaries inlet,outlet,top wall, bottom wall, front and back, by using a bounding box to specify the faces within each patch.
eg. the command I used for "inlet" was like this

faceSet inlet add boxToFace (-1 1 0.0241) (0 3.035 4.5067)

3) Once I did step 2 for all the boundaries , I do createPatch and a "polyMesh" folder is created, having the boundary file, and the remaning files in the polyMesh directory, for which I posted a link in my earlier post.

4) Then , I apply "cyclic" b.c. for inlet - outlet and the front-back pairs.
Code:

inlet
{
type cyclic;
nFaces 23808;
startFace 13950208;
matchTolerance 0.01;
neighbourPatch outlet;
}
outlet
{
type cyclic;
nFaces 23808;
startFace 13974016;
matchTolerance 0.01;
neighbourPatch inlet;
}

 front
 {
        type            cyclic;
        nFaces          25088;
        startFace      14034280;
        matchTolerance  0.01;
        neighbourPatch  back;

 }
    back
 {
        type            cyclic;
        nFaces          25088;
        startFace      14059368;
        matchTolerance  0.01;
        neighbourPatch  front;
}

5) From here on, I am not sure what to do. I checkMesh , and I get the error as stated previously. I think I should use createPatch, and I have tried it multiple times , as given here

Code:

oamFile
{
    version    2.0;
    format      ascii;
    class      dictionary;
    object      createPatchDict;
}

// This application/dictionary controls:
// - optional: create new patches from boundary faces (either given as
//  a set of patches or as a faceSet)
// - always: order faces on coupled patches such that they are opposite. This
//  is done for all coupled faces, not just for any patches created.
// - optional: synchronise points on coupled patches.

// 1. Create cyclic:
// - specify where the faces should come from
// - specify the type of cyclic. If a rotational specify the rotationAxis
//  and centre to make matching easier
// - always create both halves in one invocation with correct 'neighbourPatch'
//  setting.
// - optionally pointSync true to guarantee points to line up.

// 2. Correct incorrect cyclic:
// This will usually fail upon loading:
//  "face 0 area does not match neighbour 2 by 0.0100005%"
//  " -- possible face ordering problem."
// - in polyMesh/boundary file:
//      - loosen matchTolerance of all cyclics to get case to load
//      - or change patch type from 'cyclic' to 'patch'
//        and regenerate cyclic as above

// Do a synchronisation of coupled points after creation of any patches.
// Note: this does not work with points that are on multiple coupled patches
//      with transformations (i.e. cyclics).
pointSync false;

// Patches to create.
patches
(
    { 
        // Name of new patch
        name inlet;

        // Dictionary to construct new patch from
        patchInfo
        { 
            type cyclic;
            neighbourPatch outlet;

            // Optional: explicitly set transformation tensor.
            // Used when matching and synchronising points.
            // transform rotational;
            // rotationAxis (1 0 0);
            // rotationCentre (0 0 0);
            // transform translational;
            // separationVector (1 0 0);

            // Optional non-default tolerance to be able to define cyclics
            // on bad meshes
              matchTolerance 1E-2;
        }

        // How to construct: either from 'patches' or 'set'
        constructFrom patches;

        // If constructFrom = patches : names of patches. Wildcards allowed.
        patches (inlet);

        // If constructFrom = set : name of faceSet
        // set f0;
    }
    {
        // Name of new patch
        name outlet;

        // Dictionary to construct new patch from
        patchInfo
        {
            type cyclic;
            neighbourPatch inlet;

            // Optional: explicitly set transformation tensor.
            // Used when matching and synchronising points.
            // transform rotational;
            // rotationAxis    ( 0 0 1 );
            // rotationCentre  ( 0.3 0 0 );
        }

        // How to construct: either from 'patches' or 'set'
        constructFrom patches;

        // If constructFrom = patches : names of patches. Wildcards allowed.
        patches (outlet);

        // If constructFrom = set : name of faceSet
        // set f0;
    }
);

And this doesnt work either. The problem is I dont know exactly how to use this createPatchDict to create cyclic patches from two existing patches. Or even if I need createPatchDict at all.

I looked up tutorials using cyclic patches like "boxTurb16" and "channel395" and they have not been very helpful, since they create faces in blockMeshDict itself (whereas I am importing a mesh).

Is my procedure correct?
Meanwhile I will also be trying some different approaches, using commercial mesh gen software, different,simpler cases etc. to understand whats happening here.

One thing i observe is that the checkMesh doesnt give the "boundary definition error" for the "front" and "back" planes, even though I have defined them using exactly the same way as the "inlet" and "outlet", and have not even used CreatePatchDict for them. This confuses me even more!! :confused:
Please let me know if you need any other info...I appreciate your time and help.:)


Thank you!

wyldckat November 17, 2012 17:26

1 Attachment(s)
Hi Atm,

Attached is the tutorial "DNS/dnsFoam/boxTurb16" modified to use topoSet and createPatch.

Modified/added files:
  • Allrun
  • constant/polyMesh/blockMeshDict
  • system/topoSetDict
  • system/changePatchDict
Best regards,
Bruno

atmcfd November 21, 2012 04:39

Bruno,

That was truly awesome of you to post that tutorial!!!!! Helped me understand the concept very clearly!

And also, I figured out the reason what the actual error in the checkMesh:

So as it turns out, it was a hilariously simple solution. This was my old boundary file

Code:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                |                                                |
| \\      /  F ield        | OpenFOAM: The Open Source CFD Toolbox          |
|  \\    /  O peration    | Version:  2.1.1                                |
|  \\  /    A nd          | Web:      www.OpenFOAM.org                      |
|    \\/    M anipulation  |                                                |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version    2.0;
    format      ascii;
    class      polyBoundaryMesh;
    location    "polyMesh";
    object      boundary;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

6
(
    inlet
    {
        type            cyclic;
        nFaces          23808;
        startFace      13950208;
      matchTolerance  0.001;
      neighbourPatch  outlet;
    }
    outlet
    {
        type            cyclic;
        nFaces          23808;
        startFace      13974016;
      matchTolerance  0.001;
      neighbourPatch  inlet;
    }
    bottom
    {
        type            wall;
        nFaces          36456;
        startFace      13913752;
    }
    top
    {
        type            wall;
        nFaces          36456;
        startFace      13997824;
    }
    front
    {
        type            cyclic;
        nFaces          25088;
        startFace      14034280;
        matchTolerance  0.001;
        neighbourPatch  back;

    }
    back
    {
        type            cyclic;
        nFaces          25088;
        startFace      14059368;
        matchTolerance  0.001;
        neighbourPatch  front;

    }
)

// ************************************************************************* //

Now if I do this, the error disappears :-P
Code:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                |                                                |
| \\      /  F ield        | OpenFOAM: The Open Source CFD Toolbox          |
|  \\    /  O peration    | Version:  2.1.1                                |
|  \\  /    A nd          | Web:      www.OpenFOAM.org                      |
|    \\/    M anipulation  |                                                |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version    2.0;
    format      ascii;
    class      polyBoundaryMesh;
    location    "polyMesh";
    object      boundary;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

6
(

    bottom
    {
        type            wall;
        nFaces          36456;
        startFace      13913752;
    }

    inlet
    {
        type            cyclic;
        nFaces          23808;
        startFace      13950208;
      matchTolerance  0.001;
      neighbourPatch  outlet;
    }
    outlet
    {
        type            cyclic;
        nFaces          23808;
        startFace      13974016;
      matchTolerance  0.001;
      neighbourPatch  inlet;
    }

    top
    {
        type            wall;
        nFaces          36456;
        startFace      13997824;
    }
    front
    {
        type            cyclic;
        nFaces          25088;
        startFace      14034280;
        matchTolerance  0.001;
        neighbourPatch  back;

    }
    back
    {
        type            cyclic;
        nFaces          25088;
        startFace      14059368;
        matchTolerance  0.001;
        neighbourPatch  front;

    }
)

// ************************************************************************* //

So basically I just put the "bottom" patch definition in the beginning of the file. Looks like OF reads the startFace in an incremental manner,
startFace + nFaces = startFace of next patch, and so on. In my earlier case since I had mixed up the position of the "bottom" patch, this wasnt happening.
Might be a really basic thing, but easily overlooked.:p

Thank you again for your help :)

levka November 24, 2013 07:35

thanks
 
Quote:

Originally Posted by wyldckat (Post 391408)
Greetings atmcfd,

I'll recycle a recent post of mine:


As for your previous thread, you could have updated your question there with a new post! ;)

Best regards,
Bruno

PS: I've moved that other post into this thread instead.

THANKS BRUNO! Amazing! It really solved my problem with parallel run!


All times are GMT -4. The time now is 16:52.