[Jingxue Wang]

Dear all

I met a problem for parallel calculation on remote server OpenFoam 4.1. I adopetd the code:* mpirun ${NQSII_MPIOPTS} -np ${Nod} pimpleFoam -parallel in my job script. I want to use 2 nodes to caculate. But it always stoped after it started to excute not so long. The error infoamtion is shown in the attached file. It seems it can not successully excute in* some processors. I checked all the files of my OpenFOAM calclations. I think they are fine. And I also I try to use bash to run my job script (bash nqs.sh)and it can be successfully excute.(Many be it is excuted on the front end node and* not permitted) But anyway, parallel calculation is ok for using bash.

Could somebody have expericence for this?It stuck me for one week.... Many thanks for this!

[Taataa]

Can you attach all the log files? The decomposition, blockMesh and snappy. It seems you have a syntax error.

[Jingxue Wang]

Thanks，Taataa ！

The codes are as followings.

[Jingxue Wang]

[QUOTE=Jingxue Wang;678003]Thanks，Taataa ！

The codes are as followings

FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

application     pimpleFoam;

startFrom       latestTime;

startTime       0;

stopAt          endTime;

endTime         30;

deltaT          1;

writeControl    timeStep;

writeInterval   1;

purgeWrite      0;

//writeFormat     ascii;

//writePrecision  6;

writeFormat     binary;

writePrecision  8;

writeCompression off;

timeFormat      general;

timePrecision   6;

runTimeModifiable true;

//adjustTimeStep yes;

//maxCo 0.9;

FoamFile
{
    version         2.0;
    format          ascii;

    root            "";
    case            "";
    instance        "";
    local           "";

    class           dictionary;
    object          blockMeshDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //


convertToMeters 1;

x0  -13.8;
x1  -11.8; 
x2  9.96;  //x_total=23.76m

y0 -1.536;
y1  1.536;//y_total=3.072m

z0  0.00;
z1  0.3841; //z_total=2m
z2  2; //z_total=2m


vertices
(
    ( $x0 $y0 $z0 )
    ( $x1 $y0 $z0 )
    ( $x1 $y1 $z0 )
    ( $x0 $y1 $z0 )

    ( $x0 $y0 $z1 )
    ( $x1 $y0 $z1 )
    ( $x1 $y1 $z1 )
    ( $x0 $y1 $z1 )


    ( $x0 $y0 $z2 )
    ( $x1 $y0 $z2 )
    ( $x1 $y1 $z2 )
    ( $x0 $y1 $z2 )

    ( $x2 $y0 $z0 )
    ( $x2 $y1 $z0 )
    ( $x2 $y0 $z1 )
    ( $x2 $y1 $z1 )  
    ( $x2 $y0 $z2 )
    ( $x2 $y1 $z2 )  
);

blocks
(
    hex (0 1 2 3 4 5 6 7)       (10 24 9)  simpleGrading (0.4 1 2.64)    //x:2m,y:3.072m,z:0.3841m
    hex (4 5 6 7 8 9 10 11)       (10 24 16)  simpleGrading (0.4 1 2.2)    //x:2m,y:3.072m,z:0.035m   dao0.07
    hex (1 12 13 2 5 14 15 6)     (170 24 9)  simpleGrading (1 1 2.64)   //x:21.76m(delta x=0.128m),y:3.072m,z:1.6m
    hex (5 14 15 6 9 16 17 10)     (170 24 16)  simpleGrading (1 1 2.2)   //x:21.76m(delta x=0.128m),y:3.072m,z:1.6m
);

edges
(
);

boundary
(
  inlet
    {
        type patch;
        faces
        (
            (0 4 7 3)
            (4 8 11 7)
        );
    }
    outlet
    {
        type patch;
        faces
        (
            (12 13 15 14)
            (14 15 17 16)
        );
    }
    ground
    {
        type wall;
        faces
        (
            (0 3 2 1)
            (1 2 13 12)
        );
    }
    up
    {
        type wall;
        faces
        (
            (8 9 10 11)
            (9 16 17 10)
        );
    }
   left
    {
        type wall;
        faces
        (
            (3 7 6 2)
            (7 11 10 6)
            (2 6 15 13)
            (6 10 17 15)
        );
    }
    right
    {
        type wall;
        faces
        (
            (1 5 4 0)
            (5 9 8 4)
            (12 14 5 1)
            (14 16 9 5)
        );
    }
);
// ************************************************************************* //

FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    note        "mesh decomposition control dictionary";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains  48;

//- Keep owner and neighbour on same processor for faces in zones:
// preserveFaceZones (heater solid1 solid3);

//- Keep owner and neighbour on same processor for faces in patches:
//  (makes sense only for cyclic patches)
//preservePatches (cyclic_half0 cyclic_half1);

//- Keep all of faceSet on a single processor. This puts all cells
//  connected with a point, edge or face on the same processor.
//  (just having face connected cells might not guarantee a balanced
//  decomposition)
// The processor can be -1 (the decompositionMethod chooses the processor
// for a good load balance) or explicitly provided (upsets balance).
//singleProcessorFaceSets ((f0 -1));


//- Use the volScalarField named here as a weight for each cell in the
//  decomposition.  For example, use a particle population field to decompose
//  for a balanced number of particles in a lagrangian simulation.
// weightField dsmcRhoNMean;

// method          scotch;
 method          hierarchical;
//method           simple;
// method          metis;
// method          manual;
// method          multiLevel;
// method          structured;  // does 2D decomposition of structured mesh

multiLevelCoeffs
{
    // Decomposition methods to apply in turn. This is like hierarchical but
    // fully general - every method can be used at every level.

    level0
    {
        numberOfSubdomains  48;
        //method simple;
        //simpleCoeffs
        //{
        //    n           (2 1 1);
        //    delta       0.001;
        //}
        method scotch;
    }
    level1
    {
        numberOfSubdomains  48;
        method scotch;
    }
}

// Desired output

simpleCoeffs
{
    n           (12 4 1);
    delta       0.001;
}

hierarchicalCoeffs
{
    n           (12 4 1);
    delta       0.001;
    order       xyz;
}

metisCoeffs
{
 /*
    processorWeights
    (
        1
        1
        1
        1
    );
  */
}

scotchCoeffs
{
    //processorWeights
    //(
    //    1
    //    1
    //    1
    //    1
    //);
    //writeGraph  true;
    //strategy "b";
}

manualCoeffs
{
    dataFile    "decompositionData";
}

structuredCoeffs
{
    // Patches to do 2D decomposition on. Structured mesh only; cells have
    // to be in 'columns' on top of patches.
    patches     (bottomPatch);
}

//// Is the case distributed? Note: command-line argument -roots takes
//// precedence
//distributed     yes;
//// Per slave (so nProcs-1 entries) the directory above the case.
//roots
//(
//    "/tmp"
//    "/tmp"
//);

// ************************************************************************* //

FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      snappyHexMeshDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

// Which of the steps to run
castellatedMesh true;
snap            true;
addLayers       true; //true;


// Geometry. Definition of all surfaces. All surfaces are of class
// searchableSurface.
// Surfaces are used
// - to specify refinement for any mesh cell intersecting it
// - to specify refinement for any mesh cell inside/outside/near
// - to 'snap' the mesh boundary to the surface
geometry
{
/*
    blk01.stl
    {
        type triSurfaceMesh;
        name blk01;
    }
    blk02.stl
    {
        type triSurfaceMesh;
        name blk02;
    }
    blk03.stl
    {
        type triSurfaceMesh;
        name blk03;
    }
    spire.stl
    {
        type triSurfaceMesh;
        name spire;
    }
*/
    building.stl
    {
        type triSurfaceMesh;
        name  building;
    }
/*
    panel1.stl
    { 
     type triSurfaceMesh;
     name  panel1;
     }
    panel2.stl
    { 
     type triSurfaceMesh;
     name  panel2;
    }
    panel3.stl
    { 
     type triSurfaceMesh;
     name  panel3;
     }
    panel4.stl
    { 
     type triSurfaceMesh;
     name  panel4;
     }
    panel5.stl
    { 
     type triSurfaceMesh;
     name  panel5;
     }
    panel6.stl
    { 
     type triSurfaceMesh;
     name  panel6;
     }
    panel7.stl
    { 
     type triSurfaceMesh;
     name  panel7;
     }
*/
//----------------------------
/*
    refBoxblk00
    {
        type searchableBox;
        min (-12.8 -1.5  0);
        max ( 5 1.5  1.2);
    }
    refBoxblk01
    {
        type searchableBox;
        min (-9 -1.5  0);
        max ( 3 1.5  0.8);
    }
    refBoxblk02
    {
        type searchableBox;
        min (-4 -1.5 0);
        max ( 2 1.5  0.6);
    }
    refBoxblk03
    {
        type searchableBox;
        min (-2.2 -1.5 0);
        max ( 1.2 1.5  0.3);
    }
    refBoxSpire01
    {
        type searchableBox;
        min (-11.5 -1.5  0);
        max (-10.1 1.5  1.95);
    }
    refBoxSpire02
    {
        type searchableBox;
        min (-11.2 -1.5  0);
        max (-10.4 1.5  1.9);
    }
*/
     refBoxtg00
     {
       type searchableBox;
       min (-0.8  -0.8  0.0);
       max (1 0.8 0.7);
     }
    refBoxtg01
     {
       type searchableBox;
       min (-0.5  -0.5  0.0);
       max (0.7 0.5 0.6);
     }
    refBoxtg02
     {
       type searchableBox;
       min (-0.35  -0.35  0.0);
       max (0.5 0.35 0.5);
     }
    refBoxtg03
     {
       type searchableBox;
       min (-0.28  -0.28  0.0);
       max (0.3 0.28 0.42);
     }
};

// Settings for the castellatedMesh generation.
castellatedMeshControls
{

    // Refinement parameters
    // ~~~~~~~~~~~~~~~~~~~~~

    // While refining maximum number of cells per processor. This is basically
    // the number of cells that fit on a processor. If you choose this too small
    // it will do just more refinement iterations to obtain a similar mesh.
    maxLocalCells 20000000;

    // Overall cell limit (approximately). Refinement will stop immediately
    // upon reaching this number so a refinement level might not complete.
    // Note that this is the number of cells before removing the part which
    // is not 'visible' from the keepPoint. The final number of cells might
    // actually be a lot less.
    maxGlobalCells 240000000;//240million

    // OpenFOAM-1.7.1
    // Allow a certain level of imbalance during refining
    // (since balancing is quite expensive)
    // Expressed as fraction of perfect balance (= overall number of cells /
    // nProcs). 0=balance always.
    maxLoadUnbalance 0.10;


    // The surface refinement loop might spend lots of iterations refining just a
    // few cells. This setting will cause refinement to stop if <= minimumRefine
    // are selected for refinement. Note: it will at least do one iteration
    // (unless the number of cells to refine is 0)
    minRefinementCells 10; //10;

    // Number of buffer layers between different levels.
    // 1 means normal 2:1 refinement restriction, larger means slower
    // refinement.
    nCellsBetweenLevels 3;


    // Explicit feature edg   // This is a featureEdgeMesh, read from constant/triSurface for now.
    features
    (
        {
            file "building.eMesh";
            level 6;
        }
/*
        {
            file "panel1.eMesh";
            level 7;
        }
        {
            file "panel2.eMesh";
            level 7;
        }e refinement
    // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    // Specifies a level for any cell intersected by its edges.
    // This is a featureEdgeMesh, read from constant/triSurface for now.
    features
    (
        {
            file "building.eMesh";
            level 7;
        }
/*
        {
            file "panel1.eMesh";
            level 7;
        }
        {
            file "panel2.eMesh";
            level 7;
        }
        {
            file "panel3.eMesh";
            level 7;
        }
        {
            file "panel4.eMesh";
            level 7;
        }
        {
            file "panel5.eMesh";
            level 7;
        }
        {
            file "panel6.eMesh";
            level 7;
        }
        {
            file "panel7.eMesh";
            level 7;
        }
*/
    );


    // Surface based refinement
    // ~~~~~~~~~~~~~~~~~~~~~~~~

    // Specifies two levels for every surface. The first is the minimum level,
    // every cell intersecting a surface gets refined up to the minimum level.
    // The second level is the maximum level. Cells that 'see' multiple
    // intersections where the intersections make an
    // angle > resolveFeatureAngle get refined up to the maximum level.
    refinementSurfaces
    {
/*
        blk01
        {
            // Surface-wise min and max refinement level
	  level (3 4); // (5 6)
        }
        blk02
        {
            // Surface-wise min and max refinement level
	  level (3 4); // (5 6)
        }
        blk03
        {
            // Surface-wise min and max refinement level
	  level (4 5); // (5 6)
        }
        spire
        {
            // Surface-wise min and max refinement level
	  level (3 4); // (5 6)
        }
*/
	building
	{
	  level (6 7);
	}	
/*
	panel1
	{
	  level (6 7);
	}
	panel2
	{
	  level (6 7);
	}
	panel3
	{
	  level (6 7);
	}
	panel4
	{
	  level (6 7);
	}
	panel5
	{
	  level (6 7);
	}
	panel6
	{
	  level (6 7);
	}
	panel7
	{
	  level (6 7);
	}	
*/
    }
    // Resolve sharp angles
    resolveFeatureAngle 30;


    // Region-wise refinement
    // ~~~~~~~~~~~~~~~~~~~~~~

    // Specifies refinement level for cells in relation to a surface. One of
    // three modes
    // - distance. 'levels' specifies per distance to the surface the
    //   wanted refinement level. The distances need to be specified in
    //   descending order.
    // - inside. 'levels' is only one entry and only the level is used. All
    //   cells inside the surface get refined up to the level. The surface
    //   needs to be closed for this to be possible.
    // - outside. Same but cells outside.

    refinementRegions
    {
/*
      blk01
	{
	  mode distance;
	  levels ((0.1 2) (0.2 1));
	} 
      blk02
	{
	  mode distance;
	  levels ((0.1 3) (0.2 2));
	}   
      blk03
	{
	  mode distance;
	  levels ((0.05 4) (0.1 3));
	}
      refBoxblk00
        {
	  mode inside;
	  levels ((0 0));
        } 
      refBoxblk01
        {
            mode inside;
	    levels ((0 1));
        }
        refBoxblk02
        {
            mode inside;
	    levels ((0 2));
        }
       refBoxblk03
        {
            mode inside;
	    levels ((0 3));
        }
	refBoxSpire01
	{
	  mode inside;
	  levels ((0 1));
	}
        refBoxSpire02
        {
          mode inside;
          levels ((0 2));
        }
*/
      refBoxtg00
	{
	  mode inside;
	  levels ((1E15 3));
	}
      refBoxtg01
	{
	  mode inside;
	  levels ((1E15 4));
	}
       refBoxtg02
	{
	  mode inside;
	  levels ((1E15 5));
	}
      refBoxtg03
	{
	  mode inside;
	  levels ((1E15 6));
	}
    }
    // Mesh selection
    // ~~~~~~~~~~~~~~

    // After refinement patches get added for all refinementSurfaces and
    // all cells intersecting the surfaces get put into these patches. The
    // section reachable from the locationInMesh is kept.
    // NOTE: This point should never be on a face, always inside a cell, even
    // after refinement.
    locationInMesh (0 0 1.0);


    allowFreeStandingZoneFaces true;
}

// Settings for the snapping.
snapControls
{
    //- Number of patch smoothing iterations before finding correspondence
    //  to surface
    nSmoothPatch 3;

    //- Relative distance for points to be attracted by surface feature point
    //  or edge. True distance is this factor times local
    //  maximum edge length.
    tolerance 4.0;

    //- Number of mesh displacement relaxation iterations.
    nSolveIter 30;

    //- Maximum number of snapping relaxation iterations. Should stop
    //  before upon reaching a correct mesh.
    nRelaxIter 5;
}

// Settings for the layer addition.
addLayersControls
{
    relativeSizes true;

    // Per final patch (so not geometry!) the layer information
    layers
    {
      building //patch name
	{
	  nSurfaceLayers 3; //3
        }
    }

    // Expansion factor for layer mesh
    expansionRatio 1.5;

    //- Wanted thickness of final added cell layer. If multiple layers
    //  is the
    //  thickness of the layer furthest away from the wall.
    //  Relative to undistorted size of cell outside layer.
    finalLayerThickness 0.5; //0.05;//0.45;

    //- Minimum thickness of cell layer. If for any reason layer
    //  cannot be above minThickness do not add layer.
    //  Relative to undistorted size of cell outside layer.
    minThickness 0.1; //0.01;//0.25;

    //- If points get not extruded do nGrow layers of connected faces that are
    //  also not grown. This helps convergence of the layer addition process
    //  close to features.
    nGrow 1;


    // Advanced settings

    //- When not to extrude surface. 0 is flat surface, 90 is when two faces
    //  make straight angle.
    featureAngle 30;

    //- Maximum number of snapping relaxation iterations. Should stop
    //  before upon reaching a correct mesh.
    nRelaxIter 5;

    // Number of smoothing iterations of surface normals
    nSmoothSurfaceNormals 1;

    // Number of smoothing iterations of interior mesh movement direction
    nSmoothNormals 3;

    // Smooth layer thickness over surface patches
    nSmoothThickness 10;

    // Stop layer growth on highly warped cells
    maxFaceThicknessRatio 0.5;

    // Reduce layer growth where ratio thickness to medial
    // distance is large
    maxThicknessToMedialRatio 0.3;

    // Angle used to pick up medial axis points
    minMedianAxisAngle 130;

    // Create buffer region for new layer terminations
    nBufferCellsNoExtrude 0;

    // Overall max number of layer addition iterations
    nLayerIter 50;
}



// Generic mesh quality settings. At any undoable phase these determine
// where to undo.
meshQualityControls
{
    //- Maximum non-orthogonality allowed. Set to 180 to disable.
    maxNonOrtho 65;

    //- Max skewness allowed. Set to <0 to disable.
    maxBoundarySkewness 20;
    maxInternalSkewness 4;

    //- Max concaveness allowed. Is angle (in degrees) below which concavity
    //  is allowed. 0 is straight face, <0 would be convex face.
    //  Set to 180 to disable.
    maxConcave 80;

    //- Minimum projected area v.s. actual area. Set to -1 to disable.
    minFlatness 0.5;

    //- Minimum pyramid volume. Is absolute volume of cell pyramid.
    //  Set to very negative number (e.g. -1E30) to disable.
    minVol 1e-13;
    minTetQuality 1e-30;

    //- Minimum face area. Set to <0 to disable.
    minArea -1;

    //- Minimum face twist. Set to <-1 to disable. dot product of face normal
    //- and face centre triangles normal
    minTwist 0.02;

    //- minimum normalised cell determinant
    //- 1 = hex, <= 0 = folded or flattened illegal cell
    minDeterminant 0.001;

    //- minFaceWeight (0 -> 0.5)
    minFaceWeight 0.02;

    //- minVolRatio (0 -> 1)
    minVolRatio 0.01;

    //must be >0 for Fluent compatibility
    minTriangleTwist -1;


    // Advanced

    //- Number of error distribution iterations
    nSmoothScale 4;
    //- amount to scale back displacement at error points
    errorReduction 0.75;
}


// Advanced

// Flags for optional output
// 0 : only write final meshes
// 1 : write intermediate meshes
// 2 : write volScalarField with cellLevel for postprocessing
// 4 : write current intersections as .obj files
debug 0;


// Merge tolerance. Is fraction of overall bounding box of initial mesh.
// Note: the write tolerance needs to be higher than this.
mergeTolerance 1E-6;


// ************************************************************************* //

[Jingxue Wang]

Thanks,Taataa.

I attched the codes in the previous reply. Thanks!

[Taataa]

You didn't attach the log files, first we need to see ithe errors. You can zip the logs and attach them.

[Jingxue Wang]

I zipped the logs files and my setting parameters files. I used 2 nodes （48cpus）in the remote server. I tried this again and obtained the following log files. The error information seems different with I attached. But I don't know the reason.

Many thanks!

[Jingxue Wang]

I zipped the logs files and my setting parameters files. I used 2 nodes （48cpus）in the remote server. I tried this again and obtained the following log files. The error information seems different with I attached. But I don't know the reason.

Thanks!I really appreciate your help!

[Taataa]

Based on the error in the log files, I have three suggestions. Use scotch for decomposition, try using less cpus for now, say 8, and decrease cell size of the background mesh generated by blockMesh based on the desired maximum size of the final mesh.

Also, make sure that you have enough memory available, roughly 1G for 1 million cells.
Just a suggestion, try to clean up the dictionaries. They have lots of comments and redundant lines I usually keep the notes in a separate file so I can keep the dictionaries clean.

[Jingxue Wang]

Thanks,Taattaa. I will try again

[Jingxue Wang]

I am wondering that scocth decompose is better than others?

[Taataa]

Yes, it is. It usually does a very good job. Among the existing ones, scotch works best without having to tweak anything. I forgot to mention that for snappyHexMesh you should set it as ptscotch and for the main simulation, scotch.