|
[Sponsors] |
Parallel computing error using multiple nodes |
|
LinkBack | Thread Tools | Search this Thread | Display Modes |
January 14, 2018, 00:30 |
Parallel computing error using multiple nodes
|
#1 |
Member
Jingxue Wang
Join Date: Sep 2017
Posts: 58
Rep Power: 9 |
Dear all
I met a problem for parallel calculation on remote server OpenFoam 4.1. I adopetd the code:* mpirun ${NQSII_MPIOPTS} -np ${Nod} pimpleFoam -parallel in my job script. I want to use 2 nodes to caculate. But it always stoped after it started to excute not so long. The error infoamtion is shown in the attached file. It seems it can not successully excute in* some processors. I checked all the files of my OpenFOAM calclations. I think they are fine. And I also I try to use bash to run my job script (bash nqs.sh)and it can be successfully excute.(Many be it is excuted on the front end node and* not permitted) But anyway, parallel calculation is ok for using bash. Could somebody have expericence for this?It stuck me for one week.... Many thanks for this! |
|
January 14, 2018, 01:25 |
|
#2 |
Senior Member
Taher Chegini
Join Date: Nov 2014
Location: Houston, Texas
Posts: 125
Rep Power: 13 |
Can you attach all the log files? The decomposition, blockMesh and snappy. It seems you have a syntax error.
|
|
January 14, 2018, 01:32 |
|
#3 |
Member
Jingxue Wang
Join Date: Sep 2017
Posts: 58
Rep Power: 9 |
Thanks,Taataa !
The codes are as followings. |
|
January 14, 2018, 01:35 |
|
#4 |
Member
Jingxue Wang
Join Date: Sep 2017
Posts: 58
Rep Power: 9 |
[QUOTE=Jingxue Wang;678003]Thanks,Taataa !
The codes are as followings Code:
FoamFile { version 2.0; format ascii; class dictionary; location "system"; object controlDict; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // application pimpleFoam; startFrom latestTime; startTime 0; stopAt endTime; endTime 30; deltaT 1; writeControl timeStep; writeInterval 1; purgeWrite 0; //writeFormat ascii; //writePrecision 6; writeFormat binary; writePrecision 8; writeCompression off; timeFormat general; timePrecision 6; runTimeModifiable true; //adjustTimeStep yes; //maxCo 0.9; Code:
FoamFile { version 2.0; format ascii; root ""; case ""; instance ""; local ""; class dictionary; object blockMeshDict; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // convertToMeters 1; x0 -13.8; x1 -11.8; x2 9.96; //x_total=23.76m y0 -1.536; y1 1.536;//y_total=3.072m z0 0.00; z1 0.3841; //z_total=2m z2 2; //z_total=2m vertices ( ( $x0 $y0 $z0 ) ( $x1 $y0 $z0 ) ( $x1 $y1 $z0 ) ( $x0 $y1 $z0 ) ( $x0 $y0 $z1 ) ( $x1 $y0 $z1 ) ( $x1 $y1 $z1 ) ( $x0 $y1 $z1 ) ( $x0 $y0 $z2 ) ( $x1 $y0 $z2 ) ( $x1 $y1 $z2 ) ( $x0 $y1 $z2 ) ( $x2 $y0 $z0 ) ( $x2 $y1 $z0 ) ( $x2 $y0 $z1 ) ( $x2 $y1 $z1 ) ( $x2 $y0 $z2 ) ( $x2 $y1 $z2 ) ); blocks ( hex (0 1 2 3 4 5 6 7) (10 24 9) simpleGrading (0.4 1 2.64) //x:2m,y:3.072m,z:0.3841m hex (4 5 6 7 8 9 10 11) (10 24 16) simpleGrading (0.4 1 2.2) //x:2m,y:3.072m,z:0.035m dao0.07 hex (1 12 13 2 5 14 15 6) (170 24 9) simpleGrading (1 1 2.64) //x:21.76m(delta x=0.128m),y:3.072m,z:1.6m hex (5 14 15 6 9 16 17 10) (170 24 16) simpleGrading (1 1 2.2) //x:21.76m(delta x=0.128m),y:3.072m,z:1.6m ); edges ( ); boundary ( inlet { type patch; faces ( (0 4 7 3) (4 8 11 7) ); } outlet { type patch; faces ( (12 13 15 14) (14 15 17 16) ); } ground { type wall; faces ( (0 3 2 1) (1 2 13 12) ); } up { type wall; faces ( (8 9 10 11) (9 16 17 10) ); } left { type wall; faces ( (3 7 6 2) (7 11 10 6) (2 6 15 13) (6 10 17 15) ); } right { type wall; faces ( (1 5 4 0) (5 9 8 4) (12 14 5 1) (14 16 9 5) ); } ); // ************************************************************************* // Code:
FoamFile { version 2.0; format ascii; class dictionary; note "mesh decomposition control dictionary"; object decomposeParDict; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // numberOfSubdomains 48; //- Keep owner and neighbour on same processor for faces in zones: // preserveFaceZones (heater solid1 solid3); //- Keep owner and neighbour on same processor for faces in patches: // (makes sense only for cyclic patches) //preservePatches (cyclic_half0 cyclic_half1); //- Keep all of faceSet on a single processor. This puts all cells // connected with a point, edge or face on the same processor. // (just having face connected cells might not guarantee a balanced // decomposition) // The processor can be -1 (the decompositionMethod chooses the processor // for a good load balance) or explicitly provided (upsets balance). //singleProcessorFaceSets ((f0 -1)); //- Use the volScalarField named here as a weight for each cell in the // decomposition. For example, use a particle population field to decompose // for a balanced number of particles in a lagrangian simulation. // weightField dsmcRhoNMean; // method scotch; method hierarchical; //method simple; // method metis; // method manual; // method multiLevel; // method structured; // does 2D decomposition of structured mesh multiLevelCoeffs { // Decomposition methods to apply in turn. This is like hierarchical but // fully general - every method can be used at every level. level0 { numberOfSubdomains 48; //method simple; //simpleCoeffs //{ // n (2 1 1); // delta 0.001; //} method scotch; } level1 { numberOfSubdomains 48; method scotch; } } // Desired output simpleCoeffs { n (12 4 1); delta 0.001; } hierarchicalCoeffs { n (12 4 1); delta 0.001; order xyz; } metisCoeffs { /* processorWeights ( 1 1 1 1 ); */ } scotchCoeffs { //processorWeights //( // 1 // 1 // 1 // 1 //); //writeGraph true; //strategy "b"; } manualCoeffs { dataFile "decompositionData"; } structuredCoeffs { // Patches to do 2D decomposition on. Structured mesh only; cells have // to be in 'columns' on top of patches. patches (bottomPatch); } //// Is the case distributed? Note: command-line argument -roots takes //// precedence //distributed yes; //// Per slave (so nProcs-1 entries) the directory above the case. //roots //( // "/tmp" // "/tmp" //); // ************************************************************************* // Code:
FoamFile { version 2.0; format ascii; class dictionary; object snappyHexMeshDict; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // // Which of the steps to run castellatedMesh true; snap true; addLayers true; //true; // Geometry. Definition of all surfaces. All surfaces are of class // searchableSurface. // Surfaces are used // - to specify refinement for any mesh cell intersecting it // - to specify refinement for any mesh cell inside/outside/near // - to 'snap' the mesh boundary to the surface geometry { /* blk01.stl { type triSurfaceMesh; name blk01; } blk02.stl { type triSurfaceMesh; name blk02; } blk03.stl { type triSurfaceMesh; name blk03; } spire.stl { type triSurfaceMesh; name spire; } */ building.stl { type triSurfaceMesh; name building; } /* panel1.stl { type triSurfaceMesh; name panel1; } panel2.stl { type triSurfaceMesh; name panel2; } panel3.stl { type triSurfaceMesh; name panel3; } panel4.stl { type triSurfaceMesh; name panel4; } panel5.stl { type triSurfaceMesh; name panel5; } panel6.stl { type triSurfaceMesh; name panel6; } panel7.stl { type triSurfaceMesh; name panel7; } */ //---------------------------- /* refBoxblk00 { type searchableBox; min (-12.8 -1.5 0); max ( 5 1.5 1.2); } refBoxblk01 { type searchableBox; min (-9 -1.5 0); max ( 3 1.5 0.8); } refBoxblk02 { type searchableBox; min (-4 -1.5 0); max ( 2 1.5 0.6); } refBoxblk03 { type searchableBox; min (-2.2 -1.5 0); max ( 1.2 1.5 0.3); } refBoxSpire01 { type searchableBox; min (-11.5 -1.5 0); max (-10.1 1.5 1.95); } refBoxSpire02 { type searchableBox; min (-11.2 -1.5 0); max (-10.4 1.5 1.9); } */ refBoxtg00 { type searchableBox; min (-0.8 -0.8 0.0); max (1 0.8 0.7); } refBoxtg01 { type searchableBox; min (-0.5 -0.5 0.0); max (0.7 0.5 0.6); } refBoxtg02 { type searchableBox; min (-0.35 -0.35 0.0); max (0.5 0.35 0.5); } refBoxtg03 { type searchableBox; min (-0.28 -0.28 0.0); max (0.3 0.28 0.42); } }; // Settings for the castellatedMesh generation. castellatedMeshControls { // Refinement parameters // ~~~~~~~~~~~~~~~~~~~~~ // While refining maximum number of cells per processor. This is basically // the number of cells that fit on a processor. If you choose this too small // it will do just more refinement iterations to obtain a similar mesh. maxLocalCells 20000000; // Overall cell limit (approximately). Refinement will stop immediately // upon reaching this number so a refinement level might not complete. // Note that this is the number of cells before removing the part which // is not 'visible' from the keepPoint. The final number of cells might // actually be a lot less. maxGlobalCells 240000000;//240million // OpenFOAM-1.7.1 // Allow a certain level of imbalance during refining // (since balancing is quite expensive) // Expressed as fraction of perfect balance (= overall number of cells / // nProcs). 0=balance always. maxLoadUnbalance 0.10; // The surface refinement loop might spend lots of iterations refining just a // few cells. This setting will cause refinement to stop if <= minimumRefine // are selected for refinement. Note: it will at least do one iteration // (unless the number of cells to refine is 0) minRefinementCells 10; //10; // Number of buffer layers between different levels. // 1 means normal 2:1 refinement restriction, larger means slower // refinement. nCellsBetweenLevels 3; // Explicit feature edg // This is a featureEdgeMesh, read from constant/triSurface for now. features ( { file "building.eMesh"; level 6; } /* { file "panel1.eMesh"; level 7; } { file "panel2.eMesh"; level 7; }e refinement // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ // Specifies a level for any cell intersected by its edges. // This is a featureEdgeMesh, read from constant/triSurface for now. features ( { file "building.eMesh"; level 7; } /* { file "panel1.eMesh"; level 7; } { file "panel2.eMesh"; level 7; } { file "panel3.eMesh"; level 7; } { file "panel4.eMesh"; level 7; } { file "panel5.eMesh"; level 7; } { file "panel6.eMesh"; level 7; } { file "panel7.eMesh"; level 7; } */ ); // Surface based refinement // ~~~~~~~~~~~~~~~~~~~~~~~~ // Specifies two levels for every surface. The first is the minimum level, // every cell intersecting a surface gets refined up to the minimum level. // The second level is the maximum level. Cells that 'see' multiple // intersections where the intersections make an // angle > resolveFeatureAngle get refined up to the maximum level. refinementSurfaces { /* blk01 { // Surface-wise min and max refinement level level (3 4); // (5 6) } blk02 { // Surface-wise min and max refinement level level (3 4); // (5 6) } blk03 { // Surface-wise min and max refinement level level (4 5); // (5 6) } spire { // Surface-wise min and max refinement level level (3 4); // (5 6) } */ building { level (6 7); } /* panel1 { level (6 7); } panel2 { level (6 7); } panel3 { level (6 7); } panel4 { level (6 7); } panel5 { level (6 7); } panel6 { level (6 7); } panel7 { level (6 7); } */ } // Resolve sharp angles resolveFeatureAngle 30; // Region-wise refinement // ~~~~~~~~~~~~~~~~~~~~~~ // Specifies refinement level for cells in relation to a surface. One of // three modes // - distance. 'levels' specifies per distance to the surface the // wanted refinement level. The distances need to be specified in // descending order. // - inside. 'levels' is only one entry and only the level is used. All // cells inside the surface get refined up to the level. The surface // needs to be closed for this to be possible. // - outside. Same but cells outside. refinementRegions { /* blk01 { mode distance; levels ((0.1 2) (0.2 1)); } blk02 { mode distance; levels ((0.1 3) (0.2 2)); } blk03 { mode distance; levels ((0.05 4) (0.1 3)); } refBoxblk00 { mode inside; levels ((0 0)); } refBoxblk01 { mode inside; levels ((0 1)); } refBoxblk02 { mode inside; levels ((0 2)); } refBoxblk03 { mode inside; levels ((0 3)); } refBoxSpire01 { mode inside; levels ((0 1)); } refBoxSpire02 { mode inside; levels ((0 2)); } */ refBoxtg00 { mode inside; levels ((1E15 3)); } refBoxtg01 { mode inside; levels ((1E15 4)); } refBoxtg02 { mode inside; levels ((1E15 5)); } refBoxtg03 { mode inside; levels ((1E15 6)); } } // Mesh selection // ~~~~~~~~~~~~~~ // After refinement patches get added for all refinementSurfaces and // all cells intersecting the surfaces get put into these patches. The // section reachable from the locationInMesh is kept. // NOTE: This point should never be on a face, always inside a cell, even // after refinement. locationInMesh (0 0 1.0); allowFreeStandingZoneFaces true; } // Settings for the snapping. snapControls { //- Number of patch smoothing iterations before finding correspondence // to surface nSmoothPatch 3; //- Relative distance for points to be attracted by surface feature point // or edge. True distance is this factor times local // maximum edge length. tolerance 4.0; //- Number of mesh displacement relaxation iterations. nSolveIter 30; //- Maximum number of snapping relaxation iterations. Should stop // before upon reaching a correct mesh. nRelaxIter 5; } // Settings for the layer addition. addLayersControls { relativeSizes true; // Per final patch (so not geometry!) the layer information layers { building //patch name { nSurfaceLayers 3; //3 } } // Expansion factor for layer mesh expansionRatio 1.5; //- Wanted thickness of final added cell layer. If multiple layers // is the // thickness of the layer furthest away from the wall. // Relative to undistorted size of cell outside layer. finalLayerThickness 0.5; //0.05;//0.45; //- Minimum thickness of cell layer. If for any reason layer // cannot be above minThickness do not add layer. // Relative to undistorted size of cell outside layer. minThickness 0.1; //0.01;//0.25; //- If points get not extruded do nGrow layers of connected faces that are // also not grown. This helps convergence of the layer addition process // close to features. nGrow 1; // Advanced settings //- When not to extrude surface. 0 is flat surface, 90 is when two faces // make straight angle. featureAngle 30; //- Maximum number of snapping relaxation iterations. Should stop // before upon reaching a correct mesh. nRelaxIter 5; // Number of smoothing iterations of surface normals nSmoothSurfaceNormals 1; // Number of smoothing iterations of interior mesh movement direction nSmoothNormals 3; // Smooth layer thickness over surface patches nSmoothThickness 10; // Stop layer growth on highly warped cells maxFaceThicknessRatio 0.5; // Reduce layer growth where ratio thickness to medial // distance is large maxThicknessToMedialRatio 0.3; // Angle used to pick up medial axis points minMedianAxisAngle 130; // Create buffer region for new layer terminations nBufferCellsNoExtrude 0; // Overall max number of layer addition iterations nLayerIter 50; } // Generic mesh quality settings. At any undoable phase these determine // where to undo. meshQualityControls { //- Maximum non-orthogonality allowed. Set to 180 to disable. maxNonOrtho 65; //- Max skewness allowed. Set to <0 to disable. maxBoundarySkewness 20; maxInternalSkewness 4; //- Max concaveness allowed. Is angle (in degrees) below which concavity // is allowed. 0 is straight face, <0 would be convex face. // Set to 180 to disable. maxConcave 80; //- Minimum projected area v.s. actual area. Set to -1 to disable. minFlatness 0.5; //- Minimum pyramid volume. Is absolute volume of cell pyramid. // Set to very negative number (e.g. -1E30) to disable. minVol 1e-13; minTetQuality 1e-30; //- Minimum face area. Set to <0 to disable. minArea -1; //- Minimum face twist. Set to <-1 to disable. dot product of face normal //- and face centre triangles normal minTwist 0.02; //- minimum normalised cell determinant //- 1 = hex, <= 0 = folded or flattened illegal cell minDeterminant 0.001; //- minFaceWeight (0 -> 0.5) minFaceWeight 0.02; //- minVolRatio (0 -> 1) minVolRatio 0.01; //must be >0 for Fluent compatibility minTriangleTwist -1; // Advanced //- Number of error distribution iterations nSmoothScale 4; //- amount to scale back displacement at error points errorReduction 0.75; } // Advanced // Flags for optional output // 0 : only write final meshes // 1 : write intermediate meshes // 2 : write volScalarField with cellLevel for postprocessing // 4 : write current intersections as .obj files debug 0; // Merge tolerance. Is fraction of overall bounding box of initial mesh. // Note: the write tolerance needs to be higher than this. mergeTolerance 1E-6; // ************************************************************************* // |
|
January 14, 2018, 01:37 |
|
#5 |
Member
Jingxue Wang
Join Date: Sep 2017
Posts: 58
Rep Power: 9 |
Thanks,Taataa.
I attched the codes in the previous reply. Thanks! |
|
January 14, 2018, 09:20 |
|
#6 |
Senior Member
Taher Chegini
Join Date: Nov 2014
Location: Houston, Texas
Posts: 125
Rep Power: 13 |
You didn't attach the log files, first we need to see ithe errors. You can zip the logs and attach them.
|
|
January 14, 2018, 11:19 |
|
#7 |
Member
Jingxue Wang
Join Date: Sep 2017
Posts: 58
Rep Power: 9 |
I zipped the logs files and my setting parameters files. I used 2 nodes (48cpus)in the remote server. I tried this again and obtained the following log files. The error information seems different with I attached. But I don't know the reason.
Many thanks! |
|
January 14, 2018, 11:19 |
|
#8 |
Member
Jingxue Wang
Join Date: Sep 2017
Posts: 58
Rep Power: 9 |
I zipped the logs files and my setting parameters files. I used 2 nodes (48cpus)in the remote server. I tried this again and obtained the following log files. The error information seems different with I attached. But I don't know the reason.
Thanks!I really appreciate your help! |
|
January 14, 2018, 11:41 |
|
#9 |
Senior Member
Taher Chegini
Join Date: Nov 2014
Location: Houston, Texas
Posts: 125
Rep Power: 13 |
Based on the error in the log files, I have three suggestions. Use scotch for decomposition, try using less cpus for now, say 8, and decrease cell size of the background mesh generated by blockMesh based on the desired maximum size of the final mesh.
Also, make sure that you have enough memory available, roughly 1G for 1 million cells. Just a suggestion, try to clean up the dictionaries. They have lots of comments and redundant lines I usually keep the notes in a separate file so I can keep the dictionaries clean. Last edited by Taataa; January 14, 2018 at 11:42. Reason: added background |
|
January 14, 2018, 11:44 |
|
#10 |
Member
Jingxue Wang
Join Date: Sep 2017
Posts: 58
Rep Power: 9 |
Thanks,Taattaa. I will try again
|
|
January 14, 2018, 11:45 |
|
#11 |
Member
Jingxue Wang
Join Date: Sep 2017
Posts: 58
Rep Power: 9 |
I am wondering that scocth decompose is better than others?
|
|
January 14, 2018, 11:51 |
|
#12 |
Senior Member
Taher Chegini
Join Date: Nov 2014
Location: Houston, Texas
Posts: 125
Rep Power: 13 |
Yes, it is. It usually does a very good job. Among the existing ones, scotch works best without having to tweak anything. I forgot to mention that for snappyHexMesh you should set it as ptscotch and for the main simulation, scotch.
|
|
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Parallel computing on multiple machines | Henry Arrigo | STAR-CCM+ | 2 | July 9, 2014 03:43 |
Fluent parallel computing on multiple Workstations slower than one | peterhess | Hardware | 1 | May 18, 2011 09:08 |
Parallel Computing on more than one machine | harly | OpenFOAM Running, Solving & CFD | 4 | December 9, 2008 16:41 |
Parallel Computing | peter | Main CFD Forum | 7 | May 15, 2006 10:53 |
Parallel Computing Classes at San Diego Supercomputer Center Jan. 20-22 | Amitava Majumdar | Main CFD Forum | 0 | January 5, 1999 13:00 |