CFD Online Discussion Forums - Run parallel but low calculating speed

Dear foamers:
I used OpenFOAM to calculate turbulent boundary layers (TBL) on supercomputer. I tried to divide computing domain into 4, 10, 24parts respectively to run them in parallel.:o The point is that the more parts I divided the domain, the more calculating time it consumed. The numerical methodology I used was LES and the sub-grid model was dynSmagorinsky.
The size of computing domain was 90mm(stream-wise)*10mm(span-wise)*10mm(vertical to the smooth plane). The inlet velocity was 7m/s. There was a 19mm import part in front of the smooth plane and a 20mm export part behind the smooth plane, both of which were slip boundary. Between the smooth plane and import part was a semi-cylinder trip(r=0.5mm) to add perturbation to the field. The top boundary type was slip. The side boundary types are cyclic.
The number of grid is about 8 million. The smallest grids closed to the smooth plane are 0.002mm in length.
my checkMesh log:
Create time

Create polyMesh for time = 0

Time = 0

Mesh stats
all points: 8134521
live points: 8134521
all faces: 24033500
live faces: 24033500
internal faces: 23666500
cells: 7950000
boundary patches: 8
point zones: 0
face zones: 0
cell zones: 0

Overall number of cells of each type:
hexahedra: 7950000
prisms: 0
wedges: 0
pyramids: 0
tet wedges: 0
tetrahedra: 0
polyhedra: 0

Checking topology...
Boundary definition OK.
Point usage OK.
Upper triangular ordering OK.
Face vertices OK.
Number of regions: 1 (OK).

Checking patch topology for multiply connected surfaces ...
Patch Faces Points Area [m^2] Surface topology
INLET 10500 10721 0.0001 ok (non-closed singly connected)
OUTLET 10500 10721 0.0001 ok (non-closed singly connected)
UP 111000 111891 0.0009 ok (non-closed singly connected)
FRONT 7500 7701 0.00019 ok (non-closed singly connected)
PLANE 100500 101321 0.0005 ok (non-closed singly connected)
BEHIND 6000 6191 0.0002 ok (non-closed singly connected)
TRIP 15000 15251 1.57023e-05 ok (non-closed singly connected)
PERIODIC 106000 107742 0.00179922 ok (non-closed singly connected)

Checking geometry...
This is a 3-D mesh
Overall domain bounding box (-0.0594563 -4.33681e-19 -0.0146693) (0.0305437 0.01 -0.00466934)
Mesh (non-empty, non-wedge) directions (1 1 1)
Mesh (non-empty) directions (1 1 1)
Mesh (non-empty, non-wedge) dimensions 3
Boundary openness (1.03848e-15 1.55099e-13 2.48007e-14) Threshold = 1e-06 OK.
Max cell openness = 3.30422e-16 OK.
Max aspect ratio = 555.656 OK.
Minumum face area = 3.99642e-12. Maximum face area = 4.50651e-07. Face area magnitudes OK.
Min volume = 2.66428e-16. Max volume = 3.00434e-11. Total volume = 8.99608e-06. Cell volumes OK.
Mesh non-orthogonality Max: 43.0859 average: 6.23771 Threshold = 70
Non-orthogonality check OK.
Face pyramids OK.
Max skewness = 2.11452 OK.

Mesh OK.

--> FOAM Warning :
From function polyMesh::readUpdateState polyMesh::readUpdate()
in file meshes/polyMesh/polyMeshIO.C at line 226
Number of patches has changed. This may have unexpected consequences. Proceed with care.
Time = 1e-07

Mesh stats
all points: 8134521
live points: 8134521
all faces: 24033500
live faces: 24033500
internal faces: 23666500
cells: 7950000
boundary patches: 8
point zones: 0
face zones: 0
cell zones: 0

Overall number of cells of each type:
hexahedra: 7950000
prisms: 0
wedges: 0
pyramids: 0
tet wedges: 0
tetrahedra: 0
polyhedra: 0

Checking topology...
Boundary definition OK.
Point usage OK.
Upper triangular ordering OK.
Face vertices OK.
Number of regions: 1 (OK).

Checking patch topology for multiply connected surfaces ...
Patch Faces Points Area [m^2] Surface topology
INLET 10500 10721 0.0001 ok (non-closed singly connected)
OUTLET 10500 10721 0.0001 ok (non-closed singly connected)
UP 111000 111891 0.0009 ok (non-closed singly connected)
FRONT 7500 7701 0.00019 ok (non-closed singly connected)
PLANE 100500 101321 0.0005 ok (non-closed singly connected)
BEHIND 6000 6191 0.0002 ok (non-closed singly connected)
TRIP 15000 15251 1.57023e-05 ok (non-closed singly connected)
PERIODIC 106000 107742 0.00179922 ok (non-closed singly connected)

Checking geometry...
This is a 3-D mesh
Overall domain bounding box (-0.0594563 -4.33681e-19 -0.0146693) (0.0305437 0.01 -0.00466934)
Mesh (non-empty, non-wedge) directions (1 1 1)
Mesh (non-empty) directions (1 1 1)
Mesh (non-empty, non-wedge) dimensions 3
Boundary openness (1.03848e-15 1.55099e-13 2.48007e-14) Threshold = 1e-06 OK.
Max cell openness = 3.30422e-16 OK.
Max aspect ratio = 555.656 OK.
Minumum face area = 3.99642e-12. Maximum face area = 4.50651e-07. Face area magnitudes OK.
Min volume = 2.66428e-16. Max volume = 3.00434e-11. Total volume = 8.99608e-06. Cell volumes OK.
Mesh non-orthogonality Max: 43.0859 average: 6.23771 Threshold = 70
Non-orthogonality check OK.
Face pyramids OK.
Max skewness = 2.11452 OK.

Mesh OK.

End
fvSchemes:
ddtSchemes
{
default backward;
}

gradSchemes
{
default Gauss linear;
grad(p) Gauss linear;
grad(U) Gauss linear;
}

divSchemes
{
default none;
div(phi,U) Gauss linear;
div(phi,k) Gauss limitedLinear 1;
div(phi,B) Gauss limitedLinear 1;
div(phi,nuTilda) Gauss limitedLinear 1;
div(B) Gauss linear;
div((nuEff*dev(grad(U).T()))) Gauss linear;
}

laplacianSchemes
{
default none;
laplacian(nuEff,U) Gauss linear corrected;
laplacian((1|A(U)),p) Gauss linear corrected;
laplacian(DkEff,k) Gauss linear corrected;
laplacian(DBEff,B) Gauss linear corrected;
laplacian(DnuTildaEff,nuTilda) Gauss linear corrected;
}

interpolationSchemes
{
default linear;
interpolate(U) linear;
}

snGradSchemes
{
default corrected;
}

fluxRequired
{
default no;
p ;
}
fvSolution:
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object fvSolution;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

solvers
{
p
{
solver PCG;
preconditioner DIC;
tolerance 1e-06;
relTol 0.05;
}

pFinal
{
solver PCG;
preconditioner DIC;
tolerance 1e-06;
relTol 0;
}

U
{
solver PBiCG;
preconditioner DILU;
tolerance 1e-05;
relTol 0;
}

k
{
solver PBiCG;
preconditioner DILU;
tolerance 1e-05;
relTol 0;
}

B
{
solver PBiCG;
preconditioner DILU;
tolerance 1e-05;
relTol 0;
}

nuTilda
{
solver PBiCG;
preconditioner DILU;
tolerance 1e-05;
relTol 0;
}
}

PISO
{
nCorrectors 2;
nNonOrthogonalCorrectors 0;
}
one of the decomposeParDicts:
numberOfSubdomains 24;

method hierarchical;

simpleCoeffs
{
n (2 1 2);
delta 0.001;
}

hierarchicalCoeffs
{
n (4 2 3);
delta 0.001;
order xzy;
}
controlDict:

application pisoFoam;

startFrom startTime;

startTime 0;

stopAt endTime;

endTime 1;

deltaT 0.0000005;

writeControl timeStep;

writeInterval 2000;

purgeWrite 0;

writeFormat ascii;

writePrecision 6;

writeCompression uncompressed;

timeFormat general;

timePrecision 6;

runTimeModifiable yes;

functions
(
forces
{
type forceCoeffs;
functionObjectLibs ( "libforces.so" );
outputControl timeStep;
outputInterval 2000;
patches
(
PLANE
);
directForceDensity no;

pName p;
UName U;
rhoName rhoInf;
//log true;
rhoInf 1000;
CofR ( 0 0 0 );
liftDir ( 0 1 0 ); //y direction
dragDir ( 1 0 0 ); //x direction
pitchAxis ( 0 0 0 );
magUInf 0.39; //z direction
lRef 1;
Aref 3.14; //
rhoRef 1;
}

fieldAverage1
{
type fieldAverage;
functionObjectLibs ("libfieldFunctionObjects.so");
enabled true;
outputControl outputTime;
fields
(
U
{
mean on;
prime2Mean on;
base time;
}

p
{
mean on;
prime2Mean on;
base time;
}
);
}
);