| heavy_user |
July 2, 2013 12:55 |
particles parallelization transfere cloud move
Hi All,
i ran into a huge problem and i need help!
I am doing a LES-PDF method using MC particles and the particles carry a lot of information (each particle has about 35 doubles which are transfered via stream and not reinitiated by a preinitialization list).
The paralellization is REALLY bad!
Using 48 cores - one board, quadsocket, xeon cpus - the time is spend as follows (318087 LES cells, 1556621 particles ):
- 99.4 % of the time is used for particles(rest for LES)
- The particle move takes 96.7 % of the particle time
- and the transfer takes 94 % of that time
so essentially the computer is only bussy moving particles.
The routine consuming all the resources (in myCloud.move() ) is
Code:
while ( true ) <- what the heck is that deathloop any ways !?
....
....
..
if (!Pstream::parRun())
{
break;
}
// Allocate transfer buffers
PstreamBuffers pBufs(Pstream::nonBlocking);
// Stream into send buffers
forAll(particleTransferLists, i)
{
if (particleTransferLists[i].size())
{
UOPstream particleStream
(
neighbourProcs[i],
pBufs
);
particleStream
<< patchIndexTransferLists[i]
<< particleTransferLists[i];
}
}
// Set up transfers when in non-blocking mode. Returns sizes (in bytes)
// to be sent/received.
labelListList allNTrans(Pstream::nProcs());
pBufs.finishedSends(allNTrans);
bool transfered = false;
forAll(allNTrans, i)
{
forAll(allNTrans[i], j)
{
if (allNTrans[i][j])
{
transfered = true;
break;
}
}
}
if (!transfered)
{
break;
}
// Retrieve from receive buffers
forAll(neighbourProcs, i)
{
label neighbProci = neighbourProcs[i];
label nRec = allNTrans[neighbProci][Pstream::myProcNo()];
if (nRec)
{
UIPstream particleStream(neighbProci, pBufs);
labelList receivePatchIndex(particleStream);
IDLList<myHardBallParticle> newParticles
(
particleStream,
//typename myHardBallParticle::iNew(polyMesh_)
myHardBallParticle::iNew(*this)
);
label pI = 0;
forAllIter(myIncompressibleCloud, newParticles, newpIter)
{
myHardBallParticle& newp = newpIter();
label patchI = procPatches[receivePatchIndex[pI++]];
newp.correctAfterParallelTransfer(patchI, td);
addParticle(newParticles.remove(&newp));
}
}
Thats from OpenFOAM-2.1x (which promised new particle tracking :( )
I would appreciate any ideas or comments!!! |
