[fedvasu]

Hi,

I am calculating averages on physical boundaries of my domain. However when I am calculating these things in parallel my solver crashes.

Because, my calculation happens on processor patches too. As my calculation has some stochastic component to it, the values across processor patches are never exactly same.

For example face values on procBoundary0to1 on processor 0 won't be same as procBoundary1to0 on processor 1.

I tried keeping my face values at procBoundary* to 0 and even tried not doing anything at all. However it seems I need proper values there.

So How can I ensure procBoundary0to1 on processor 0 = procBoundary1to0 on processor 1.

It seems i can't reduce or synchronize fvPatchScalarField&.

[fedvasu]

Here is a simplified code snippet

forAll(T_.mesh().boundary(), patchi)
{
//here patchi also includes procBoundary, I can get it's patchid and name just fine
const fvPatchScalarField& pZ    = Z_.boundaryField()[patchi];
    forAll(pZ , facei)
    {
          if(pZ<MIN_VAL)
          {
             // do deterministic calculation
          }

         else
         {
            //do stochastic calcuation
         }

    }
           
}

[hjasak]

You should not be doing this. The processor0To1 and processor1To0 actually rerepsent the same face, on two processors. If they are inconsistent in ANY WAY, this is no longer the same face.

Think of this as 1 single internal mesh face: you need to repeat the exact same operation, but fetching the data in a different way - using patchInternalField() and patchNeighbourField()

If you do anything else, the code may appear to work, but you will break the linear algebra package.

Hope this helps,

Hrv

[fedvasu]

Originally Posted by hjasak

You should not be doing this. The processor0To1 and processor1To0 actually rerepsent the same face, on two processors. If they are inconsistent in ANY WAY, this is no longer the same face.

Think of this as 1 single internal mesh face: you need to repeat the exact same operation, but fetching the data in a different way - using patchInternalField() and patchNeighbourField()

If you do anything else, the code may appear to work, but you will break the linear algebra package.

Hope this helps,

Hrv

Thank You Dr Jasak, I tried using patchInternalField() and patchNeighbourField() for procBoundary but it didn't work, so I am not sure what you meant by that. I am bit confused.

Here is a simplified code snippet THIS DOESN"T WORK!

forAll(T_.mesh().boundary(), patchi)
{
const fvPatchScalarField& pZ    = Z_.boundaryField()[patchi];
const fvPatch& curPatch = T_.mesh().boundary()[patchi];
std::string patchName   = curPatch.name();

//if (isA<processorPolyPatch>(curPatch))
if(patchName.rfind("procBoundary",0) == 0)
{
    const scalarField ppZ    = pZ.patchNeighbourField(); //pZ.patchInternalField(); 
    forAll(ppZ, facei)
    {
        if(ppZ<MIN_VAL)
          {
             // do deterministic calculation
          }

         else
         {
            //do stochastic calcuation
         }
    }
    
}

else
{
    forAll(pZ , facei)
    {
          if(pZ<MIN_VAL)
          {
             // do deterministic calculation
          }

         else
         {
            //do stochastic calcuation
         }

    }
}          
}

However this DOES WORK.

forAll(T_.mesh().boundary(), patchi)
{
const fvPatchScalarField& pZ    = Z_.boundaryField()[patchi];
const fvPatch& curPatch = T_.mesh().boundary()[patchi];
std::string patchName   = curPatch.name();

//if (isA<processorPolyPatch>(curPatch))
if(patchName.rfind("procBoundary",0) == 0)
{
   pT.evaluate(); //or  pT.evaluate(Pstream::blocking);
  //all other patchFields follow
}

else
{
    forAll(pZ , facei)
    {
          if(pZ<MIN_VAL)
          {
             // do deterministic calculation
          }

         else
         {
            //do stochastic calcuation
         }

    }
}          
}

[fedvasu]

I would like to post THIS WORKING solution again for clarity.

forAll(T_.mesh().boundary(), patchi)
{
const fvPatchScalarField& pZ    = Z_.boundaryField()[patchi];
const fvPatch& curPatch = T_.mesh().boundary()[patchi];
std::string patchName   = curPatch.name();

//if (isA<processorPolyPatch>(curPatch))
if(patchName.rfind("procBoundary",0) == 0)
{
  pT.evaluate(); //or  pT.evaluate(Pstream::blocking);
  //all other patchFields follow 
}

else
{
    forAll(pZ , facei)
    {
          if(pZ<MIN_VAL)
          {
             // do deterministic calculation
          }

         else
         {
            //do stochastic calcuation
         }

    }
}          
}

I still don't understand how exactly the evaluate() communicates across processor boundary, but this solves my problem.

Anyone knows exactly what evaluate() does and how it avoids my problem feel free to add to this thread. I will keep this thread unsolved till then.