creating faceSet
 

I need to create a list of faceSets as follows

PtrList<faceSet> faceSetList(hFaceLevelsTotal) ;
forAll(faceSetList,hLevel) {
boxToFace thisLevel(mesh, treeBoundBox(point(0.0, 0.0, double(hLevel)*dz-dz/10.0),point(L,L, double(hLevel)*dz+dz/10.0))) ;
faceSetList.set(hLevel, new faceSet(mesh,"blah",0)) ;
thisLevel.applyToSet(topoSetSource::NEW, faceSetList[hLevel]) ;
}

I then loop over the faces to get the average in the faceSet as follows.

forAll(faceSetList, hLevel) {
bAvg[hLevel] = 0 ;
forAllConstIter(faceSet, faceSetList[hLevel], iter) {
label faceI = iter.key() ;
bAvg[hLevel] = bAvg[hLevel]+[faceI];
}
}

This seems to work on a smaller grid with 32 procs. But when I scale it up to a much larger grid with 96 procs it always fails after 3 iterations no matter what my initial condition is. The code runs fine without this averaging step. I have a feeling it has something to do with how I am using iter.key(). Can anyone throw some light on this? Also is there a way to make the boxToFace avoid boundary faces?

ganesh

Wrong method
 

This is the wrong way to create the faceSet. It turns out that if we try to create the faceSet after domain decomposition, additional processor boundary faces are added. The best way to do this was to use a script with setSet, use setsToZones and then use simpleFunctionObjects like before to average them. The only problem was that I had to write a script to write the functionObjects part of the controlDict.