Parallel communication with cellZones
 

Hi all,

I am trying to program a functionObject, in which a cellZone is assigned to every face of one predefined patch. This is needed, because for every face of this patch, all the cells of the assigned cellZone are checked for a condition. If one of the cells contained in the cellZone does fullfil the condition, the next face (and its belonging cellZone) is examined.

This is basically done in a for loop over all the faces of my regarded patch. This was not too complicated to program for a serial run. But in parallel run the problem I am having is that faces and cells of the assigned cellZone are going to lie on different processors, so that for some faces not their full cellZones are examined.

A basic example is shown below:

Does anybody have an idea how I can solve this problem? Maybe I have to do all the calculation on the master processor, but this would mean that I would have to have all the cellZones and faces on this processor. That would be very uneffective.

Thanks in advance.

Any ideas?

Hi Pete,

The following will merely be a sketch of a solution, so I hope that you can fill in the blanks yourself.

First of all, you will have to stop using the forAll command for this particular investigation. When you are using it, you are only looping over the set of indices on a given processor. You, on the other hand, want to loop over all indices of a given set irrespectively of their presence on a given processor or not.

Task 1: Create a list of indices, which is gathered from all processors. gatherList/scatterList will prove useful.
Task 2: Make sure that you know, which part belongs to which processor.
Task 3: Loop over this list and investigate the face condition, but only on the relevant processor. Before the involved if-statement, define "label test = 0;" on all processors and assign the result of the if-statement to test (but again, only for the relevant processor).
Task 4: Apply "Foam::reduce(test, maxOp<label>);", which distributes the test result across all processors.

Now you know (on all processors), whether corrections needs to be assigned to the cells in the relevant cellZone.

Good luck,

Niels

Hi Niels,

thanks for your input. I did pick up your ideas and made some investigations, also with lots of gathering and scattering of data.

Now, it looks like the programming is not the problem itself, but my generation of the cellZones. I have written a small program, which is doing the following:

Lookup all the faces for a predefined patch name.
Generate a dictionary for topoSet, which will then create a cellSet for every single one of these faces.
The cells in this set are selected with a sphere with the face centre as mid point.
So the cells around this face are selected within a defined distance.

The problem I am having appears, when doing this in parallel:
1. I have to generate the cellSet on every processor using my small program. If a face is now positioned near the boundary to a neighbouring processor, the cellSet is not going to contain all the cells that should be selected by the sphere, because some of these cells are on the neighbouring processor.
2. These cells cannot be selected for this cellSet, because the cellSets on the neighbouring processor are all for other faces... the faces on this processor.


So does somebody have an idea, how I can select cells around a face in a predefined distance (with a sphere) and access them across multiple processors?

Good evening,

If you create the cellSets in serial, turn them into cellZones and then decompose, the cellZones are all decomposed.

Hope that it helps,

Nieæs

Hi Niels,

thanks for your input again. This is what I have also figured out now. As you said, I executed topoSet, setsToZones in serial. Afterwards, I decomposed the case and do now have all the cellZones of my serial case on each processor. Just the cellZones on the local processors are not empty, which is exactly what I want.

I have now written a temporary solution for my problem, which looks like this:

I am giving the cellZones names which contain the faceCentre coordinates. I thought of that as an "ID" that is the same for the serial case and for the parallel case, since the faceIDs of serial and decomposed cases are different.
What I do then is collect the vectors (e.g. U field) for all the cells in my cellZones locally on every processor. Then I gather these vectors for the cellZones from all processors and reorder them so that I have all the vectors on every processor.
I can then commit only a portion of these vectors, when I need to check my condition.
Finally, I loop over my boundaryMesh and reconstruct this faceCentre based ID and lookup the names of the cellZones, where the U vectors for each cellZone are in.

Very complicated right now and probably not as efficient as it can be, but it gets the job done right now. Still, I am working on a better solution than this "breaking-up-the-parallelization".

Thanks for your input again.


Best regards,
Andreas

Hi Pete,

Thanks for the update. The use of the coordinate as consistent naming scheme is a good one. I previously worked on a similar problem for point indices, but I did not think of using the coordinates as a marker, why it turned out to be limited to serial runs, as the reconstruction of the connectivity will otherwise become quite tedious.

Kind regards,

Niels

Hi Niels,

glad that you could also get something out of this!

Those are exactly my thoughts. The centres seem like a good idea.
I was also worried about some numerical differences (e.g. due to rounding) of the faceCentres between the serial and parallel run, but it seems to work so far.

Downside is probably that when you have parts of your mesh moving, the coordinates of the faces may change. The connection of communication should then probably be established at construction of the functionObject or w/e you are using. This is something I am working on right now, since I now read in the velocities every time step. This should definately be prevented.

Kind regards,
Andreas