CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Programming & Development

Parallel communication with cellZones

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree2Likes
  • 1 Post By ngj
  • 1 Post By A_Pete

Reply
 
LinkBack Thread Tools Display Modes
Old   February 21, 2014, 08:59
Default Parallel communication with cellZones
  #1
Member
 
Join Date: Jul 2011
Posts: 33
Rep Power: 6
A_Pete is on a distinguished road
Hi all,

I am trying to program a functionObject, in which a cellZone is assigned to every face of one predefined patch. This is needed, because for every face of this patch, all the cells of the assigned cellZone are checked for a condition. If one of the cells contained in the cellZone does fullfil the condition, the next face (and its belonging cellZone) is examined.

This is basically done in a for loop over all the faces of my regarded patch. This was not too complicated to program for a serial run. But in parallel run the problem I am having is that faces and cells of the assigned cellZone are going to lie on different processors, so that for some faces not their full cellZones are examined.

A basic example is shown below:

Code:
//- Loop over the patch
forAll(p.boundaryField()[patchID], i)
{
      //- Lookup of cellZone mesh, IDs etc. has happened before
      const labelUList& cellZone(mesh.cellZones()[zoneID]);
      forAll(cellZone, j)
      {
      //- Check of conditions where both the local cells of the cellZone
      // and the regarded face are involved
      }
}
Does anybody have an idea how I can solve this problem? Maybe I have to do all the calculation on the master processor, but this would mean that I would have to have all the cellZones and faces on this processor. That would be very uneffective.

Thanks in advance.
A_Pete is offline   Reply With Quote

Old   February 26, 2014, 08:27
Default
  #2
Member
 
Join Date: Jul 2011
Posts: 33
Rep Power: 6
A_Pete is on a distinguished road
Any ideas?
A_Pete is offline   Reply With Quote

Old   February 26, 2014, 15:24
Default
  #3
ngj
Senior Member
 
Niels Gjoel Jacobsen
Join Date: Mar 2009
Location: Deltares, Delft, The Netherlands
Posts: 1,609
Rep Power: 25
ngj will become famous soon enoughngj will become famous soon enough
Hi Pete,

The following will merely be a sketch of a solution, so I hope that you can fill in the blanks yourself.

First of all, you will have to stop using the forAll command for this particular investigation. When you are using it, you are only looping over the set of indices on a given processor. You, on the other hand, want to loop over all indices of a given set irrespectively of their presence on a given processor or not.

Task 1: Create a list of indices, which is gathered from all processors. gatherList/scatterList will prove useful.
Task 2: Make sure that you know, which part belongs to which processor.
Task 3: Loop over this list and investigate the face condition, but only on the relevant processor. Before the involved if-statement, define "label test = 0;" on all processors and assign the result of the if-statement to test (but again, only for the relevant processor).
Task 4: Apply "Foam::reduce(test, maxOp<label>);", which distributes the test result across all processors.

Now you know (on all processors), whether corrections needs to be assigned to the cells in the relevant cellZone.

Good luck,

Niels
A_Pete likes this.
__________________
Please note that I do not use the Friend-feature, so do not be offended, if I do not accept a request.
ngj is offline   Reply With Quote

Old   March 6, 2014, 06:47
Default
  #4
Member
 
Join Date: Jul 2011
Posts: 33
Rep Power: 6
A_Pete is on a distinguished road
Hi Niels,

thanks for your input. I did pick up your ideas and made some investigations, also with lots of gathering and scattering of data.

Now, it looks like the programming is not the problem itself, but my generation of the cellZones. I have written a small program, which is doing the following:

Lookup all the faces for a predefined patch name.
Generate a dictionary for topoSet, which will then create a cellSet for every single one of these faces.
The cells in this set are selected with a sphere with the face centre as mid point.
So the cells around this face are selected within a defined distance.

The problem I am having appears, when doing this in parallel:
1. I have to generate the cellSet on every processor using my small program. If a face is now positioned near the boundary to a neighbouring processor, the cellSet is not going to contain all the cells that should be selected by the sphere, because some of these cells are on the neighbouring processor.
2. These cells cannot be selected for this cellSet, because the cellSets on the neighbouring processor are all for other faces... the faces on this processor.


So does somebody have an idea, how I can select cells around a face in a predefined distance (with a sphere) and access them across multiple processors?
A_Pete is offline   Reply With Quote

Old   March 6, 2014, 17:06
Default
  #5
ngj
Senior Member
 
Niels Gjoel Jacobsen
Join Date: Mar 2009
Location: Deltares, Delft, The Netherlands
Posts: 1,609
Rep Power: 25
ngj will become famous soon enoughngj will become famous soon enough
Good evening,

If you create the cellSets in serial, turn them into cellZones and then decompose, the cellZones are all decomposed.

Hope that it helps,

NieŠs
__________________
Please note that I do not use the Friend-feature, so do not be offended, if I do not accept a request.
ngj is offline   Reply With Quote

Old   March 7, 2014, 04:13
Default
  #6
Member
 
Join Date: Jul 2011
Posts: 33
Rep Power: 6
A_Pete is on a distinguished road
Hi Niels,

thanks for your input again. This is what I have also figured out now. As you said, I executed topoSet, setsToZones in serial. Afterwards, I decomposed the case and do now have all the cellZones of my serial case on each processor. Just the cellZones on the local processors are not empty, which is exactly what I want.

I have now written a temporary solution for my problem, which looks like this:

I am giving the cellZones names which contain the faceCentre coordinates. I thought of that as an "ID" that is the same for the serial case and for the parallel case, since the faceIDs of serial and decomposed cases are different.
What I do then is collect the vectors (e.g. U field) for all the cells in my cellZones locally on every processor. Then I gather these vectors for the cellZones from all processors and reorder them so that I have all the vectors on every processor.
I can then commit only a portion of these vectors, when I need to check my condition.
Finally, I loop over my boundaryMesh and reconstruct this faceCentre based ID and lookup the names of the cellZones, where the U vectors for each cellZone are in.

Very complicated right now and probably not as efficient as it can be, but it gets the job done right now. Still, I am working on a better solution than this "breaking-up-the-parallelization".

Thanks for your input again.


Best regards,
Andreas
ngj likes this.
A_Pete is offline   Reply With Quote

Old   March 7, 2014, 04:22
Default
  #7
ngj
Senior Member
 
Niels Gjoel Jacobsen
Join Date: Mar 2009
Location: Deltares, Delft, The Netherlands
Posts: 1,609
Rep Power: 25
ngj will become famous soon enoughngj will become famous soon enough
Hi Pete,

Thanks for the update. The use of the coordinate as consistent naming scheme is a good one. I previously worked on a similar problem for point indices, but I did not think of using the coordinates as a marker, why it turned out to be limited to serial runs, as the reconstruction of the connectivity will otherwise become quite tedious.

Kind regards,

Niels
__________________
Please note that I do not use the Friend-feature, so do not be offended, if I do not accept a request.
ngj is offline   Reply With Quote

Old   March 7, 2014, 04:54
Default
  #8
Member
 
Join Date: Jul 2011
Posts: 33
Rep Power: 6
A_Pete is on a distinguished road
Hi Niels,

glad that you could also get something out of this!

Those are exactly my thoughts. The centres seem like a good idea.
I was also worried about some numerical differences (e.g. due to rounding) of the faceCentres between the serial and parallel run, but it seems to work so far.

Downside is probably that when you have parts of your mesh moving, the coordinates of the faces may change. The connection of communication should then probably be established at construction of the functionObject or w/e you are using. This is something I am working on right now, since I now read in the velocities every time step. This should definately be prevented.

Kind regards,
Andreas
A_Pete is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
OF-2.2.x: Can't access cellZones in parallel run A_Pete OpenFOAM Programming & Development 1 February 14, 2014 05:24
simpleFoam parallel AndrewMortimer OpenFOAM Running, Solving & CFD 10 November 12, 2013 18:03
simpleFoam in parallel issue plucas OpenFOAM Running, Solving & CFD 3 July 17, 2013 11:30
Parallel communication problem, port 8676 jreig FLUENT 3 May 28, 2010 02:24
Basic Parallel Communication Issues deepsterblue OpenFOAM Programming & Development 1 July 6, 2009 22:25


All times are GMT -4. The time now is 12:38.