CFD Online Logo CFD Online URL
Home > Forums > OpenFOAM Running, Solving & CFD

Find neighbours across processors in parallel computing

Register Blogs Members List Search Today's Posts Mark Forums Read

LinkBack Thread Tools Display Modes
Old   December 13, 2008, 13:12
Default The same message but posted un
Heng Xiao
Join Date: Mar 2009
Location: Zurich, Switzerland
Posts: 58
Rep Power: 8
xiao is on a distinguished road
The same message but posted under a more illustrative name.

By Jaswinder Pal Singh on Monday, November 10, 2008 - 02:17 am: Edit Post

Dear Forum

Good Morning

I have some doubts related to parallelization of an OpenFOAM application or solver.

A modiication done to interFoam works fine on single processor but crashes, when run in parallel. A simple test such as :

mesh.ncells() or field.size() give different results when run on single and in parallel.

Also the changes to the solver do some explicit operation. For example :

Look for a cell with a particular value of the volume fraction field gamma and take their neighbours and do some calculation. How to make such operations parallel aware. As far as my understanding goes one has to explicitly take care of it. In code this might look:

forAll(gamma, cellI)
if (0.01 < gamma < 0.99)
perform calcs ....



I request the users with experience of this aspect to please share the information. It would be great if we could compile a list of guidelines one may follow to make sure that their solver is completely compatible to parallel runs.

Hope that I will find some contributors

Kind Regards

By Jaswinder Pal Singh on Monday, November 10, 2008 - 07:23 am: Edit Post

Some more search shows that a reduce operation has to be done whereever something processor dependent is being done. I tried the following:

label cellCount = mesh.nCells();
reduce(cellCount, sumOp<label>());

Info << "Mesh cell count : " << cellCount << endl;

returns the same value for single and parallel, but when I do the same for faces:

label faceCount = mesh.nFaces();
reduce(faceCount, sumOp<label>());

Info << "Mesh face count : " << faceCount << endl;

the output is different as the faceCount now includes the number of faces for the processor patches as well so be aware of that fact. I do not how yet how to get the correct number of faces using reduction.

It seems that the reduce operation can be done for various operations such as

reduce(foundLabel, orOp<bool>())

Still digging....

Any body having a clear understanding please contribute

Kind Regards

By Sandeep Menon on Monday, November 10, 2008 - 07:35 am: Edit Post

I have a suggestion - although I'm not too sure about the feasibility of implementation. MPI inherently numbers processes between 0 and (n-1); where n is the number of subdomains. I'm guessing that every processor patch can identify which MPI process / sub-domain it talks to. So, for every processor patch, identify if the neighbouring sub-domain is numbered higher than the current one. If yes, then add the number of faces for the processor patch to the current sub-domain's face-count, and discard if otherwise. Finally, you can do a conventional reduce with sumOp, to sum up the face count for all sub-domains.

By Henrik Rusche on Sunday, November 30, 2008 - 06:11 am: Edit Post

OpenFOAM uses domain decomposition. Therefore each processor only knows about the cells and faces residing on it.

Your problem is difficult because you want to identify cells and then do calculations with their neighbours. This stretches the "domain decomposition" paradigmn because the neighbour might reside on a different processor.

My recipie for you:

1) Create an indicator field locally
2) interpolate the result to the faces (This should update the processor-processor boundaries)
3) Identify the neighbour cells: They have faces with non-zero values in the interpolated indicator field
4) Do the calc involving neighbours

There is probably a much more elegant solution, but I would have to dig a lot.


By Ivan Flaminio Cozza on Sunday, November 30, 2008 - 07:58 am: Edit Post

I'm intrested to in this kind of problem as I will create a solver that needs to know for each cell the neighbours, and I want to make it work in parallel.
My question is: does exist any structure in Pstream that store the local cell location?
So, something that tells us if a cell in the x domain is on the boundary with the y domain.

By Nat Trask on Tuesday, December 02, 2008 - 02:29 pm: Edit Post

processorPolyPatch contains a list of faces on the patch between different processors and functions for getting the processor number of each patch.

I believe what you want to do is loop over the faces of your cell, check to see if the faces are on a processorPolyPatch, and then use the functions in processorPolyPatch to return the two processor IDs.


By Heng Xiao on Saturday, December 13, 2008 - 10:09 am: Edit Post

Anyone has a follow-up on this issue? Or are the suggestions (recipie) of Nat and Henrik are the optimal ones already?

Suggestions are appreciated!

xiao is offline   Reply With Quote

Old   December 15, 2008, 13:14
Default I changed to a more illustrati
Heng Xiao
Join Date: Mar 2009
Location: Zurich, Switzerland
Posts: 58
Rep Power: 8
xiao is on a distinguished road
I changed to a more illustrative name hoping someone with this experience would say it and share their experience, but apparently it was not very successful.

Or maybe this is indeed "stretching" OpenFoam design principles, as Henrik Rusche pointed out.
xiao is offline   Reply With Quote


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Parallel fluent not using all processors specified Paul FLUENT 17 April 7, 2008 13:18
Parallel with 2 processors dual-core. Sam FLUENT 4 October 10, 2007 10:43
Parallel Computing on Multi-Core Processors Upgrading Hardware CFX 6 June 7, 2007 15:54
64-bit processors for home computing Ananda Himansu Main CFD Forum 2 March 16, 2004 13:48
Parallel processors Joel Phoenics 1 April 18, 2002 08:09

All times are GMT -4. The time now is 16:16.