CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Programming & Development (http://www.cfd-online.com/Forums/openfoam-programming-development/)
-   -   Problem with Dynamic Load Balacing via PyFoam (http://www.cfd-online.com/Forums/openfoam-programming-development/110431-problem-dynamic-load-balacing-via-pyfoam.html)

knalldi December 11, 2012 07:44

Problem with Dynamic Load Balacing via PyFoam
 
I am trying to implement a so to say load balancing by reconstructing and decomposing every output time during an InterDyMFoam run with dynamicMeshRefinement based on alpha.

My problems are the reconstructParMesh and probably the decomposePar Utilities do not treat respective refinementHistory and pointLevel outputs for the dynamicRefineFvMesh class.

Does anyone have an Idea how to implement the necessary changes to these Utilities?

Edit: I just realized that instead of reconstruct+decomposepar one can just take redistributePar. But the Problem stays the same. I need the labelList pointlevel in the polymesh folder redistributed along with all the other Fields such as u or p hoping that the dynamicrefinemesh class doesnt refine already refined cells anylonger, without the need to have the possibilty of coarsening these cells again, as refinementHistory is a tough output to redistribute. I would greatly appreciate any input.

gschaider December 11, 2012 13:52

Quote:

Originally Posted by knalldi (Post 396890)
I am trying to implement a so to say load balancing by reconstructing and decomposing every output time during an InterDyMFoam run with dynamicMeshRefinement based on alpha.

My problems are the reconstructParMesh and probably the decomposePar Utilities do not treat respective refinementHistory and pointLevel outputs for the dynamicRefineFvMesh class.

Does anyone have an Idea how to implement the necessary changes to these Utilities?

Edit: I just realized that instead of reconstruct+decomposepar one can just take redistributePar. But the Problem stays the same. I need the labelList pointlevel in the polymesh folder redistributed along with all the other Fields such as u or p hoping that the dynamicrefinemesh class doesnt refine already refined cells anylonger, without the need to have the possibilty of coarsening these cells again, as refinementHistory is a tough output to redistribute. I would greatly appreciate any input.

OK. I don't see the connection to PyFoam (that's what made me notice the thread). Probably you're using it for writing the driver-script.

Anyway: are you aware of the utility redistributeParMesh? Haven't used it myself, but in the source I read "Redistribute existing decomposed mesh ... " and "... balances mesh". That sounds like what you want to do in a single package

knalldi December 11, 2012 14:30

As I stated (unclearly, sorry about that) in my thread, the problem is that neither redistributePar, nor reconstructPar(Mesh) take the generated outputs of the dynamicRefineMesh class with them, namely pointLevel and refinementHistory, located in the processorX/polymesh folders. Without this information the automatic Mesh refinement starts again at given refined mesh, and refines it further, generating way to much cells each loop in phython.

My planned phython loop looks like that:

-decompose mesh

-solver solves to first outputtime with dynamic mesh refinement

-mesh reconstructed, endTime is increased

-decompose new Mesh with more cells -> loadbalancing

-solver solves to increased endtime .. and so forth

gschaider December 11, 2012 15:26

Quote:

Originally Posted by knalldi (Post 396984)
As I stated (unclearly, sorry about that) in my thread, the problem is that neither redistributePar, nor reconstructPar(Mesh) take the generated outputs of the dynamicRefineMesh class with them, namely pointLevel and refinementHistory, located in the processorX/polymesh folders. Without this information the automatic Mesh refinement starts again at given refined mesh, and refines it further, generating way to much cells each loop in phython.

My planned phython loop looks like that:

-decompose mesh

-solver solves to first outputtime with dynamic mesh refinement

-mesh reconstructed, endTime is increased

-decompose new Mesh with more cells -> loadbalancing

-solver solves to increased endtime .. and so forth

Those utilities only decompose/reconstruct proper GeometricFields. So what might be a possibility would be to "fill" those informations into appropriate geometric fields (volScalarField/pointScalarField). Then the reconstruct/decompose (or the resdistribute) will transfer them. Afterwards create the labelLists and go on

Or you take the sources of these utilities, create your own variant that reads and redistributes these labelListrs (at some point in these utilities there are lists available that describe the mappings of cells/points form/to parallel/serial (they're needed for distributing the GeometricField)

Or: you write your own dynamicMesh-class (by copy/paste/adapt from resdistributePar) that basically extends the one you're currently using to redistribute everything in memory from time to time

But I'm afraid all of these solutions involve some C++-programming (maybe someone here has a better idea)

knalldi December 14, 2012 07:12

Quote:

Originally Posted by gschaider (Post 396992)
Those utilities only decompose/reconstruct proper GeometricFields. So what might be a possibility would be to "fill" those informations into appropriate geometric fields (volScalarField/pointScalarField). Then the reconstruct/decompose (or the resdistribute) will transfer them. Afterwards create the labelLists and go on

Or you take the sources of these utilities, create your own variant that reads and redistributes these labelListrs (at some point in these utilities there are lists available that describe the mappings of cells/points form/to parallel/serial (they're needed for distributing the GeometricField)

Or: you write your own dynamicMesh-class (by copy/paste/adapt from resdistributePar) that basically extends the one you're currently using to redistribute everything in memory from time to time

But I'm afraid all of these solutions involve some C++-programming (maybe someone here has a better idea)

Thanks for your input. I guess sooner or later you have to program something, my hope was that this something already exists out there.

kmooney December 19, 2012 22:27

I'm currently working on extending load balancer to be able to map the refinementHistory from one distribution to another. I'll let you know if we get it working!

Here is some previous discussion on the matter:
http://www.cfd-online.com/Forums/ope...tion-maps.html

alpalbayrak August 6, 2015 05:40

Any news?
 
I am wondering if you managed to solve the issue. I read other similar threads without success. I am using version 2.3.x.

Summary of problem:
If you run a dynamically refined solver in parallel (example interdymfoam), then it is not possible to reconstruct the case properly. The reason is that the refinement associated files (cellLevel, pointLevel, refinementHistory) cannot be reconstructed by the command 'reconstructparMesh'. This leads to re-refinement of already refined cells, if solver runs from latest time.

Is there any steps that i am missing? I haven't tested the redistributepar command yet. But it would be silly, if we could redistribute the refined mesh properly, but not reconstruct it.

kmooney August 7, 2015 01:58

I understand there may be limitations to the code but why are you reconstructing your case only to decompose it again?

alpalbayrak August 7, 2015 08:19

I have also tested redistributepar, but it also fails to process refinement files.

I would like to reconstruct the case for several reasons;
If reconstruct process works fine, then I can decompose again for "manual" load balancing. Also I could change the number of processors.

Moreover, it would be easier to make changes in boundary conditions and internal fields on reconstructed case. One could easily add a new field. Also archiving becomes easier. But these are minor issues, since one can hopefully handle them with simple scripts.


All times are GMT -4. The time now is 16:17.