CFD Online Discussion Forums

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)
-   OpenFOAM Meshing & Mesh Conversion (https://www.cfd-online.com/Forums/openfoam-meshing/)
-   -   Load balancing with dynamic mesh (https://www.cfd-online.com/Forums/openfoam-meshing/103602-load-balancing-dynamic-mesh.html)

tikulju June 21, 2012 07:10

Load balancing with dynamic mesh
 
Hi!
I implemnetd mesh refinement from interDyMFoam to multiphaseEulerFoam and it seems to work well. But after a while I'm stuck with load balancing issues. Does anybody know, is there a way to redistribute the mesh during the simulation? Or if I have to code it, does anybody have idea where to start?

Thanks,
Timo

kmooney June 21, 2012 15:42

Quote:

Originally Posted by tikulju (Post 367622)
Hi!
I implemnetd mesh refinement from interDyMFoam to multiphaseEulerFoam and it seems to work well. But after a while I'm stuck with load balancing issues. Does anybody know, is there a way to redistribute the mesh during the simulation? Or if I have to code it, does anybody have idea where to start?

Thanks,
Timo

I believe that the fvMeshDistribute library might do it for you but its implementation might not be so straightforward. The source code is in the dynamicMesh sub directories.

tikulju June 25, 2012 01:23

Thanks Kyle, I'll try to start from that one.

- Timo

tikulju June 26, 2012 09:20

Quote:

Originally Posted by kmooney (Post 367722)
I believe that the fvMeshDistribute library might do it for you but its implementation might not be so straightforward. The source code is in the dynamicMesh sub directories.

I managed to do it somehow by applying redistributePar code. But the mesh is not the same as is the used in a simulation (it is read from the directory). Any idea how to operate with the mesh, which is used in the main code (i.e. dynamicFvMesh and apparently stored in memory)?

- Timo

tikulju June 28, 2012 06:11

Quote:

Originally Posted by tikulju (Post 368369)
I managed to do it somehow by applying redistributePar code. But the mesh is not the same as is the used in a simulation (it is read from the directory). Any idea how to operate with the mesh, which is used in the main code (i.e. dynamicFvMesh and apparently stored in memory)?

- Timo

Ok, I ended up using the redistributePar-code by modifying it a bit. Now my implementation involves writing and reading of mesh and fields to and from HD, but I couldn't make up any wiser way to do it. But now it works anyway :)

- Timo

kmooney October 2, 2012 16:32

Timo,

I am currently working in coupling interDyMFoam to fvMeshDistribute. I can properly load balance the mesh, however, the refinement engine seems to get confused about the old-to-new cell refinement levels after the balance.

Were you able to fix this?

tikulju October 3, 2012 05:27

Kyle,

I didn't get it actually to work. It hanged after repartition... With manual repartition the problem was the same. If I remember correctly, it had something to do with refinementHistory, which probably needs also to be updated. If you have any ideas how it could be fixed, I would be curious to know.

- Timo

kmooney October 3, 2012 09:13

I have not figured it out quite yet. The refinementHistory class as a ::distribute(mapDisributePolyMesh) member function, as does the hexRef8 class which actually calls refinementHistory::distribute(...).

I added some extra pointers to dyanmicFvMesh and hacked the redistributePar source to load the dynamic meshes. This way, after the load balance I can send the processor-processor map to the mesh cutter and related structures as such:

in redistributePar.C ~line 1000
Code:

    // Do actual sending/receiving of mesh
    autoPtr<mapDistributePolyMesh> map = distributor.distribute(finalDecomp);

    mesh.meshCutter().distribute(map());

I get a seg fault at line 282 in mapDistributeTemplates.C at
Code:

                    forAll(map, i)
                    {
                        subField[i] = field[map[i]];
                    }

The segfault occurs when its preparing to send the cellLevel integer lists to processor neighbors. This is getting a little deep into mpi comms for my programming skills so it will take me some time to proceed at this point.

I've attempted to use the cellLevel volScalarField that is dumped out for post processing but without the pointLevel field I don't think the hexRef8 engine knows what to do with it. I attempted to map a pointLevel pointScalarField but unfortunately I couldn't get it to map during redistribution.

wyldckat October 3, 2012 10:29

Greetings to all!

This bug report might interest you guys: http://www.openfoam.org/mantisbt/view.php?id=638
Try using the latest 2.1.x and see if you still have the same problem!

Best regards,
Bruno

kmooney October 3, 2012 10:42

Quote:

Originally Posted by wyldckat (Post 384737)
Greetings to all!

This bug report might interest you guys: http://www.openfoam.org/mantisbt/view.php?id=638
Try using the latest 2.1.x and see if you still have the same problem!

Best regards,
Bruno

Hi Bruno,

I am indeed using the latest and greatest 2.1.x repo version. This bug report contains some useful discussion on mapping, however, my problem surrounds load balancing which appears to demand quite a lot of subtle processor communication. I will try and gain some insight from it regardless. I appreciate your help!

Cheers,

Kyle

Xinze May 25, 2017 09:28

Big demand on the feature of dynamic load balance
 
I am not sure whether you have got the solution or any update for this problem. Recently I also encountered this neck of performance, especially using ~DyMFoam. So I want to active this thread again.

I find this codes but it seems exist some questions.
https://github.com/tgvoskuilen/meshBalancing
https://bugs.openfoam.org/view.php?id=2461

https://bugs.openfoam.org/view.php?id=638

If possible, anyone may update the latest progress on this problem.

XZ

kmooney May 25, 2017 12:25

Quote:

Originally Posted by Xinze (Post 650239)
I am not sure whether you have got the solution or any update for this problem. Recently I also encountered this neck of performance, especially using ~DyMFoam. So I want to active this thread again.

I find this codes but it seems exist some questions.
https://github.com/tgvoskuilen/meshBalancing
https://bugs.openfoam.org/view.php?id=2461

https://bugs.openfoam.org/view.php?id=638

If possible, anyone may update the latest progress on this problem.

XZ

The ptscotch dgraph error comes up quite a bit. If you switch to hierarchical its not the perfect decomp but its far more robust.

Xinze May 25, 2017 23:04

@kmooney
Thank you for you constructive information. I will try to adapt this trick.

Addition, there two clues deposit here and is possible useful for improving the feature:
(1)
https://bugs.openfoam.org/view.php?id=2461
PIMPLE: iteration 1
Selected 119 cells for refinement out of 115612.
Refined from 115612 to 116445 cells.
Selected 217 split points out of a possible 10399.
Unrefined from 116445 to 114926 cells.
Maximum imbalance = 27.2854 %
Re-balancing dynamically refined mesh
Selecting decompositionMethod ptscotch
(1): ERROR: dgraphFold2: out of memory (2)
(2): ERROR: dgraphFold2: out of memory (2)
(4): ERROR: dgraphFold2: out of memory (2)

This bug can be tracked on here:
https://github.com/poulson/scotch/bl.../dgraph_fold.c

fldvelolocnbr = (orggrafptr->veloloctax != NULL) ? fldvertlocnbr : 0;

if (memAllocGroup ((void **) (void *) /* Allocate distributed graph public data */
&fldgrafptr->vertloctax, (size_t) ((fldvertlocnbr + 1) * sizeof (Gnum)),
&fldgrafptr->vnumloctax, (size_t) ( fldvertlocnbr * sizeof (Gnum)),
&fldgrafptr->veloloctax, (size_t) ( fldvelolocnbr * sizeof (Gnum)), NULL) == NULL) {
errorPrint ("dgraphFold2: out of memory (2)");
cheklocval = 1;
}

(2) Alternatively, it maybe solve these problems if update to a new parallel partitioning libraray, such as Zoltan.

http://www.cs.sandia.gov/~kddevin/pa...dagstuhl09.pdf
http://www.cs.sandia.gov/zoltan/Zoltan.html

XZ


All times are GMT -4. The time now is 16:52.