CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Meshing & Mesh Conversion

[Other] Load balancing with dynamic mesh

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree2Likes
  • 1 Post By kmooney
  • 1 Post By Xinze

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   June 21, 2012, 08:10
Default Load balancing with dynamic mesh
  #1
New Member
 
Timo Kulju
Join Date: Aug 2009
Posts: 22
Rep Power: 17
tikulju is on a distinguished road
Hi!
I implemnetd mesh refinement from interDyMFoam to multiphaseEulerFoam and it seems to work well. But after a while I'm stuck with load balancing issues. Does anybody know, is there a way to redistribute the mesh during the simulation? Or if I have to code it, does anybody have idea where to start?

Thanks,
Timo
tikulju is offline   Reply With Quote

Old   June 21, 2012, 16:42
Default
  #2
Senior Member
 
kmooney's Avatar
 
Kyle Mooney
Join Date: Jul 2009
Location: San Francisco, CA USA
Posts: 323
Rep Power: 18
kmooney is on a distinguished road
Quote:
Originally Posted by tikulju View Post
Hi!
I implemnetd mesh refinement from interDyMFoam to multiphaseEulerFoam and it seems to work well. But after a while I'm stuck with load balancing issues. Does anybody know, is there a way to redistribute the mesh during the simulation? Or if I have to code it, does anybody have idea where to start?

Thanks,
Timo
I believe that the fvMeshDistribute library might do it for you but its implementation might not be so straightforward. The source code is in the dynamicMesh sub directories.
kmooney is offline   Reply With Quote

Old   June 25, 2012, 02:23
Default
  #3
New Member
 
Timo Kulju
Join Date: Aug 2009
Posts: 22
Rep Power: 17
tikulju is on a distinguished road
Thanks Kyle, I'll try to start from that one.

- Timo
tikulju is offline   Reply With Quote

Old   June 26, 2012, 10:20
Default
  #4
New Member
 
Timo Kulju
Join Date: Aug 2009
Posts: 22
Rep Power: 17
tikulju is on a distinguished road
Quote:
Originally Posted by kmooney View Post
I believe that the fvMeshDistribute library might do it for you but its implementation might not be so straightforward. The source code is in the dynamicMesh sub directories.
I managed to do it somehow by applying redistributePar code. But the mesh is not the same as is the used in a simulation (it is read from the directory). Any idea how to operate with the mesh, which is used in the main code (i.e. dynamicFvMesh and apparently stored in memory)?

- Timo
tikulju is offline   Reply With Quote

Old   June 28, 2012, 07:11
Default
  #5
New Member
 
Timo Kulju
Join Date: Aug 2009
Posts: 22
Rep Power: 17
tikulju is on a distinguished road
Quote:
Originally Posted by tikulju View Post
I managed to do it somehow by applying redistributePar code. But the mesh is not the same as is the used in a simulation (it is read from the directory). Any idea how to operate with the mesh, which is used in the main code (i.e. dynamicFvMesh and apparently stored in memory)?

- Timo
Ok, I ended up using the redistributePar-code by modifying it a bit. Now my implementation involves writing and reading of mesh and fields to and from HD, but I couldn't make up any wiser way to do it. But now it works anyway

- Timo
tikulju is offline   Reply With Quote

Old   October 2, 2012, 17:32
Default
  #6
Senior Member
 
kmooney's Avatar
 
Kyle Mooney
Join Date: Jul 2009
Location: San Francisco, CA USA
Posts: 323
Rep Power: 18
kmooney is on a distinguished road
Timo,

I am currently working in coupling interDyMFoam to fvMeshDistribute. I can properly load balance the mesh, however, the refinement engine seems to get confused about the old-to-new cell refinement levels after the balance.

Were you able to fix this?
kmooney is offline   Reply With Quote

Old   October 3, 2012, 06:27
Default
  #7
New Member
 
Timo Kulju
Join Date: Aug 2009
Posts: 22
Rep Power: 17
tikulju is on a distinguished road
Kyle,

I didn't get it actually to work. It hanged after repartition... With manual repartition the problem was the same. If I remember correctly, it had something to do with refinementHistory, which probably needs also to be updated. If you have any ideas how it could be fixed, I would be curious to know.

- Timo
tikulju is offline   Reply With Quote

Old   October 3, 2012, 10:13
Default
  #8
Senior Member
 
kmooney's Avatar
 
Kyle Mooney
Join Date: Jul 2009
Location: San Francisco, CA USA
Posts: 323
Rep Power: 18
kmooney is on a distinguished road
I have not figured it out quite yet. The refinementHistory class as a ::distribute(mapDisributePolyMesh) member function, as does the hexRef8 class which actually calls refinementHistory::distribute(...).

I added some extra pointers to dyanmicFvMesh and hacked the redistributePar source to load the dynamic meshes. This way, after the load balance I can send the processor-processor map to the mesh cutter and related structures as such:

in redistributePar.C ~line 1000
Code:
    // Do actual sending/receiving of mesh
    autoPtr<mapDistributePolyMesh> map = distributor.distribute(finalDecomp);

    mesh.meshCutter().distribute(map());
I get a seg fault at line 282 in mapDistributeTemplates.C at
Code:
                    forAll(map, i)
                    {
                        subField[i] = field[map[i]];
                    }
The segfault occurs when its preparing to send the cellLevel integer lists to processor neighbors. This is getting a little deep into mpi comms for my programming skills so it will take me some time to proceed at this point.

I've attempted to use the cellLevel volScalarField that is dumped out for post processing but without the pointLevel field I don't think the hexRef8 engine knows what to do with it. I attempted to map a pointLevel pointScalarField but unfortunately I couldn't get it to map during redistribution.
hua1015 likes this.
kmooney is offline   Reply With Quote

Old   October 3, 2012, 11:29
Default
  #9
Retired Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,981
Blog Entries: 45
Rep Power: 128
wyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to all
Greetings to all!

This bug report might interest you guys: http://www.openfoam.org/mantisbt/view.php?id=638
Try using the latest 2.1.x and see if you still have the same problem!

Best regards,
Bruno
__________________
wyldckat is offline   Reply With Quote

Old   October 3, 2012, 11:42
Default
  #10
Senior Member
 
kmooney's Avatar
 
Kyle Mooney
Join Date: Jul 2009
Location: San Francisco, CA USA
Posts: 323
Rep Power: 18
kmooney is on a distinguished road
Quote:
Originally Posted by wyldckat View Post
Greetings to all!

This bug report might interest you guys: http://www.openfoam.org/mantisbt/view.php?id=638
Try using the latest 2.1.x and see if you still have the same problem!

Best regards,
Bruno
Hi Bruno,

I am indeed using the latest and greatest 2.1.x repo version. This bug report contains some useful discussion on mapping, however, my problem surrounds load balancing which appears to demand quite a lot of subtle processor communication. I will try and gain some insight from it regardless. I appreciate your help!

Cheers,

Kyle
kmooney is offline   Reply With Quote

Old   May 25, 2017, 10:28
Default Big demand on the feature of dynamic load balance
  #11
New Member
 
Xinze
Join Date: Mar 2009
Location: Shenyang, China
Posts: 15
Rep Power: 17
Xinze is on a distinguished road
I am not sure whether you have got the solution or any update for this problem. Recently I also encountered this neck of performance, especially using ~DyMFoam. So I want to active this thread again.

I find this codes but it seems exist some questions.
https://github.com/tgvoskuilen/meshBalancing
https://bugs.openfoam.org/view.php?id=2461

https://bugs.openfoam.org/view.php?id=638

If possible, anyone may update the latest progress on this problem.

XZ
iLyrics likes this.
Xinze is offline   Reply With Quote

Old   May 25, 2017, 13:25
Default
  #12
Senior Member
 
kmooney's Avatar
 
Kyle Mooney
Join Date: Jul 2009
Location: San Francisco, CA USA
Posts: 323
Rep Power: 18
kmooney is on a distinguished road
Quote:
Originally Posted by Xinze View Post
I am not sure whether you have got the solution or any update for this problem. Recently I also encountered this neck of performance, especially using ~DyMFoam. So I want to active this thread again.

I find this codes but it seems exist some questions.
https://github.com/tgvoskuilen/meshBalancing
https://bugs.openfoam.org/view.php?id=2461

https://bugs.openfoam.org/view.php?id=638

If possible, anyone may update the latest progress on this problem.

XZ
The ptscotch dgraph error comes up quite a bit. If you switch to hierarchical its not the perfect decomp but its far more robust.
kmooney is offline   Reply With Quote

Old   May 26, 2017, 00:04
Default
  #13
New Member
 
Xinze
Join Date: Mar 2009
Location: Shenyang, China
Posts: 15
Rep Power: 17
Xinze is on a distinguished road
@kmooney
Thank you for you constructive information. I will try to adapt this trick.

Addition, there two clues deposit here and is possible useful for improving the feature:
(1)
https://bugs.openfoam.org/view.php?id=2461
PIMPLE: iteration 1
Selected 119 cells for refinement out of 115612.
Refined from 115612 to 116445 cells.
Selected 217 split points out of a possible 10399.
Unrefined from 116445 to 114926 cells.
Maximum imbalance = 27.2854 %
Re-balancing dynamically refined mesh
Selecting decompositionMethod ptscotch
(1): ERROR: dgraphFold2: out of memory (2)
(2): ERROR: dgraphFold2: out of memory (2)
(4): ERROR: dgraphFold2: out of memory (2)

This bug can be tracked on here:
https://github.com/poulson/scotch/bl.../dgraph_fold.c

fldvelolocnbr = (orggrafptr->veloloctax != NULL) ? fldvertlocnbr : 0;

if (memAllocGroup ((void **) (void *) /* Allocate distributed graph public data */
&fldgrafptr->vertloctax, (size_t) ((fldvertlocnbr + 1) * sizeof (Gnum)),
&fldgrafptr->vnumloctax, (size_t) ( fldvertlocnbr * sizeof (Gnum)),
&fldgrafptr->veloloctax, (size_t) ( fldvelolocnbr * sizeof (Gnum)), NULL) == NULL) {
errorPrint ("dgraphFold2: out of memory (2)");
cheklocval = 1;
}

(2) Alternatively, it maybe solve these problems if update to a new parallel partitioning libraray, such as Zoltan.

http://www.cs.sandia.gov/~kddevin/pa...dagstuhl09.pdf
http://www.cs.sandia.gov/zoltan/Zoltan.html

XZ
Xinze is offline   Reply With Quote

Reply

Tags
dynamic mesh, load balance, parallel, refine mesh

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Dynamic Mesh kennyboy FLUENT 1 February 23, 2019 02:52
Local mesh refinement definition in a DEFORMING dynamic mesh zone using Dynamic Mesh Emanuele88 FLUENT 0 February 9, 2016 12:39
dynamic mesh for drop interface IndrajitW FLUENT 0 March 30, 2013 09:03
dynamic mesh refinement and rhoCentralFoam ChrisA OpenFOAM Running, Solving & CFD 1 March 21, 2013 09:00
Dynamic Mesh moving interface help akash.iitb FLUENT 0 August 24, 2010 00:53


All times are GMT -4. The time now is 04:58.