CFD Online Logo CFD Online URL
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Pre-Processing

massively parallel AMI: decomposePar with singleProcessorFaceSets constraint

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By louisgag

LinkBack Thread Tools Search this Thread Display Modes
Old   April 9, 2020, 13:47
Default massively parallel AMI: decomposePar with singleProcessorFaceSets constraint
Senior Member
louisgag's Avatar
Louis Gagnon
Join Date: Mar 2009
Location: Stuttgart, Germany
Posts: 304
Rep Power: 14
louisgag is on a distinguished road
Send a message via ICQ to louisgag
I'm not able to get a balanced distribution with the automatic algorithms (scotch, kahip) when I have many patches pairs that are constrained to remain on the same processor.

I have a single AMI zone but It has more cells that what a balanced decomposition would give me. Both AMI patches together touch 44k cells but I would like to have 10k-14k cells per processor.

What I do is to slice the cylinder into more AMI patch pairs, convert them to sets, and then give those as constrained set in decomposeParDict.

When I leave the processor as -1, every set gets assigned to the same processor and I am back to having an unbalanced decomposition. I think it has to do with these lines of code:

When I impose different processors myself, taking for example 5 or 50 subslices of my AMI cylinder, I still end up unbalanced. Some processor will even have 0 cells!

Is my only option to go to manual decomposition? As an alternative I'm thinking of constraining the AMI to at least each remain on the same computer socket, using multilevel or multiregion decomposition, but I have the feeling that means a lot of hand tuning and wasted time.

Anyone has experience with AMI interfaces that have more cells that an optimal parallel simulation can take? Comments/advice would be appreciated.
louisgag is offline   Reply With Quote

Old   December 1, 2020, 03:17
New Member
Alexandra H
Join Date: May 2020
Posts: 2
Rep Power: 0
simply-alex is on a distinguished road
Hey Louis, did you ever find an effective solution to this issue? I am having the same problem.

All I can think of, is splitting the large cyclic patches into multiple smaller ones and grouping them into multiple faceSets
simply-alex is offline   Reply With Quote

Old   December 1, 2020, 04:32
Senior Member
louisgag's Avatar
Louis Gagnon
Join Date: Mar 2009
Location: Stuttgart, Germany
Posts: 304
Rep Power: 14
louisgag is on a distinguished road
Send a message via ICQ to louisgag
Hi Alexandra,

I did try what you mention (so making "slices" of the AMIs), it works to some extent, but I usually still end up with a struggling decomposition algorithm that gives a quite unbalanced decomposition, even away from the AMI patches. It was not helping parallelization speedup.

For my last attempts, I ended up doing multi-region decomposition:
1. split the domain into regions between outer, AMI-faces, and inner regions.
2. run a modified* decomposePar with the -cellDist and by regions, using for example scotch on the outer and inner regions and a simple, manual, or hierarchical decomposition on the faces of the AMIs.
3. use the resulting cell distribution to manually decompose your original mesh (the one that was not split into regions)
*: I had to write my own decomposePar utility to do it, because otherwise each region is decomposed using the same processors, but here you want to have, for example, region0 on procs[00-05], region1 on procs[06-07], region2 on procs[08-16], etc and if you do it using python on the created cellDist files it takes forever, but with the modified decomposePar utility its quite fast.

Just as a heads up, in the end I do not gain parallel speedup by doing so, the AMI algorithm is apparently not well parallelized and thus even when all the cells are on one or a few processors you still are slow.
I made a splash talk about this last summer, only images here, but the plot on the last slide is self explanatory (free decomp is when I did not do anything to maintain the AMI faces/cells on a single or set of processors)...

I was also hinted by Eugene de Villiers, that a precomputation of the AMI weights would help and was somehow already done by someone, but I never found the related code.

Kind regards,
simply-alex likes this.
louisgag is offline   Reply With Quote


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
[swak4Foam] reconstructPar error in MRFsimpleFoam (ami apprach) using GroovyBC velocity inlet Krao OpenFOAM Community Contributions 3 August 19, 2019 06:40
MPI error in parallel application usv001 OpenFOAM Programming & Development 2 September 14, 2017 12:30
chtMultiRegionSimpleFoam: crash on parallel run student666 OpenFOAM Running, Solving & CFD 3 April 20, 2017 12:05
snappyHexMesh in parallel with AMI louisgag OpenFOAM Pre-Processing 8 September 15, 2014 03:57
New sixDoFRigidBody BC working with laplaceFaceDecomposition Ya_Squall2010 OpenFOAM Running, Solving & CFD 13 April 17, 2013 03:04

All times are GMT -4. The time now is 06:58.