CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM

parallel decomposition method in openfoam

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree3Likes
  • 2 Post By cfdonline2mohsen
  • 1 Post By cfdonline2mohsen

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   March 20, 2013, 15:44
Question parallel decomposition method in openfoam
  #1
Senior Member
 
Join Date: Nov 2012
Posts: 168
Rep Power: 10
hz283 is on a distinguished road
Hi All,

In openfoam, the hierarchical method to split the computational domain gives the options to specify the order, e.g. xyz, does anybody know how this order affects the simulation results? Thanks.
hz283 is offline   Reply With Quote

Old   March 20, 2013, 16:29
Default
  #2
Senior Member
 
Lieven
Join Date: Dec 2011
Location: Leuven, Belgium
Posts: 299
Rep Power: 19
Lieven will become famous soon enough
Hi Hz283,

It could influence the simulation time, but that's basically it. It certainly does not affect the simulation results. At least it should not, else there is something wrong with the code.

Cheers,

L
Lieven is offline   Reply With Quote

Old   March 20, 2013, 17:57
Question
  #3
Senior Member
 
Join Date: Nov 2012
Posts: 168
Rep Power: 10
hz283 is on a distinguished road
Hi Lieven,

Thank you very much. Could you please say more about how it can affect the simulation time?

Thanks.

Quote:
Originally Posted by Lieven View Post
Hi Hz283,

It could influence the simulation time, but that's basically it. It certainly does not affect the simulation results. At least it should not, else there is something wrong with the code.

Cheers,

L
hz283 is offline   Reply With Quote

Old   March 21, 2013, 04:27
Default
  #4
Senior Member
 
Lieven
Join Date: Dec 2011
Location: Leuven, Belgium
Posts: 299
Rep Power: 19
Lieven will become famous soon enough
Sorry, not an expert in that field so I can't give you an exact answer to that.

Cheers,

L
Lieven is offline   Reply With Quote

Old   March 25, 2013, 05:52
Default
  #5
Senior Member
 
Join Date: Mar 2010
Location: Germany
Posts: 154
Rep Power: 12
cutter is on a distinguished road
Hi,

as far as I know, a general answer cannot be given. This substantially depends on your computational grid. Try to spread the work evenly on all processors/ cores. At the same time try to minimize the the interfaces of the mesh parts to reduce the communication between the nodes.

cutter
cutter is offline   Reply With Quote

Old   October 7, 2013, 17:16
Default
  #6
Senior Member
 
Join Date: Nov 2012
Posts: 168
Rep Power: 10
hz283 is on a distinguished road
Thank you very much for your suggestions. About the method hierarchical, I found that we can change the order of xyz to yzx or zyx and so on. I was wondering in which case I need to do this. Is this helpful to improve the computational efficiency?

Maybe the best way for me is to directly test the case with different partition method. But my case is a little large. So can I try get some suggestions from you before using this method?

Thank you in advance!
h

Quote:
Originally Posted by cutter View Post
Hi,

as far as I know, a general answer cannot be given. This substantially depends on your computational grid. Try to spread the work evenly on all processors/ cores. At the same time try to minimize the the interfaces of the mesh parts to reduce the communication between the nodes.

cutter
hz283 is offline   Reply With Quote

Old   October 7, 2013, 18:07
Default
  #7
Senior Member
 
cfdonline2mohsen's Avatar
 
Mohsen KiaMansouri
Join Date: Jan 2010
Location: CFD Lab
Posts: 118
Rep Power: 13
cfdonline2mohsen is on a distinguished road
Dear hz283
1) In my experience, one of the parameters that can affect your parallel simulation is the delta parameter which is equal to 0.001 by default. But if you have a mesh with faces of highly skewed cells you should increase this value to e.g. 0.01. you can find your cell skewness by running the following command in Terminal:
Code:
checkMesh -allGeometry
I remember that I had a simulation which worked just fine in a single processor but after decomposePar on several processors it would diverge (the courant number became very large). after increasing this delta parameter, it worked fine.
see the following post for more info:
http://www.cfd-online.com/Forums/ope...tml#post447958

2) Also there is another thing that you should notice. use the complex methods for decomposePar only if you have a complex mesh. but if you have a simple mesh, use the simpler methods. I have a experience in this field that by using complex methods (such as scotch) for my complex mesh (film cooling of turbine blades), my run time reduced to 1/3.
but by using scotch method for a simple mesh (flow around a cube) neither my run time nor the results did not change significantly in comparison with the simple method.
Summary: use complex methods for decomposePar only for complex meshes.
Also see the following post:
http://www.cfd-online.com/Forums/ope...-ptscotch.html

3) And finally about your 1st question:
I have not worked with hierarchical method and I don't know how to choose the order of xyz to yzx or zyx. but it definitely depends on your geometry.
Please share your experiences with hierarchical method in here.
hz283 and Ramzy1990 like this.
__________________
“If you have an apple and I have an apple and we exchange these apples then you and I will still each have one apple. But if you have an idea and I have an idea and we exchange these ideas, then each of us will have two ideas.”
cfdonline2mohsen is offline   Reply With Quote

Old   October 7, 2013, 18:41
Question
  #8
Senior Member
 
Join Date: Nov 2012
Posts: 168
Rep Power: 10
hz283 is on a distinguished road
Thank you so much for your so detailed reply. Besides, I found that the metis and scotch methods are intensively discussed in this forum. Most people think scotch is good. But I use OF211 and found that there is not such method. What kind of version are you using?

Thanks.

Quote:
Originally Posted by cfdonline2mohsen View Post
Dear hz283
1) In my experience, one of the parameters that can affect your parallel simulation is the delta parameter which is equal to 0.001 by default. But if you have a mesh with faces of highly skewed cells you should increase this value to e.g. 0.01. you can find your cell skewness by running the following command in Terminal:
Code:
checkMesh -allGeometry
I remember that I had a simulation which worked just fine in a single processor but after decomposePar on several processors it would diverge (the courant number became very large). after increasing this delta parameter, it worked fine.
see the following post for more info:
http://www.cfd-online.com/Forums/ope...tml#post447958

2) Also there is another thing that you should notice. use the complex methods for decomposePar only if you have a complex mesh. but if you have a simple mesh, use the simpler methods. I have a experience in this field that by using complex methods (such as scotch) for my complex mesh (film cooling of turbine blades), my run time reduced to 1/3.
but by using scotch method for a simple mesh (flow around a cube) neither my run time nor the results did not change significantly in comparison with the simple method.
Summary: use complex methods for decomposePar only for complex meshes.
Also see the following post:
http://www.cfd-online.com/Forums/ope...-ptscotch.html

3) And finally about your 1st question:
I have not worked with hierarchical method and I don't know how to choose the order of xyz to yzx or zyx. but it definitely depends on your geometry.
Please share your experiences with hierarchical method in here.
hz283 is offline   Reply With Quote

Old   October 7, 2013, 19:09
Default
  #9
Senior Member
 
cfdonline2mohsen's Avatar
 
Mohsen KiaMansouri
Join Date: Jan 2010
Location: CFD Lab
Posts: 118
Rep Power: 13
cfdonline2mohsen is on a distinguished road
Are you sure?
https://github.com/OpenFOAM/OpenFOAM...llel/decompose
hz283 likes this.
__________________
“If you have an apple and I have an apple and we exchange these apples then you and I will still each have one apple. But if you have an idea and I have an idea and we exchange these ideas, then each of us will have two ideas.”
cfdonline2mohsen is offline   Reply With Quote

Old   October 7, 2013, 19:11
Question
  #10
Senior Member
 
Join Date: Nov 2012
Posts: 168
Rep Power: 10
hz283 is on a distinguished road
Ah, Thanks!!

I only saw the option in the file system/decompose, in which only three options. Could you please share me one of this file in your case? Thank you a million!

Quote:
Originally Posted by cfdonline2mohsen View Post
hz283 is offline   Reply With Quote

Old   October 7, 2013, 19:53
Question
  #11
Senior Member
 
Join Date: Nov 2012
Posts: 168
Rep Power: 10
hz283 is on a distinguished road
Hi cfdonline2mohsen,

About the metis coefficients, if all the cores are totally the same, the coefficients for all the cores are only needed to be set to 1? Any comments about this point? Thank you very much!

Quote:
Originally Posted by cfdonline2mohsen View Post
hz283 is offline   Reply With Quote

Old   October 8, 2013, 05:35
Default
  #12
Senior Member
 
cfdonline2mohsen's Avatar
 
Mohsen KiaMansouri
Join Date: Jan 2010
Location: CFD Lab
Posts: 118
Rep Power: 13
cfdonline2mohsen is on a distinguished road
I don't recommend the metis method because if you review the cfd-online posts, most of them stated that metis has some restrictions about licensing problems.
It was just a recommendation. Search more about this matter.
I've used the scotch method with the same coefficients that exist in OpenFOAM tutorials.
__________________
“If you have an apple and I have an apple and we exchange these apples then you and I will still each have one apple. But if you have an idea and I have an idea and we exchange these ideas, then each of us will have two ideas.”
cfdonline2mohsen is offline   Reply With Quote

Old   January 8, 2021, 10:01
Default
  #13
New Member
 
Victor Vilceanu
Join Date: Jan 2021
Posts: 1
Rep Power: 0
VictorV66 is on a distinguished road
Quote:
Originally Posted by cfdonline2mohsen View Post
Dear hz283
1) In my experience, one of the parameters that can affect your parallel simulation is the delta parameter which is equal to 0.001 by default. But if you have a mesh with faces of highly skewed cells you should increase this value to e.g. 0.01. you can find your cell skewness by running the following command in Terminal:
Code:
checkMesh -allGeometry
I remember that I had a simulation which worked just fine in a single processor but after decomposePar on several processors it would diverge (the courant number became very large). after increasing this delta parameter, it worked fine.
see the following post for more info:
http://www.cfd-online.com/Forums/ope...tml#post447958

2) Also there is another thing that you should notice. use the complex methods for decomposePar only if you have a complex mesh. but if you have a simple mesh, use the simpler methods. I have a experience in this field that by using complex methods (such as scotch) for my complex mesh (film cooling of turbine blades), my run time reduced to 1/3.
but by using scotch method for a simple mesh (flow around a cube) neither my run time nor the results did not change significantly in comparison with the simple method.
Summary: use complex methods for decomposePar only for complex meshes.
Also see the following post:
http://www.cfd-online.com/Forums/ope...-ptscotch.html

3) And finally about your 1st question:
I have not worked with hierarchical method and I don't know how to choose the order of xyz to yzx or zyx. but it definitely depends on your geometry.
Please share your experiences with hierarchical method in here.
Hello cfdonline2mohsen,

I have read the comments, but I am still facing an issue for my complex mesh (by using scotch), when I am running the decomposePar (after running the blockMesh and checking if everything is fine). I have to admit that I changed the size cells in blockMesh, but still the same error. The same errors occur with other methods, for example, hierarchical. I am facing the following error:

--> FOAM FATAL IO ERROR:
size 65662 is not equal to the given value of 250000

file: /.../MyCase_!/0/Co from line 18 to line 65708.

From function Foam::Field<Type>::Field(const Foam::word&, const Foam::dictionary&, Foam::label) [with Type = double; Foam::label = int]
in file /home/ubuntu/OpenFOAM/OpenFOAM-8/src/OpenFOAM/lnInclude/Field.C at line 213.

FOAM exiting


Here's my decomposeParDict:

numberOfSubdomains 16;

method scotch;

Any solution to this or opinions?
VictorV66 is offline   Reply With Quote

Old   Today, 06:39
Default
  #14
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 1,109
Rep Power: 28
olesen will become famous soon enougholesen will become famous soon enough
Your Co field is likely left over from a previous simulation and after you generate a new mesh with blockMesh there is no reason to believe that this field is still valid (ie, correct size, number and size of boundaries). The error message is correct.
olesen is offline   Reply With Quote

Old   Today, 06:58
Default
  #15
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: http://olesenm.github.io/
Posts: 1,109
Rep Power: 28
olesen will become famous soon enougholesen will become famous soon enough
Licensing for metis has evolved of the years. It is no longer an issue. ParMetis still has some restrictions.
I did some tests a while back (2018 perhaps) to compare the min/max/avg number of neighbours between domains since this is a reasonably good metric for the decomposition quality and gives an idea of how well balanced the decomposition is. Metis did surprising well and was consistent. Scotch was quite similar, but you can hit some oddities very occasionally where the decomposition is just a bit 'weird' - probably related to the initial seeding. Kahip was also very similar but without the scotch oddness.
To quickly test things, use 'decomposePar -dry-run -cellDist'. This will perform the decomposition internally (does not create processor directories) obtain some statistics and then write out the distribution as a serial field, which you can visualize in paraview. If you want a bit more insight, compile the application/test/decomposePar. This creates a Test-decomposePar that let's you specify method and number of domains as command line options and also provides some better statistics. Use this to conviently (and quickly) test different approaches.
olesen is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
[mesh manipulation] Problem with RenumberMesh in parallel in OpenFOAM 2.1.1 srini_esi OpenFOAM Meshing & Mesh Conversion 1 November 8, 2013 03:48
OpenFOAM 2.0.0. and 2.0.1 doesn't work in parallel mode rv82 OpenFOAM Running, Solving & CFD 3 October 3, 2011 11:47
OpenFoam parallel crashes at random prapanj OpenFOAM Running, Solving & CFD 3 April 22, 2009 08:49
Parallel performance OpenFoam Vs Fluent prapanj Main CFD Forum 0 March 26, 2009 06:43
IcoFoam parallel woes msrinath80 OpenFOAM Running, Solving & CFD 9 July 22, 2007 03:58


All times are GMT -4. The time now is 20:16.