CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Pre-Processing

Decomposing meshes

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By Tobi

Reply
 
LinkBack Thread Tools Display Modes
Old   September 8, 2014, 12:24
Default Decomposing meshes
  #1
Super Moderator
 
Tobi's Avatar
 
Tobias Holzmann
Join Date: Oct 2010
Location: Leoben (Austria)
Posts: 1,803
Blog Entries: 6
Rep Power: 32
Tobi has a spectacular aura aboutTobi has a spectacular aura about
Send a message via ICQ to Tobi Send a message via Skype™ to Tobi
Hi all,

I have a few questions to the decompose process.

The first question is about the decomposition output and general stuff:
  • First of all I think a main topic is to reduce the faces between the processors as good as you can but
  • on the other hand you should not have 1000 faces on processor1 and 2 faces on processor2 (if you are on processor3).
Is that correct? If I am right, then the following decomposition should be very very bad and decrease your speed extremly due to the fact of the bad face distribution (marked red)



Code:
Processor 0
    Number of cells = 79703
    Number of faces shared with processor 1 = 2347
    Number of faces shared with processor 3 = 1252
    Number of faces shared with processor 6 = 1983
    Number of faces shared with processor 7 = 4
    Number of faces shared with processor 9 = 6
    Number of processor patches = 5
    Number of processor faces = 5592
    Number of boundary faces = 13097

Processor 1
    Number of cells = 30927
    Number of faces shared with processor 0 = 2347
    Number of faces shared with processor 2 = 2691
    Number of faces shared with processor 3 = 5
    Number of faces shared with processor 4 = 605
    Number of faces shared with processor 5 = 1
    Number of faces shared with processor 6 = 6
    Number of faces shared with processor 7 = 1223
    Number of faces shared with processor 8 = 35
    Number of faces shared with processor 9 = 1
    Number of processor patches = 9
    Number of processor faces = 6914
    Number of boundary faces = 6206

Processor 2
    Number of cells = 98548
    Number of faces shared with processor 1 = 2691
    Number of faces shared with processor 4 = 6
    Number of faces shared with processor 5 = 1538
    Number of faces shared with processor 8 = 2181
    Number of faces shared with processor 11 = 1
    Number of processor patches = 5
    Number of processor faces = 6417
    Number of boundary faces = 17893

Processor 3
    Number of cells = 72934
    Number of faces shared with processor 0 = 1252
    Number of faces shared with processor 1 = 5
    Number of faces shared with processor 4 = 2949
    Number of faces shared with processor 6 = 1
    Number of faces shared with processor 9 = 1460
    Number of processor patches = 5
    Number of processor faces = 5667
    Number of boundary faces = 18218

Processor 4
    Number of cells = 45303
    Number of faces shared with processor 1 = 605
    Number of faces shared with processor 2 = 6
    Number of faces shared with processor 3 = 2949
    Number of faces shared with processor 5 = 3615
    Number of faces shared with processor 9 = 28
    Number of faces shared with processor 10 = 1574
    Number of faces shared with processor 11 = 24
    Number of processor patches = 7
    Number of processor faces = 8801
    Number of boundary faces = 8444

Processor 5
    Number of cells = 66574
    Number of faces shared with processor 1 = 1
    Number of faces shared with processor 2 = 1538
    Number of faces shared with processor 4 = 3615
    Number of faces shared with processor 10 = 5
    Number of faces shared with processor 11 = 1430
    Number of processor patches = 5
    Number of processor faces = 6589
    Number of boundary faces = 14245

Processor 6
    Number of cells = 38329
    Number of faces shared with processor 0 = 1983
    Number of faces shared with processor 1 = 6
    Number of faces shared with processor 3 = 1
    Number of faces shared with processor 7 = 1036
    Number of faces shared with processor 9 = 700
    Number of faces shared with processor 12 = 2172
    Number of faces shared with processor 13 = 2
    Number of faces shared with processor 15 = 8
    Number of processor patches = 8
    Number of processor faces = 5908
    Number of boundary faces = 6084

Processor 7
    Number of cells = 96970
    Number of faces shared with processor 0 = 4
    Number of faces shared with processor 1 = 1223
    Number of faces shared with processor 6 = 1036
    Number of faces shared with processor 8 = 1488
    Number of faces shared with processor 9 = 4
    Number of faces shared with processor 10 = 593
    Number of faces shared with processor 11 = 4
    Number of faces shared with processor 12 = 2
    Number of faces shared with processor 13 = 248
    Number of faces shared with processor 14 = 2
    Number of faces shared with processor 16 = 2
    Number of processor patches = 11
    Number of processor faces = 4606
    Number of boundary faces = 20630

Processor 8
    Number of cells = 45017
    Number of faces shared with processor 1 = 35
    Number of faces shared with processor 2 = 2181
    Number of faces shared with processor 7 = 1488
    Number of faces shared with processor 11 = 593
    Number of faces shared with processor 14 = 1197
    Number of faces shared with processor 17 = 2
    Number of processor patches = 6
    Number of processor faces = 5496
    Number of boundary faces = 8169

Processor 9
    Number of cells = 33585
    Number of faces shared with processor 0 = 6
    Number of faces shared with processor 1 = 1
    Number of faces shared with processor 3 = 1460
    Number of faces shared with processor 4 = 28
    Number of faces shared with processor 6 = 700
    Number of faces shared with processor 7 = 4
    Number of faces shared with processor 10 = 910
    Number of faces shared with processor 15 = 1436
    Number of faces shared with processor 16 = 5
    Number of processor patches = 9
    Number of processor faces = 4550
    Number of boundary faces = 7381

Processor 10
    Number of cells = 144720
    Number of faces shared with processor 4 = 1574
    Number of faces shared with processor 5 = 5
    Number of faces shared with processor 7 = 593
    Number of faces shared with processor 9 = 910
    Number of faces shared with processor 11 = 1497
    Number of faces shared with processor 15 = 2
    Number of faces shared with processor 16 = 575
    Number of faces shared with processor 17 = 4
    Number of processor patches = 8
    Number of processor faces = 5160
    Number of boundary faces = 29770

Processor 11
    Number of cells = 35368
    Number of faces shared with processor 2 = 1
    Number of faces shared with processor 4 = 24
    Number of faces shared with processor 5 = 1430
    Number of faces shared with processor 7 = 4
    Number of faces shared with processor 8 = 593
    Number of faces shared with processor 10 = 1497
    Number of faces shared with processor 16 = 6
    Number of faces shared with processor 17 = 1100
    Number of processor patches = 8
    Number of processor faces = 4655
    Number of boundary faces = 8624

Processor 12
    Number of cells = 97632
    Number of faces shared with processor 6 = 2172
    Number of faces shared with processor 7 = 2
    Number of faces shared with processor 13 = 3143
    Number of faces shared with processor 15 = 1410
    Number of processor patches = 4
    Number of processor faces = 6727
    Number of boundary faces = 20346

Processor 13
    Number of cells = 31151
    Number of faces shared with processor 6 = 2
    Number of faces shared with processor 7 = 248
    Number of faces shared with processor 12 = 3143
    Number of faces shared with processor 14 = 1952
    Number of faces shared with processor 15 = 7
    Number of faces shared with processor 16 = 682
    Number of faces shared with processor 17 = 3
    Number of processor patches = 7
    Number of processor faces = 6037
    Number of boundary faces = 6840

Processor 14
    Number of cells = 72706
    Number of faces shared with processor 7 = 2
    Number of faces shared with processor 8 = 1197
    Number of faces shared with processor 13 = 1952
    Number of faces shared with processor 17 = 1281
    Number of processor patches = 4
    Number of processor faces = 4432
    Number of boundary faces = 12889

Processor 15
    Number of cells = 71806
    Number of faces shared with processor 6 = 8
    Number of faces shared with processor 9 = 1436
    Number of faces shared with processor 10 = 2
    Number of faces shared with processor 12 = 1410
    Number of faces shared with processor 13 = 7
    Number of faces shared with processor 16 = 3801
    Number of processor patches = 6
    Number of processor faces = 6664
    Number of boundary faces = 17329

Processor 16
    Number of cells = 44918
    Number of faces shared with processor 7 = 2
    Number of faces shared with processor 9 = 5
    Number of faces shared with processor 10 = 575
    Number of faces shared with processor 11 = 6
    Number of faces shared with processor 13 = 682
    Number of faces shared with processor 15 = 3801
    Number of faces shared with processor 17 = 3951
    Number of processor patches = 7
    Number of processor faces = 9022
    Number of boundary faces = 8759

Processor 17
    Number of cells = 75775
    Number of faces shared with processor 8 = 2
    Number of faces shared with processor 10 = 4
    Number of faces shared with processor 11 = 1100
    Number of faces shared with processor 13 = 3
    Number of faces shared with processor 14 = 1281
    Number of faces shared with processor 16 = 3951
    Number of processor patches = 6
    Number of processor faces = 6341
    Number of boundary faces = 15767

Number of processor faces = 54789
Max number of cells = 144720 (120.392% above average 65664.8)
Max number of processor patches = 11 (65% above average 6.66667)
Max number of faces between processors = 9022 (48.2013% above average 6087.67)

Time = 0

The second question is about simple and hierarchical


  • it is known that hierarchical is only a extended simple algorithm
  • you can specify which axis has to decomposed first (e.g. zxy)
But what advantage do you have? If you decompose like (3 1 2) in zxy you split first z (2) then x (3) and y (1 = none). At least you have 6 domains.

If you split like (3 1 2) in xyz, you split first x (3) then y (1) and z (2). At least you also get 6 domains which should be (in my imagination) the same as above, or not?



The third question is about scotch and metis

  • Metis and scotch are decompose algorithm that attmpt to minimize the proc boundary faces between each core by using some special algorithm, aren't they?
  • Therefor you should be able to get a better (more balanced mesh) by using these algorithm instead of simple or hierarchical
Due to the fact that at the moment the metis lib is not available (if you do not compile it yourself) you have the ability to use scotch method. In the scotchDecomp.C file there are some hints how the algorithm is working but unfortunatelly I dont understand the things. Is there a paper, literature about this method?


At least one more questions

  • The more cores you have the more neigbourfaces you get. So if you have something like above (decompose output), could it be better to reduce the cores to get a better decomposition mesh? At least speed up the simulation because of better cpu communication?
  • How can you check out if your mesh is decomposed well or not
    • only with the decomposePar output?
    • What can i derivate of the green marked lines in the code block above?

Thanks in advance and for reading the topic.
Any hints and experiences are welcome.
makaveli_lcf likes this.
__________________
Keep foaming,
Tobias Holzmann
Tobi is offline   Reply With Quote

Old   September 9, 2014, 05:30
Default
  #2
Super Moderator
 
Tobi's Avatar
 
Tobias Holzmann
Join Date: Oct 2010
Location: Leoben (Austria)
Posts: 1,803
Blog Entries: 6
Rep Power: 32
Tobi has a spectacular aura aboutTobi has a spectacular aura about
Send a message via ICQ to Tobi Send a message via Skype™ to Tobi
Hi all,

today I made some tests with my college using decompose method hierarchical. Question two is not anymore necessary to explain. I miss understand something. For all who are interested here is the answer:

  • using hierarchical or simple split your mesh into regions with same amount of cells
  • I thought that the splitting is done in the center or due to the length of a geometry; e.g. if you have a pipe with a length of 2m then I thought with splitting of (2 1 1) (xyz), the mesh will be splitted in the middle at 1m. Thats correct if you have same amount of cells at each side. But if you have (e.g. a refinement region at one side) the mesh will not be splitted in the middle because of different cell amount. Its splitted so, that both processors will have same amount of cells.
__________________
Keep foaming,
Tobias Holzmann
Tobi is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Hex and Tet meshes - simplefoam comparison danvica OpenFOAM Running, Solving & CFD 10 January 4, 2013 02:18
Getting prism to inflate into mixed tet-hex meshes Joe CFX 16 October 10, 2011 07:06
HELP!! How could I obtain structured-orthogonal-body fitted meshes???? DajeMoo ANSYS 0 January 28, 2011 13:52
Dynamic Meshes Cfdtoy FLUENT 2 February 6, 2004 13:14
Large 3D tetrahedral meshes Aldo Bonfiglioli Main CFD Forum 4 August 27, 1999 03:33


All times are GMT -4. The time now is 20:29.