CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Pre-Processing

DecomposePar unequal number of shared faces

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   November 24, 2005, 14:33
Default I'm decomposing a channel (wit
  #1
Senior Member
 
Maka Mohu
Join Date: Mar 2009
Posts: 305
Rep Power: 9
maka is on a distinguished road
I'm decomposing a channel (with periodic boundary conditions. The mesh is done blockMesh) that is 64x24x64. The decomposition takes place in the streamwise direction. I expected that for any processor i the no of faces shared with processor j is equal for all processors. wall boundary faces can off course differ in the number of cells.

What I got is not normal. Note the shared no of cells is halfed when going from 2 to 3 processor. Can any body help me explain why this happens? Thanks in advance.

2 processors:
Processor 0
Number of cells = 98304
Number of faces shared with processor 1 = 6144
Number of boundary faces = 7168
Processor 1
Number of cells = 98304
Number of faces shared with processor 0 = 6144
Number of boundary faces = 7168
--------------------------------------------------
3 processors
Processor 0
Number of cells = 65536
Number of faces shared with processor 1 = 3188
Number of faces shared with processor 2 = 3072
Number of boundary faces = 4746
Processor 1
Number of cells = 65536
Number of faces shared with processor 0 = 3188
Number of faces shared with processor 2 = 3188
Number of boundary faces = 4652
Processor 2
Number of cells = 65536
Number of faces shared with processor 1 = 3188
Number of faces shared with processor 0 = 3072
Number of boundary faces = 4746
--------------------------------------------------
4 processors:
Processor 0
Number of cells = 49152
Number of faces shared with processor 1 = 3072
Number of faces shared with processor 3 = 3072
Number of boundary faces = 3584
Processor 1
Number of cells = 49152
Number of faces shared with processor 0 = 3072
Number of faces shared with processor 2 = 3072
Number of boundary faces = 3584
Processor 2
Number of cells = 49152
Number of faces shared with processor 1 = 3072
Number of faces shared with processor 3 = 3072
Number of boundary faces = 3584
Processor 3
Number of cells = 49152
Number of faces shared with processor 2 = 3072
Number of faces shared with processor 0 = 3072
Number of boundary faces = 3584
--------------------------------------------------
5 processors:
Processor 0
Number of cells = 39322
Number of faces shared with processor 1 = 3188
Number of faces shared with processor 4 = 3072
Number of boundary faces = 2790

Processor 1
Number of cells = 39322
Number of faces shared with processor 0 = 3188
Number of faces shared with processor 2 = 3189
Number of boundary faces = 2791
Processor 2
Number of cells = 39322
Number of faces shared with processor 1 = 3189
Number of faces shared with processor 3 = 3188
Number of boundary faces = 2791
Processor 3
Number of cells = 39321
Number of faces shared with processor 2 = 3188
Number of faces shared with processor 4 = 3188
Number of boundary faces = 2790
Processor 4
Number of cells = 39321
Number of faces shared with processor 3 = 3188
Number of faces shared with processor 0 = 3072
Number of boundary faces = 2790

--------------------------------------------------
maka is offline   Reply With Quote

Old   November 24, 2005, 14:41
Default Because periodic boundaries be
  #2
Senior Member
 
Eugene de Villiers
Join Date: Mar 2009
Posts: 725
Rep Power: 12
eugene is on a distinguished road
Because periodic boundaries become processor boundaries if they have to communicate between processors, i.e. the two sides of the cyclic boundary no longer reside on the same processor.
eugene is offline   Reply With Quote

Old   November 25, 2005, 04:27
Default I think you refer to the cycli
  #3
Senior Member
 
Maka Mohu
Join Date: Mar 2009
Posts: 305
Rep Power: 9
maka is on a distinguished road
I think you refer to the cyclic boundary that is normal to the direction I decompose in (x). I can not see why other cyclic boundries does not belong to the same processor as long as they are parllel to the x direction, since they will be communicated within the same processor. To make my point more clear, I expected that, when decomposing in x direction, Number of faces shared with processor i = Ny*Nz = 24*64 = 1536 = constant (no matter how many domain cuts I have). Please detail more for me to understand. Thanks for your reply
maka is offline   Reply With Quote

Old   November 25, 2005, 07:03
Default Here is a 1d domain: |-----
  #4
Senior Member
 
Eugene de Villiers
Join Date: Mar 2009
Posts: 725
Rep Power: 12
eugene is on a distinguished road
Here is a 1d domain:

|--------|

It contains 8 cells and 2 cyclic boundaries.

Here is the same domain decomposed into 2:

|----||----|

Each domain now talks to the other domain through 2 boundary faces, the ones in the middle and the cyclics which now have become "processor" boundaries.

Here is the same domain decomposed into 4:

|--||--||--||--|

As you can see each domain communicates with its neighbour through only a single shared face. This inlcudes the first and the last domains which talk to each other through the cyclic-processor boundaries.

No matter how many decompositions you make after 2 the number of faces will remain constant. Two domains is different because the cyclics act as processor boundaries as well, doubling the number of interprocessor faces.

For the channel flow example the boundaries in both the x and z directions are cyclics.

As to the small variations in the number of shared faces at higher processor counts, this is due to tolerance issues that cause an unequal number of cells to be distributed. Try hierarchical instead of simple decomposition method and/or decrease the decomposition tolerance "delta".

Clear?
eugene is offline   Reply With Quote

Old   November 25, 2005, 09:07
Default Yes. Many thanks.
  #5
Senior Member
 
Maka Mohu
Join Date: Mar 2009
Posts: 305
Rep Power: 9
maka is on a distinguished road
Yes. Many thanks.
maka is offline   Reply With Quote

Old   May 23, 2007, 20:42
Default Hi All, I am having problem
  #6
New Member
 
marks
Join Date: Mar 2009
Posts: 2
Rep Power: 0
msang is on a distinguished road
Hi All,

I am having problems running decomposePar with merged patchs. The trace is from a simple test case. This test case is just the icoFoam cavity with two blocks merged (it works fine with one processor). I am using OF 1.4. Does anyone have any ideas what my problem is?

Thanks in advance

Mark

m3:run$decomposePar . cavity-2blocks
/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.4 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

Exec : decomposePar . cavity-2blocks
Date : Jul 21 2007
Time : 17:12:48
Host : m3
PID : 8943
Root : /home/marks/OpenFOAM/marks-1.4/run
Case : cavity-2blocks
Nprocs : 1
Create time

Time = 0
Create mesh


Calculating distribution of cells
Selecting decompositionMethod simple

Finished decomposition in 0 s

Calculating original mesh data

Distributing cells to processors

Distributing faces to processors

Calculating processor boundary addressing

Distributing points to processors

Constructing processor meshes
*** glibc detected *** decomposePar: malloc(): memory corruption: 0x0000000000843520 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2af1aa1958fe]
/lib64/libc.so.6[0x2af1aa197787]
/lib64/libc.so.6(__libc_malloc+0x86)[0x2af1aa199386]
/usr/lib64/libstdc++.so.6(_Znwm+0x1d)[0x2af1a9a80fcd]
/usr/lib64/libstdc++.so.6(_Znam+0x9)[0x2af1a9a810e9]
decomposePar(_ZN4Foam4ListIiEC1EiRKi+0x5c)[0x43e2bc]
decomposePar[0x4d8f1f]
decomposePar[0x43ffab]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2af1aa146ae4]
decomposePar(_ZN4Foam6fvMesh10readUpdateEv+0x169)[0x43ad49]
======= Memory map: ========
00400000-00511000 r-xp 00000000 08:06 983061 /home/marks/OpenFOAM/OpenFOAM-1.4/applications/bin/linux64Gcc4DPOpt/decomposePar
00711000-00719000 r--p 00111000 08:06 983061 /home/marks/OpenFOAM/OpenFOAM-1.4/applications/bin/linux64Gcc4DPOpt/decomposePar
00719000-0071a000 rw-p 00119000 08:06 983061 /home/marks/OpenFOAM/OpenFOAM-1.4/applications/bin/linux64Gcc4DPOpt/decomposePar
0071a000-0086c000 rw-p 0071a000 00:00 0 [heap]
2af1a7b4a000-2af1a7b66000 r-xp 00000000 08:05 2002754 /lib64/ld-2.5.so
2af1a7b66000-2af1a7b68000 rw-p 2af1a7b66000 00:00 0
2af1a7d66000-2af1a7d68000 rw-p 0001c000 08:05 2002754 /lib64/ld-2.5.so
2af1a7d68000-2af1a8741000 r-xp 00000000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so
2af1a8741000-2af1a8940000 ---p 009d9000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so
2af1a8940000-2af1a8968000 r--p 009d8000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so
2af1a8968000-2af1a896f000 rw-p 00a00000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so
2af1a896f000-2af1a8973000 rw-p 2af1a896f000 00:00 0
2af1a8973000-2af1a8999000 r-xp 00000000 08:06 970182 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libdecompositionMethods.s o
2af1a8999000-2af1a8a98000 ---p 00026000 08:06 970182 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libdecompositionMethods.s o
2af1a8a98000-2af1a8a9a000 rw-p 00025000 08:06 970182 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libdecompositionMethods.s o
2af1a8a9a000-2af1a8a9c000 r-xp 00000000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so
2af1a8a9c000-2af1a8c9b000 ---p 00002000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so
2af1a8c9b000-2af1a8c9c000 r--p 00001000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so
2af1a8c9c000-2af1a8c9d000 rw-p 00002000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so
2af1a8c9d000-2af1a8c9e000 rw-p 2af1a8c9d000 00:00 0
2af1a8c9e000-2af1a8e16000 r-xp 00000000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so
2af1a8e16000-2af1a9015000 ---p 00178000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so
2af1a9015000-2af1a9019000 r--p 00177000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so
2af1a9019000-2af1a901c000 rw-p 0017b000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so
2af1a901c000-2af1a901d000 rw-p 2af1a901c000 00:00 0
2af1a901d000-2af1a901e000 r-xp 00000000 08:06 970184 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfoamUtil.so
2af1a901e000-2af1a921d000 ---p 00001000 08:06 970184 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfoamUtil.so
2af1a921d000-2af1a921e000 r--p 00000000 08:06 Aborted
m3:run$
msang is offline   Reply With Quote

Old   August 12, 2010, 09:01
Default
  #7
Member
 
Andrea Petronio
Join Date: Mar 2009
Location: Trieste, Italy
Posts: 38
Rep Power: 8
andrea is on a distinguished road
Hi Marks,
hope you're already solved the problem, but maybe it's useful for others.
I solved just by deleting non essential files as a cellZone and the sets folder.

Andrea
andrea is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Inconsistent number of faces derath OpenFOAM Native Meshers: blockMesh 9 July 18, 2012 05:01
DecomposePar jadavis1 OpenFOAM Running, Solving & CFD 0 January 28, 2009 16:07
How to access the number of faces on a boundary jam OpenFOAM Running, Solving & CFD 1 January 20, 2008 14:13
Number of Faces on a thread inside a UDF Nathan FLUENT 1 April 26, 2006 11:15
Combining mesh fatal error: Unequal number of ... Bogesz CFX 1 March 5, 2002 08:57


All times are GMT -4. The time now is 22:36.