# DecomposePar unequal number of shared faces

 Register Blogs Members List Search Today's Posts Mark Forums Read

 November 24, 2005, 14:33 I'm decomposing a channel (wit #1 Senior Member   Maka Mohu Join Date: Mar 2009 Posts: 305 Rep Power: 9 I'm decomposing a channel (with periodic boundary conditions. The mesh is done blockMesh) that is 64x24x64. The decomposition takes place in the streamwise direction. I expected that for any processor i the no of faces shared with processor j is equal for all processors. wall boundary faces can off course differ in the number of cells. What I got is not normal. Note the shared no of cells is halfed when going from 2 to 3 processor. Can any body help me explain why this happens? Thanks in advance. 2 processors: Processor 0 Number of cells = 98304 Number of faces shared with processor 1 = 6144 Number of boundary faces = 7168 Processor 1 Number of cells = 98304 Number of faces shared with processor 0 = 6144 Number of boundary faces = 7168 -------------------------------------------------- 3 processors Processor 0 Number of cells = 65536 Number of faces shared with processor 1 = 3188 Number of faces shared with processor 2 = 3072 Number of boundary faces = 4746 Processor 1 Number of cells = 65536 Number of faces shared with processor 0 = 3188 Number of faces shared with processor 2 = 3188 Number of boundary faces = 4652 Processor 2 Number of cells = 65536 Number of faces shared with processor 1 = 3188 Number of faces shared with processor 0 = 3072 Number of boundary faces = 4746 -------------------------------------------------- 4 processors: Processor 0 Number of cells = 49152 Number of faces shared with processor 1 = 3072 Number of faces shared with processor 3 = 3072 Number of boundary faces = 3584 Processor 1 Number of cells = 49152 Number of faces shared with processor 0 = 3072 Number of faces shared with processor 2 = 3072 Number of boundary faces = 3584 Processor 2 Number of cells = 49152 Number of faces shared with processor 1 = 3072 Number of faces shared with processor 3 = 3072 Number of boundary faces = 3584 Processor 3 Number of cells = 49152 Number of faces shared with processor 2 = 3072 Number of faces shared with processor 0 = 3072 Number of boundary faces = 3584 -------------------------------------------------- 5 processors: Processor 0 Number of cells = 39322 Number of faces shared with processor 1 = 3188 Number of faces shared with processor 4 = 3072 Number of boundary faces = 2790 Processor 1 Number of cells = 39322 Number of faces shared with processor 0 = 3188 Number of faces shared with processor 2 = 3189 Number of boundary faces = 2791 Processor 2 Number of cells = 39322 Number of faces shared with processor 1 = 3189 Number of faces shared with processor 3 = 3188 Number of boundary faces = 2791 Processor 3 Number of cells = 39321 Number of faces shared with processor 2 = 3188 Number of faces shared with processor 4 = 3188 Number of boundary faces = 2790 Processor 4 Number of cells = 39321 Number of faces shared with processor 3 = 3188 Number of faces shared with processor 0 = 3072 Number of boundary faces = 2790 --------------------------------------------------

 November 24, 2005, 14:41 Because periodic boundaries be #2 Senior Member   Eugene de Villiers Join Date: Mar 2009 Posts: 725 Rep Power: 12 Because periodic boundaries become processor boundaries if they have to communicate between processors, i.e. the two sides of the cyclic boundary no longer reside on the same processor.

 November 25, 2005, 04:27 I think you refer to the cycli #3 Senior Member   Maka Mohu Join Date: Mar 2009 Posts: 305 Rep Power: 9 I think you refer to the cyclic boundary that is normal to the direction I decompose in (x). I can not see why other cyclic boundries does not belong to the same processor as long as they are parllel to the x direction, since they will be communicated within the same processor. To make my point more clear, I expected that, when decomposing in x direction, Number of faces shared with processor i = Ny*Nz = 24*64 = 1536 = constant (no matter how many domain cuts I have). Please detail more for me to understand. Thanks for your reply

 November 25, 2005, 07:03 Here is a 1d domain: |----- #4 Senior Member   Eugene de Villiers Join Date: Mar 2009 Posts: 725 Rep Power: 12 Here is a 1d domain: |--------| It contains 8 cells and 2 cyclic boundaries. Here is the same domain decomposed into 2: |----||----| Each domain now talks to the other domain through 2 boundary faces, the ones in the middle and the cyclics which now have become "processor" boundaries. Here is the same domain decomposed into 4: |--||--||--||--| As you can see each domain communicates with its neighbour through only a single shared face. This inlcudes the first and the last domains which talk to each other through the cyclic-processor boundaries. No matter how many decompositions you make after 2 the number of faces will remain constant. Two domains is different because the cyclics act as processor boundaries as well, doubling the number of interprocessor faces. For the channel flow example the boundaries in both the x and z directions are cyclics. As to the small variations in the number of shared faces at higher processor counts, this is due to tolerance issues that cause an unequal number of cells to be distributed. Try hierarchical instead of simple decomposition method and/or decrease the decomposition tolerance "delta". Clear?

 November 25, 2005, 09:07 Yes. Many thanks. #5 Senior Member   Maka Mohu Join Date: Mar 2009 Posts: 305 Rep Power: 9 Yes. Many thanks.

 May 23, 2007, 20:42 Hi All, I am having problem #6 New Member   marks Join Date: Mar 2009 Posts: 2 Rep Power: 0 Hi All, I am having problems running decomposePar with merged patchs. The trace is from a simple test case. This test case is just the icoFoam cavity with two blocks merged (it works fine with one processor). I am using OF 1.4. Does anyone have any ideas what my problem is? Thanks in advance Mark m3:run\$decomposePar . cavity-2blocks /*---------------------------------------------------------------------------*\ | ========= | | | \ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \ / O peration | Version: 1.4 | | \ / A nd | Web: http://www.openfoam.org | | \/ M anipulation | | \*---------------------------------------------------------------------------*/ Exec : decomposePar . cavity-2blocks Date : Jul 21 2007 Time : 17:12:48 Host : m3 PID : 8943 Root : /home/marks/OpenFOAM/marks-1.4/run Case : cavity-2blocks Nprocs : 1 Create time Time = 0 Create mesh Calculating distribution of cells Selecting decompositionMethod simple Finished decomposition in 0 s Calculating original mesh data Distributing cells to processors Distributing faces to processors Calculating processor boundary addressing Distributing points to processors Constructing processor meshes *** glibc detected *** decomposePar: malloc(): memory corruption: 0x0000000000843520 *** ======= Backtrace: ========= /lib64/libc.so.6[0x2af1aa1958fe] /lib64/libc.so.6[0x2af1aa197787] /lib64/libc.so.6(__libc_malloc+0x86)[0x2af1aa199386] /usr/lib64/libstdc++.so.6(_Znwm+0x1d)[0x2af1a9a80fcd] /usr/lib64/libstdc++.so.6(_Znam+0x9)[0x2af1a9a810e9] decomposePar(_ZN4Foam4ListIiEC1EiRKi+0x5c)[0x43e2bc] decomposePar[0x4d8f1f] decomposePar[0x43ffab] /lib64/libc.so.6(__libc_start_main+0xf4)[0x2af1aa146ae4] decomposePar(_ZN4Foam6fvMesh10readUpdateEv+0x169)[0x43ad49] ======= Memory map: ======== 00400000-00511000 r-xp 00000000 08:06 983061 /home/marks/OpenFOAM/OpenFOAM-1.4/applications/bin/linux64Gcc4DPOpt/decomposePar 00711000-00719000 r--p 00111000 08:06 983061 /home/marks/OpenFOAM/OpenFOAM-1.4/applications/bin/linux64Gcc4DPOpt/decomposePar 00719000-0071a000 rw-p 00119000 08:06 983061 /home/marks/OpenFOAM/OpenFOAM-1.4/applications/bin/linux64Gcc4DPOpt/decomposePar 0071a000-0086c000 rw-p 0071a000 00:00 0 [heap] 2af1a7b4a000-2af1a7b66000 r-xp 00000000 08:05 2002754 /lib64/ld-2.5.so 2af1a7b66000-2af1a7b68000 rw-p 2af1a7b66000 00:00 0 2af1a7d66000-2af1a7d68000 rw-p 0001c000 08:05 2002754 /lib64/ld-2.5.so 2af1a7d68000-2af1a8741000 r-xp 00000000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so 2af1a8741000-2af1a8940000 ---p 009d9000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so 2af1a8940000-2af1a8968000 r--p 009d8000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so 2af1a8968000-2af1a896f000 rw-p 00a00000 08:06 970185 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfiniteVolume.so 2af1a896f000-2af1a8973000 rw-p 2af1a896f000 00:00 0 2af1a8973000-2af1a8999000 r-xp 00000000 08:06 970182 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libdecompositionMethods.s o 2af1a8999000-2af1a8a98000 ---p 00026000 08:06 970182 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libdecompositionMethods.s o 2af1a8a98000-2af1a8a9a000 rw-p 00025000 08:06 970182 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libdecompositionMethods.s o 2af1a8a9a000-2af1a8a9c000 r-xp 00000000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so 2af1a8a9c000-2af1a8c9b000 ---p 00002000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so 2af1a8c9b000-2af1a8c9c000 r--p 00001000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so 2af1a8c9c000-2af1a8c9d000 rw-p 00002000 08:06 970179 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/liblagrangian.so 2af1a8c9d000-2af1a8c9e000 rw-p 2af1a8c9d000 00:00 0 2af1a8c9e000-2af1a8e16000 r-xp 00000000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so 2af1a8e16000-2af1a9015000 ---p 00178000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so 2af1a9015000-2af1a9019000 r--p 00177000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so 2af1a9019000-2af1a901c000 rw-p 0017b000 08:06 970191 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libmeshTools.so 2af1a901c000-2af1a901d000 rw-p 2af1a901c000 00:00 0 2af1a901d000-2af1a901e000 r-xp 00000000 08:06 970184 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfoamUtil.so 2af1a901e000-2af1a921d000 ---p 00001000 08:06 970184 /home/marks/OpenFOAM/OpenFOAM-1.4/lib/linux64Gcc4DPOpt/libfoamUtil.so 2af1a921d000-2af1a921e000 r--p 00000000 08:06 Aborted m3:run\$

 August 12, 2010, 09:01 #7 Member   Andrea Petronio Join Date: Mar 2009 Location: Trieste, Italy Posts: 38 Rep Power: 8 Hi Marks, hope you're already solved the problem, but maybe it's useful for others. I solved just by deleting non essential files as a cellZone and the sets folder. Andrea

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is OffTrackbacks are On Pingbacks are On Refbacks are On Forum Rules

 Similar Threads Thread Thread Starter Forum Replies Last Post derath OpenFOAM Native Meshers: blockMesh 9 July 18, 2012 05:01 jadavis1 OpenFOAM Running, Solving & CFD 0 January 28, 2009 16:07 jam OpenFOAM Running, Solving & CFD 1 January 20, 2008 14:13 Nathan FLUENT 1 April 26, 2006 11:15 Bogesz CFX 1 March 5, 2002 08:57

All times are GMT -4. The time now is 19:20.