# decomposition of multi domain cases for large simulations

 Register Blogs Members List Search Today's Posts Mark Forums Read

 July 3, 2013, 04:59 decomposition of multi domain cases for large simulations #1 Senior Member     Roman Thiele Join Date: Aug 2009 Location: Stockholm, Sweden Posts: 360 Rep Power: 12 Hej, the cases that we want to simulate contains three different domains, one fluid and two solid domains. The solid domains are rather small compared to the fluid domain, this is also true for the cell count. When I want to decompose the case over let's say 100 cores, then I automatically decompose also the solid domains over 100 cores, resulting in a large number of interfaces which needs to send data back and forth and the amount of cells on every core is very small (much less than the recommended 20k to 50k per core), slowing down the overall computations. I was wondering if there is a way to decompose the water over the 100 cores and then for example decompose the solid domains on only 8 cores for example? Has anybody experience with decomposition of multi domain cases for large simulations? __________________ ~roman

 July 4, 2013, 03:50 #2 Member   Timo K. Join Date: Feb 2010 Location: University of Stuttgart Posts: 66 Rep Power: 7 Hi Roman, I do not have experience with multidomain cases, but you could try a manual decomposition. Make a decomposition with e.g. 100 and then reduce the amount of solids to e.g. 2.

 August 16, 2013, 03:24 #3 New Member   Klas J Join Date: Oct 2011 Location: Göteborg, Sweden Posts: 10 Rep Power: 5 Hi Roman, Perhaps you have already solved this issue, but if not perhaps this could be of help. My experience with multidomain cases is that it is much easier to use a manual decomposition. What I usually do is to produce a list of all the cells for all the regions in my simulation. Then I supply the lists (one per region) to the manual mode of decompasePar. The decomposeParDict would read: Code: ```numberOfSubdomains N; method manual; manualCoeffs { dataFile "cellDecomposition"; } distributed no;``` and the cellDecomposition would look something like: Code: ```FoamFile { version 2.0; format ascii; class labelList; location "constant"; object cellDecomposition; } ZZ ( X .... X )``` where ZZ are the number of cells for this region and X are the processor numbers where you want this cell to be placed. Now, the interesting part is how you produce such lists. When the problems have very specific geometries or specific interfaces I often prefer to determine the processor boundaries somewhat manual. For example, if you have repeating geometry (like a set of fuel pins in a reactor) the point of automatic decomposition is much smaller. I use a utility application to perform the splitting using topoSets. The application reads some splitting coordinates from a dictionary and then computes the processor for each of the cells. If you are interested I could share this small application. Best regards, Klas

August 16, 2013, 05:06
#4
Senior Member

Roman Thiele
Join Date: Aug 2009
Location: Stockholm, Sweden
Posts: 360
Rep Power: 12
Quote:
 Originally Posted by KlasJ Hi Roman, Perhaps you have already solved this issue, but if not perhaps this could be of help. My experience with multidomain cases is that it is much easier to use a manual decomposition. What I usually do is to produce a list of all the cells for all the regions in my simulation. Then I supply the lists (one per region) to the manual mode of decompasePar. The decomposeParDict would read: Code: ```numberOfSubdomains N; method manual; manualCoeffs { dataFile "cellDecomposition"; } distributed no;``` and the cellDecomposition would look something like: Code: ```FoamFile { version 2.0; format ascii; class labelList; location "constant"; object cellDecomposition; } ZZ ( X .... X )``` where ZZ are the number of cells for this region and X are the processor numbers where you want this cell to be placed. Now, the interesting part is how you produce such lists. When the problems have very specific geometries or specific interfaces I often prefer to determine the processor boundaries somewhat manual. For example, if you have repeating geometry (like a set of fuel pins in a reactor) the point of automatic decomposition is much smaller. I use a utility application to perform the splitting using topoSets. The application reads some splitting coordinates from a dictionary and then computes the processor for each of the cells. If you are interested I could share this small application. Best regards, Klas
Hej Klas,

I would be very much interested in that application if you could share here that would be very nice.
__________________
~roman

 October 7, 2013, 07:39 #5 Member   Join Date: May 2012 Posts: 55 Rep Power: 6 I'm very interested in this application, too. Would you mind sharing this application?

May 30, 2014, 07:13
RE: decomposition of multi domain cases for large simulations
#6
New Member

Pedro
Join Date: Apr 2014
Posts: 1
Rep Power: 0
Hello to you all,

I have a similar problem with OpenFOAM. I am working with chtMultiRegionFoam solver and everything works fine in parallel when I use the "same" number of processors for "each" region, e.g. 4 for the fluid region and 4 for the solid region and run the application with mpirun -n 4 chtMultiRegionFoam -parallel.

However, I am interested in running this solver with a different number of processors per region, e.g. 2 processors for the fluid region (left) and 1 processor for the solid region (right). I attached a figure in order to illustrate this: the fluid and solid regions are separated by a vertical line at the middle of the computational domain. Each colour designs a processor.

I decompose the computational domain as follows:

1) I use the "scotch" mode in the fluid region (left) with 2 processors with the -cellDist option. This creates a file "cellDecomposition" similar to the one described by Klas in "constant/fluid" directory, which is ready to be used with a "manual" decomposition method. The OF command is: decomposePar -region fluid -cellDist

2) I run the decomposePar command but now using the "manual" mode (using the cellDecomposition file created in previous step) and overwriting the previous decomposition. OF command: decomposePar -region fluid -force.

3) I repeat previous steps with the solid region (right), specifying only 1 processor for this region during step 1), and then specifying 2 processors in step 2), which gives a warning message as the second processor has 0 cells assigned. By doing this, I can run the solver using 2 processors and let 1 processor idle when solving for this region.

After doing that I can visualize the mesh, which gives the image I attached. I run the simulation using 2 processors, as the fluid (left) needs them, with the command: mpirun -n 2 chtMultiRegionFoam -parallel. However, the simulation fails because of MPI communications errors. This error makes sense to me as the chtMultiRegionFoam is solving "sequentially" (partitioned approach) each region and, therefore, if the user runs the application specifying 2 processors (using the mpirun command), each region "should" be using 2 processors...

Could you please, Klas and Roman or anyone else, help me with this issue? It seems that you do not encounter this problem in your cases. Probably I am missing something. Any help will be highly appreciated.

Regards,
Attached Images
 processors2.jpg (24.3 KB, 10 views)

Last edited by ipedrojm; May 30, 2014 at 07:20. Reason: Modification in the attached figure

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is OffTrackbacks are On Pingbacks are On Refbacks are On Forum Rules

 Similar Threads Thread Thread Starter Forum Replies Last Post Rodrigo Basniak CFX 0 April 1, 2008 09:35 mike CFX 1 March 19, 2008 08:22 Stu CFX 8 June 18, 2007 18:57 mohammed zubair CFX 1 June 27, 2006 09:30 rajesh Main CFD Forum 2 August 31, 1999 04:22

All times are GMT -4. The time now is 10:18.