CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Pre-Processing

decomposition of multi domain cases for large simulations

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   July 3, 2013, 04:59
Default decomposition of multi domain cases for large simulations
  #1
Senior Member
 
romant's Avatar
 
Roman Thiele
Join Date: Aug 2009
Location: Stockholm, Sweden
Posts: 360
Rep Power: 12
romant is on a distinguished road
Hej,

the cases that we want to simulate contains three different domains, one fluid and two solid domains. The solid domains are rather small compared to the fluid domain, this is also true for the cell count. When I want to decompose the case over let's say 100 cores, then I automatically decompose also the solid domains over 100 cores, resulting in a large number of interfaces which needs to send data back and forth and the amount of cells on every core is very small (much less than the recommended 20k to 50k per core), slowing down the overall computations.

I was wondering if there is a way to decompose the water over the 100 cores and then for example decompose the solid domains on only 8 cores for example? Has anybody experience with decomposition of multi domain cases for large simulations?
__________________
~roman
romant is offline   Reply With Quote

Old   July 4, 2013, 03:50
Default
  #2
Member
 
Timo K.
Join Date: Feb 2010
Location: University of Stuttgart
Posts: 66
Rep Power: 7
timo_IHS is on a distinguished road
Hi Roman,

I do not have experience with multidomain cases, but you could try a manual decomposition. Make a decomposition with e.g. 100 and then reduce the amount of solids to e.g. 2.
timo_IHS is offline   Reply With Quote

Old   August 16, 2013, 03:24
Default
  #3
New Member
 
Klas J
Join Date: Oct 2011
Location: Göteborg, Sweden
Posts: 10
Rep Power: 5
KlasJ is on a distinguished road
Hi Roman,

Perhaps you have already solved this issue, but if not perhaps this could be of help.

My experience with multidomain cases is that it is much easier to use a manual decomposition. What I usually do is to produce a list of all the cells for all the regions in my simulation. Then I supply the lists (one per region) to the manual mode of decompasePar. The decomposeParDict would read:

Code:
numberOfSubdomains N;
method          manual;
manualCoeffs
{
        dataFile "cellDecomposition";
}
distributed no;
and the cellDecomposition would look something like:

Code:
FoamFile
{
    version     2.0;
    format      ascii;
    class       labelList;
    location    "constant";
    object      cellDecomposition;
}

ZZ
(
X
....
X
)
where ZZ are the number of cells for this region and X are the processor numbers where you want this cell to be placed.

Now, the interesting part is how you produce such lists. When the problems have very specific geometries or specific interfaces I often prefer to determine the processor boundaries somewhat manual. For example, if you have repeating geometry (like a set of fuel pins in a reactor) the point of automatic decomposition is much smaller.

I use a utility application to perform the splitting using topoSets. The application reads some splitting coordinates from a dictionary and then computes the processor for each of the cells. If you are interested I could share this small application.

Best regards,

Klas
KlasJ is offline   Reply With Quote

Old   August 16, 2013, 05:06
Default
  #4
Senior Member
 
romant's Avatar
 
Roman Thiele
Join Date: Aug 2009
Location: Stockholm, Sweden
Posts: 360
Rep Power: 12
romant is on a distinguished road
Quote:
Originally Posted by KlasJ View Post
Hi Roman,

Perhaps you have already solved this issue, but if not perhaps this could be of help.

My experience with multidomain cases is that it is much easier to use a manual decomposition. What I usually do is to produce a list of all the cells for all the regions in my simulation. Then I supply the lists (one per region) to the manual mode of decompasePar. The decomposeParDict would read:

Code:
numberOfSubdomains N;
method          manual;
manualCoeffs
{
        dataFile "cellDecomposition";
}
distributed no;
and the cellDecomposition would look something like:

Code:
FoamFile
{
    version     2.0;
    format      ascii;
    class       labelList;
    location    "constant";
    object      cellDecomposition;
}

ZZ
(
X
....
X
)
where ZZ are the number of cells for this region and X are the processor numbers where you want this cell to be placed.

Now, the interesting part is how you produce such lists. When the problems have very specific geometries or specific interfaces I often prefer to determine the processor boundaries somewhat manual. For example, if you have repeating geometry (like a set of fuel pins in a reactor) the point of automatic decomposition is much smaller.

I use a utility application to perform the splitting using topoSets. The application reads some splitting coordinates from a dictionary and then computes the processor for each of the cells. If you are interested I could share this small application.

Best regards,

Klas
Hej Klas,

I would be very much interested in that application if you could share here that would be very nice.
__________________
~roman
romant is offline   Reply With Quote

Old   October 7, 2013, 07:39
Default
  #5
Member
 
Join Date: May 2012
Posts: 55
Rep Power: 6
styleworker is on a distinguished road
I'm very interested in this application, too. Would you mind sharing this application?
styleworker is offline   Reply With Quote

Old   May 30, 2014, 07:13
Default RE: decomposition of multi domain cases for large simulations
  #6
New Member
 
Pedro
Join Date: Apr 2014
Posts: 1
Rep Power: 0
ipedrojm is on a distinguished road
Hello to you all,

I have a similar problem with OpenFOAM. I am working with chtMultiRegionFoam solver and everything works fine in parallel when I use the "same" number of processors for "each" region, e.g. 4 for the fluid region and 4 for the solid region and run the application with mpirun -n 4 chtMultiRegionFoam -parallel.

However, I am interested in running this solver with a different number of processors per region, e.g. 2 processors for the fluid region (left) and 1 processor for the solid region (right). I attached a figure in order to illustrate this: the fluid and solid regions are separated by a vertical line at the middle of the computational domain. Each colour designs a processor.

I decompose the computational domain as follows:

1) I use the "scotch" mode in the fluid region (left) with 2 processors with the -cellDist option. This creates a file "cellDecomposition" similar to the one described by Klas in "constant/fluid" directory, which is ready to be used with a "manual" decomposition method. The OF command is: decomposePar -region fluid -cellDist

2) I run the decomposePar command but now using the "manual" mode (using the cellDecomposition file created in previous step) and overwriting the previous decomposition. OF command: decomposePar -region fluid -force.

3) I repeat previous steps with the solid region (right), specifying only 1 processor for this region during step 1), and then specifying 2 processors in step 2), which gives a warning message as the second processor has 0 cells assigned. By doing this, I can run the solver using 2 processors and let 1 processor idle when solving for this region.

After doing that I can visualize the mesh, which gives the image I attached. I run the simulation using 2 processors, as the fluid (left) needs them, with the command: mpirun -n 2 chtMultiRegionFoam -parallel. However, the simulation fails because of MPI communications errors. This error makes sense to me as the chtMultiRegionFoam is solving "sequentially" (partitioned approach) each region and, therefore, if the user runs the application specifying 2 processors (using the mpirun command), each region "should" be using 2 processors...

Could you please, Klas and Roman or anyone else, help me with this issue? It seems that you do not encounter this problem in your cases. Probably I am missing something. Any help will be highly appreciated.

Regards,
Attached Images
File Type: jpg processors2.jpg (24.3 KB, 10 views)

Last edited by ipedrojm; May 30, 2014 at 07:20. Reason: Modification in the attached figure
ipedrojm is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Question about multi domain Rodrigo Basniak CFX 0 April 1, 2008 09:35
CFX Solver Memory Error mike CFX 1 March 19, 2008 08:22
Multi domain performance penalty???? Stu CFX 8 June 18, 2007 18:57
multi domain heat transfer prob mohammed zubair CFX 1 June 27, 2006 09:30
Domain decomposition rajesh Main CFD Forum 2 August 31, 1999 04:22


All times are GMT -4. The time now is 10:18.