CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Pre-Processing

How to run decomposePar with parMetis?

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   March 19, 2012, 03:40
Unhappy How to run decomposePar with parMetis?
  #1
New Member
 
xiaoweii
Join Date: Mar 2012
Posts: 7
Rep Power: 5
xiaow_g is on a distinguished road
parMetis is a parallel decompose method, that means we can decompose the mesh using several processes.
I just don't know how to do this?
After change the decomposeParDict, run "decomposePar" just executes in serial, and the command "mpirun -n 4 decomposePar -parallel" cannot work.
xiaow_g is offline   Reply With Quote

Old   April 5, 2012, 06:19
Default
  #2
Member
 
Martin
Join Date: Nov 2011
Posts: 30
Rep Power: 5
wernsen is on a distinguished road
Hello xiaow_g,

the decomposing is normally done after blockMesh to run snappyHexMesh in parallel (parallel meshing) or to decompose the mesh after meshing with snappyHexMesh to run the calculation in parallel (parallel calculation).

The decomposing needs only the decomposeParDict so that you can run it by simply type decomposePar in the terminal. For the next steps parallel Meshing / calculation you need to use the mpirun -np ... (snappyHexMesh/rhoSimpleFoam) -parallel ...

I hope my explanation is not to difficult
MARTIN
wernsen is offline   Reply With Quote

Old   April 5, 2012, 10:00
Default
  #3
New Member
 
xiaoweii
Join Date: Mar 2012
Posts: 7
Rep Power: 5
xiaow_g is on a distinguished road
Quote:
Originally Posted by wernsen View Post
Hello xiaow_g,

the decomposing is normally done after blockMesh to run snappyHexMesh in parallel (parallel meshing) or to decompose the mesh after meshing with snappyHexMesh to run the calculation in parallel (parallel calculation).

The decomposing needs only the decomposeParDict so that you can run it by simply type decomposePar in the terminal. For the next steps parallel Meshing / calculation you need to use the mpirun -np ... (snappyHexMesh/rhoSimpleFoam) -parallel ...

I hope my explanation is not to difficult
MARTIN
Hello,
I got your idea. You told me how to "parallel meshing" and "parallel calculation", but I am confused about how to "parallel decomposing". If running the decomposing in serial , I just need to run decomposePar in the terminal. but parMetis or ptScotch is parallel decomposing method, and the decomposing method can be set in the decomposeParDict.
xiaow_g is offline   Reply With Quote

Old   April 13, 2012, 08:06
Default
  #4
Member
 
Martin
Join Date: Nov 2011
Posts: 30
Rep Power: 5
wernsen is on a distinguished road
There is nothing like parallel decomposing. The only thing what the decomposer is doing is to decompose the mesh or field into processor-folders. This is accomplished in serial by running decomposePar. The methods how the field is beeing decomposed is described in the dictionary (hierarical, scotch, metis ...).
wernsen is offline   Reply With Quote

Old   April 13, 2012, 15:49
Default
  #5
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,318
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all!

Martin is right. The only thing that the ptscotch and parmetis methods are good for is for parallel mesh manipulation.

One example, already mentioned by Martin, is snappyHexMesh when executed in parallel, which allows mesh manipulation in parallel... although I don't know if cells are exchanged between processors in a load balancing system or if only the mesh structure is accessible between processors.

If you study the script "incompressible/simpleFoam/turbineSiting/Allrun", you'll notice exactly this: snappyHexMesh must use ptscotch...
... or in other words, it can't use scotch, because I think I've been able to use simple and hierarchical with snappyHexMesh...

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   October 1, 2012, 08:26
Default
  #6
Senior Member
 
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 259
Blog Entries: 5
Rep Power: 11
tomislav_maric is on a distinguished road
Quote:
Originally Posted by wyldckat View Post
Greetings to all!

Martin is right. The only thing that the ptscotch and parmetis methods are good for is for parallel mesh manipulation.

One example, already mentioned by Martin, is snappyHexMesh when executed in parallel, which allows mesh manipulation in parallel... although I don't know if cells are exchanged between processors in a load balancing system or if only the mesh structure is accessible between processors.

If you study the script "incompressible/simpleFoam/turbineSiting/Allrun", you'll notice exactly this: snappyHexMesh must use ptscotch...
... or in other words, it can't use scotch, because I think I've been able to use simple and hierarchical with snappyHexMesh...

Best regards,
Bruno
Hi,

I'm bumping this thread, because I need to use dynamic load balancing for the code I wrote. It's a two phase library with a geometrical transport of the volume fraction field, so the "action" is happening mostly near the interface. Is there a "recipe" for using dynamic load balancing in OpenFOAM in general? If there is none, can you at least name the ingredients?

How do I use ptscotch and parmetis together with the dynamicRefineFvMesh?

Tomislav
tomislav_maric is offline   Reply With Quote

Old   October 1, 2012, 17:06
Default
  #7
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,318
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Tomislav,

I think it's best that you ask in a new thread in the "Running" sub-forum, because that's more closely related to your question .

The closest I know about load balancing in OpenFOAM is the redistributePar application, but this would require you to stop the solver once in a while for redistribution and then running it again to continue. You can have a look in the folder "application/test" for ideas.

Last but not least: I suspect that said capability is possible in OpenFOAM, but probably it's not implemented yet.

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   October 2, 2012, 02:04
Default
  #8
Senior Member
 
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 259
Blog Entries: 5
Rep Power: 11
tomislav_maric is on a distinguished road
Quote:
Originally Posted by wyldckat View Post
Hi Tomislav,

I think it's best that you ask in a new thread in the "Running" sub-forum, because that's more closely related to your question .

The closest I know about load balancing in OpenFOAM is the redistributePar application, but this would require you to stop the solver once in a while for redistribution and then running it again to continue. You can have a look in the folder "application/test" for ideas.

Last but not least: I suspect that said capability is possible in OpenFOAM, but probably it's not implemented yet.

Best regards,
Bruno

Thanks, I will check it out.

Best regards,
Tomislav
tomislav_maric is offline   Reply With Quote

Old   October 2, 2012, 16:38
Default
  #9
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,318
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Tomislav,

Here you go, someone else is working on this subject as well: Load balancing with dynamic mesh
My guess is that you weren't lucky enough to find that thread in the first place...

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   October 4, 2012, 05:25
Default
  #10
Senior Member
 
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 259
Blog Entries: 5
Rep Power: 11
tomislav_maric is on a distinguished road
Quote:
Originally Posted by wyldckat View Post
Hi Tomislav,

Here you go, someone else is working on this subject as well: Load balancing with dynamic mesh
My guess is that you weren't lucky enough to find that thread in the first place...

Best regards,
Bruno
Thanks, we are already in contact with each other, it's time to bring the frustration to the Forum...
tomislav_maric is offline   Reply With Quote

Old   April 11, 2013, 16:45
Default
  #11
New Member
 
Anirban Jana
Join Date: Apr 2010
Location: Pittsburgh, PA, USA
Posts: 19
Rep Power: 7
jans is on a distinguished road
Hi All,
Shouldn't the function ParMETIS _V3_PartMeshKway function, for example, be able to do decomposition of the mesh in parallel? This is based on a parallel multi-level graph-partitioning algorithm. The paper describing this algorithm can be found here: http://www.cs.ucsb.edu/~gilbert/cs29...mlevel_mpi.pdf. ptScotch also has parallel graph partitioning routines. Hence it seems to me xiaoweii has a point. It should be possible to parallelize decomposePar itself.

It seems to me from the discussion in this thread though that currently only a serial version of decomposePar is available. Am I right? Is there at least some near term plan to parallelize decomposePar using these parallel graph partioning algorithms?

Thanks
Anirban

Last edited by jans; April 11, 2013 at 17:17.
jans is offline   Reply With Quote

Old   April 14, 2013, 07:16
Default
  #12
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,318
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Anirban,

AFAIK, as I've mentioned in one of the old posts:
  • decomposePar is meant to be used for initial decomposition, which is why it's executed in serial mode only.
  • redistributePar can be used in parallel and therefore it should be able to use "ptscotch" and similar algorithms.
If you check the file "applications/utilities/parallelProcessing/decomposePar/decomposeParDict" (see here online), you'll see that there is an interesting method named "multiLevel", which allows you to do multi-level decomposition.
Thinking a bit further on this last option, it gives the idea that we can do the following:
  • Firstly we decompose with the N partitions we want, but using the "simple" or "hierarchical" methods. Don't forget to run decomposePar.
  • Then we change in the file "system/decomposeParDict" from the method "simple" to "ptscotch".
  • Then run redistributePar in parallel:
    Code:
    foamJob -s -p redistributePar
    Or if you prefer the hardcore way of doing it (replace "N" with the number of cores):
    Code:
    mpirun -np N redistributePar -parallel
For more information on how "ptscotch" is being used, check the file: src/parallel/decompose/ptscotchDecomp/ptscotchDecomp.C


Best regards,Bruno
wyldckat is offline   Reply With Quote

Old   April 16, 2013, 18:14
Default
  #13
New Member
 
Anirban Jana
Join Date: Apr 2010
Location: Pittsburgh, PA, USA
Posts: 19
Rep Power: 7
jans is on a distinguished road
Hi Bruno,
Thanks much. Your suggestions are very interesting. I play with these and get back on what I discover.

I still do not entirely convinced why even the initial decomposition cannot be done in parallel. It seems like you are saying it is impossible algorithmically. I am not sure why. Oh well!

Best regards
Anirban
jans is offline   Reply With Quote

Old   April 17, 2013, 16:06
Default
  #14
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,318
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Anirban,

AFAIK, the current OpenFOAM framework contemplates only a few parallel mesh paradigms:
  • The mesh is loaded into a single process and decomposed into several sub-domains.
  • Each sub-domain is loaded by a single process (standard OpenFOAM parallel operations);
    • and may or may not exchange cells with another sub-domain (snappyHexMesh and redistributePar do this).
  • A single process reads each sub-domain and merges them into a single one, or it even manipulated each sub-domain on a single process.
Currently I'm not aware of any OpenFOAM application that runs in parallel and loads the same full mesh onto memory on all processes and then partition things among them.


And I think I forgot to mention in the previous posts a certain important aspect of how the official OpenFOAM version works: it only uses MPI for parallel processing. It does not use OpenMP nor is it prepared to work with multi-threaded processes.
The Extend variant of OpenFOAM on the other hand, is suppose to be able to work with multi-threaded capabilities, but I'm not familiar enough with it so I don't know what it can do with those multiple threads...


Now back on the multi-level topic: this option might be insinuating that the authors of OpenFOAM are already working on this kind of situation, namely a situation where they have hundreds of million cells in a mesh that need to be decomposed to thousands of cores, therefore the distribution has to be made in a multi-level manner, in order to propagate the mesh gradually throughout the whole system.
Nonetheless, I'm not aware of any specific documentation on this topic; and from what I know, it requires to start with at least with a seed decomposition of 2 sub-domains and then rely on redistributePar to take care of things and lead them to the thousands.


Either way, you do have some more possible ways to find more about this:
  • By looking at the source code;
  • By asking at the support section on openfoam.com
  • By making a feature request on the bug reporting section at openfoam.org
Best regards and please do share with us what you may find out on this topic!
Bruno
wyldckat is offline   Reply With Quote

Old   June 10, 2013, 14:35
Default
  #15
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,318
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all!

@Anirban: Apparently redistributePar can redistribute from one processor to multiple processors, as somewhat explained on the following thread: redistributePar do not interpolate properly on the processor boundaries

It's even implied in the header documentation: https://github.com/OpenFOAM/OpenFOAM...istributePar.C
Quote:
Code:
 Can also work like decomposePar:
\verbatim
# Create empty processor directories (have to exist for argList)
mkdir processor0
..
mkdir processorN

# Copy undecomposed polyMesh
cp -r constant processor0

# Distribute
mpirun -np ddd redistributePar -parallel
\endverbatim
Best regards,
Bruno
wyldckat is offline   Reply With Quote

Reply

Tags
decomposepar, parallel.ptscotch, parmetis

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Error trying to run steady-state sonicFoam dancfd OpenFOAM Running, Solving & CFD 2 February 12, 2013 04:15
SnappyHexMesh OF-1.6-ext crashes on a parallel run norman1981 OpenFOAM Bugs 5 December 7, 2011 13:48
how to map results from steady state run to tranient case? phsieh2005 OpenFOAM Running, Solving & CFD 2 November 7, 2010 07:20
How to run and save tranisent case on Cray XD1 Leon FLUENT 0 October 3, 2006 21:59
Windows 64-bit, Distributed Parallel Run Issues... Erich CFX 3 March 28, 2006 16:36


All times are GMT -4. The time now is 21:45.