CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Pre-Processing (http://www.cfd-online.com/Forums/openfoam-pre-processing/)
-   -   How to run decomposePar with parMetis? (http://www.cfd-online.com/Forums/openfoam-pre-processing/98777-how-run-decomposepar-parmetis.html)

xiaow_g March 19, 2012 03:40

How to run decomposePar with parMetis?
 
parMetis is a parallel decompose method, that means we can decompose the mesh using several processes.
I just don't know how to do this?
After change the decomposeParDict, run "decomposePar" just executes in serial, and the command "mpirun -n 4 decomposePar -parallel" cannot work.

wernsen April 5, 2012 06:19

Hello xiaow_g,

the decomposing is normally done after blockMesh to run snappyHexMesh in parallel (parallel meshing) or to decompose the mesh after meshing with snappyHexMesh to run the calculation in parallel (parallel calculation).

The decomposing needs only the decomposeParDict so that you can run it by simply type decomposePar in the terminal. For the next steps parallel Meshing / calculation you need to use the mpirun -np ... (snappyHexMesh/rhoSimpleFoam) -parallel ...

I hope my explanation is not to difficult :)
MARTIN

xiaow_g April 5, 2012 10:00

Quote:

Originally Posted by wernsen (Post 353273)
Hello xiaow_g,

the decomposing is normally done after blockMesh to run snappyHexMesh in parallel (parallel meshing) or to decompose the mesh after meshing with snappyHexMesh to run the calculation in parallel (parallel calculation).

The decomposing needs only the decomposeParDict so that you can run it by simply type decomposePar in the terminal. For the next steps parallel Meshing / calculation you need to use the mpirun -np ... (snappyHexMesh/rhoSimpleFoam) -parallel ...

I hope my explanation is not to difficult :)
MARTIN

Hello,
I got your idea. You told me how to "parallel meshing" and "parallel calculation", but I am confused about how to "parallel decomposing". If running the decomposing in serial , I just need to run decomposePar in the terminal. but parMetis or ptScotch is parallel decomposing method, and the decomposing method can be set in the decomposeParDict.

wernsen April 13, 2012 08:06

There is nothing like parallel decomposing. The only thing what the decomposer is doing is to decompose the mesh or field into processor-folders. This is accomplished in serial by running decomposePar. The methods how the field is beeing decomposed is described in the dictionary (hierarical, scotch, metis ...).

wyldckat April 13, 2012 15:49

Greetings to all!

Martin is right. The only thing that the ptscotch and parmetis methods are good for is for parallel mesh manipulation.

One example, already mentioned by Martin, is snappyHexMesh when executed in parallel, which allows mesh manipulation in parallel... although I don't know if cells are exchanged between processors in a load balancing system or if only the mesh structure is accessible between processors.

If you study the script "incompressible/simpleFoam/turbineSiting/Allrun", you'll notice exactly this: snappyHexMesh must use ptscotch...
... or in other words, it can't use scotch, because I think I've been able to use simple and hierarchical with snappyHexMesh...

Best regards,
Bruno

tomislav_maric October 1, 2012 08:26

Quote:

Originally Posted by wyldckat (Post 354615)
Greetings to all!

Martin is right. The only thing that the ptscotch and parmetis methods are good for is for parallel mesh manipulation.

One example, already mentioned by Martin, is snappyHexMesh when executed in parallel, which allows mesh manipulation in parallel... although I don't know if cells are exchanged between processors in a load balancing system or if only the mesh structure is accessible between processors.

If you study the script "incompressible/simpleFoam/turbineSiting/Allrun", you'll notice exactly this: snappyHexMesh must use ptscotch...
... or in other words, it can't use scotch, because I think I've been able to use simple and hierarchical with snappyHexMesh...

Best regards,
Bruno

Hi,

I'm bumping this thread, because I need to use dynamic load balancing for the code I wrote. It's a two phase library with a geometrical transport of the volume fraction field, so the "action" is happening mostly near the interface. Is there a "recipe" for using dynamic load balancing in OpenFOAM in general? If there is none, can you at least name the ingredients? :)

How do I use ptscotch and parmetis together with the dynamicRefineFvMesh?

Tomislav

wyldckat October 1, 2012 17:06

Hi Tomislav,

I think it's best that you ask in a new thread in the "Running" sub-forum, because that's more closely related to your question ;).

The closest I know about load balancing in OpenFOAM is the redistributePar application, but this would require you to stop the solver once in a while for redistribution and then running it again to continue. You can have a look in the folder "application/test" for ideas.

Last but not least: I suspect that said capability is possible in OpenFOAM, but probably it's not implemented yet.

Best regards,
Bruno

tomislav_maric October 2, 2012 02:04

Quote:

Originally Posted by wyldckat (Post 384424)
Hi Tomislav,

I think it's best that you ask in a new thread in the "Running" sub-forum, because that's more closely related to your question ;).

The closest I know about load balancing in OpenFOAM is the redistributePar application, but this would require you to stop the solver once in a while for redistribution and then running it again to continue. You can have a look in the folder "application/test" for ideas.

Last but not least: I suspect that said capability is possible in OpenFOAM, but probably it's not implemented yet.

Best regards,
Bruno


Thanks, I will check it out. :)

Best regards,
Tomislav

wyldckat October 2, 2012 16:38

Hi Tomislav,

Here you go, someone else is working on this subject as well: http://www.cfd-online.com/Forums/ope...tml#post384606
My guess is that you weren't lucky enough to find that thread in the first place...

Best regards,
Bruno

tomislav_maric October 4, 2012 05:25

Quote:

Originally Posted by wyldckat (Post 384607)
Hi Tomislav,

Here you go, someone else is working on this subject as well: http://www.cfd-online.com/Forums/ope...tml#post384606
My guess is that you weren't lucky enough to find that thread in the first place...

Best regards,
Bruno

Thanks, we are already in contact with each other, it's time to bring the frustration to the Forum...:D

jans April 11, 2013 16:45

Hi All,
Shouldn't the function ParMETIS _V3_PartMeshKway function, for example, be able to do decomposition of the mesh in parallel? This is based on a parallel multi-level graph-partitioning algorithm. The paper describing this algorithm can be found here: http://www.cs.ucsb.edu/~gilbert/cs29...mlevel_mpi.pdf. ptScotch also has parallel graph partitioning routines. Hence it seems to me xiaoweii has a point. It should be possible to parallelize decomposePar itself.

It seems to me from the discussion in this thread though that currently only a serial version of decomposePar is available. Am I right? Is there at least some near term plan to parallelize decomposePar using these parallel graph partioning algorithms?

Thanks
Anirban

wyldckat April 14, 2013 07:16

Greetings Anirban,

AFAIK, as I've mentioned in one of the old posts:
  • decomposePar is meant to be used for initial decomposition, which is why it's executed in serial mode only.
  • redistributePar can be used in parallel and therefore it should be able to use "ptscotch" and similar algorithms.
If you check the file "applications/utilities/parallelProcessing/decomposePar/decomposeParDict" (see here online), you'll see that there is an interesting method named "multiLevel", which allows you to do multi-level decomposition.
Thinking a bit further on this last option, it gives the idea that we can do the following:
  • Firstly we decompose with the N partitions we want, but using the "simple" or "hierarchical" methods. Don't forget to run decomposePar.
  • Then we change in the file "system/decomposeParDict" from the method "simple" to "ptscotch".
  • Then run redistributePar in parallel:
    Code:

    foamJob -s -p redistributePar
    Or if you prefer the hardcore way of doing it (replace "N" with the number of cores):
    Code:

    mpirun -np N redistributePar -parallel
For more information on how "ptscotch" is being used, check the file: src/parallel/decompose/ptscotchDecomp/ptscotchDecomp.C


Best regards,Bruno

jans April 16, 2013 18:14

Hi Bruno,
Thanks much. Your suggestions are very interesting. I play with these and get back on what I discover.

I still do not entirely convinced why even the initial decomposition cannot be done in parallel. It seems like you are saying it is impossible algorithmically. I am not sure why. Oh well!

Best regards
Anirban

wyldckat April 17, 2013 16:06

Hi Anirban,

AFAIK, the current OpenFOAM framework contemplates only a few parallel mesh paradigms:
  • The mesh is loaded into a single process and decomposed into several sub-domains.
  • Each sub-domain is loaded by a single process (standard OpenFOAM parallel operations);
    • and may or may not exchange cells with another sub-domain (snappyHexMesh and redistributePar do this).
  • A single process reads each sub-domain and merges them into a single one, or it even manipulated each sub-domain on a single process.
Currently I'm not aware of any OpenFOAM application that runs in parallel and loads the same full mesh onto memory on all processes and then partition things among them.


And I think I forgot to mention in the previous posts a certain important aspect of how the official OpenFOAM version works: it only uses MPI for parallel processing. It does not use OpenMP nor is it prepared to work with multi-threaded processes.
The Extend variant of OpenFOAM on the other hand, is suppose to be able to work with multi-threaded capabilities, but I'm not familiar enough with it so I don't know what it can do with those multiple threads...


Now back on the multi-level topic: this option might be insinuating that the authors of OpenFOAM are already working on this kind of situation, namely a situation where they have hundreds of million cells in a mesh that need to be decomposed to thousands of cores, therefore the distribution has to be made in a multi-level manner, in order to propagate the mesh gradually throughout the whole system.
Nonetheless, I'm not aware of any specific documentation on this topic; and from what I know, it requires to start with at least with a seed decomposition of 2 sub-domains and then rely on redistributePar to take care of things and lead them to the thousands.


Either way, you do have some more possible ways to find more about this:
  • By looking at the source code;
  • By asking at the support section on openfoam.com
  • By making a feature request on the bug reporting section at openfoam.org
Best regards and please do share with us what you may find out on this topic!
Bruno

wyldckat June 10, 2013 14:35

Greetings to all!

@Anirban: Apparently redistributePar can redistribute from one processor to multiple processors, as somewhat explained on the following thread: http://www.cfd-online.com/Forums/ope...oundaries.html

It's even implied in the header documentation: https://github.com/OpenFOAM/OpenFOAM...istributePar.C
Quote:

Code:

Can also work like decomposePar:
\verbatim
# Create empty processor directories (have to exist for argList)
mkdir processor0
..
mkdir processorN

# Copy undecomposed polyMesh
cp -r constant processor0

# Distribute
mpirun -np ddd redistributePar -parallel
\endverbatim


Best regards,
Bruno


All times are GMT -4. The time now is 15:52.