[xiaow_g]

parMetis is a parallel decompose method, that means we can decompose the mesh using several processes.
I just don't know how to do this?
After change the decomposeParDict, run "decomposePar" just executes in serial, and the command "mpirun -n 4 decomposePar -parallel" cannot work.

[wernsen]

Hello xiaow_g,

the decomposing is normally done after blockMesh to run snappyHexMesh in parallel (parallel meshing) or to decompose the mesh after meshing with snappyHexMesh to run the calculation in parallel (parallel calculation).

The decomposing needs only the decomposeParDict so that you can run it by simply type decomposePar in the terminal. For the next steps parallel Meshing / calculation you need to use the mpirun -np ... (snappyHexMesh/rhoSimpleFoam) -parallel ...

I hope my explanation is not to difficult
MARTIN

[xiaow_g]

Originally Posted by wernsen

Hello xiaow_g,

the decomposing is normally done after blockMesh to run snappyHexMesh in parallel (parallel meshing) or to decompose the mesh after meshing with snappyHexMesh to run the calculation in parallel (parallel calculation).

The decomposing needs only the decomposeParDict so that you can run it by simply type decomposePar in the terminal. For the next steps parallel Meshing / calculation you need to use the mpirun -np ... (snappyHexMesh/rhoSimpleFoam) -parallel ...

I hope my explanation is not to difficult
MARTIN

Hello,
I got your idea. You told me how to "parallel meshing" and "parallel calculation", but I am confused about how to "parallel decomposing". If running the decomposing in serial , I just need to run decomposePar in the terminal. but parMetis or ptScotch is parallel decomposing method, and the decomposing method can be set in the decomposeParDict.

[wernsen]

There is nothing like parallel decomposing. The only thing what the decomposer is doing is to decompose the mesh or field into processor-folders. This is accomplished in serial by running decomposePar. The methods how the field is beeing decomposed is described in the dictionary (hierarical, scotch, metis ...).

[wyldckat]

Greetings to all!

Martin is right. The only thing that the ptscotch and parmetis methods are good for is for parallel mesh manipulation.

One example, already mentioned by Martin, is snappyHexMesh when executed in parallel, which allows mesh manipulation in parallel... although I don't know if cells are exchanged between processors in a load balancing system or if only the mesh structure is accessible between processors.

If you study the script "incompressible/simpleFoam/turbineSiting/Allrun", you'll notice exactly this: snappyHexMesh must use ptscotch...
... or in other words, it can't use scotch, because I think I've been able to use simple and hierarchical with snappyHexMesh...

Best regards,
Bruno

[tomislav_maric]

Originally Posted by wyldckat

Greetings to all!

Martin is right. The only thing that the ptscotch and parmetis methods are good for is for parallel mesh manipulation.

One example, already mentioned by Martin, is snappyHexMesh when executed in parallel, which allows mesh manipulation in parallel... although I don't know if cells are exchanged between processors in a load balancing system or if only the mesh structure is accessible between processors.

If you study the script "incompressible/simpleFoam/turbineSiting/Allrun", you'll notice exactly this: snappyHexMesh must use ptscotch...
... or in other words, it can't use scotch, because I think I've been able to use simple and hierarchical with snappyHexMesh...

Best regards,
Bruno

Hi,

I'm bumping this thread, because I need to use dynamic load balancing for the code I wrote. It's a two phase library with a geometrical transport of the volume fraction field, so the "action" is happening mostly near the interface. Is there a "recipe" for using dynamic load balancing in OpenFOAM in general? If there is none, can you at least name the ingredients?

How do I use ptscotch and parmetis together with the dynamicRefineFvMesh?

Tomislav

[wyldckat]

Hi Tomislav,

I think it's best that you ask in a new thread in the "Running" sub-forum, because that's more closely related to your question .

The closest I know about load balancing in OpenFOAM is the redistributePar application, but this would require you to stop the solver once in a while for redistribution and then running it again to continue. You can have a look in the folder "application/test" for ideas.

Last but not least: I suspect that said capability is possible in OpenFOAM, but probably it's not implemented yet.

Best regards,
Bruno

[tomislav_maric]

Originally Posted by wyldckat

Hi Tomislav,

I think it's best that you ask in a new thread in the "Running" sub-forum, because that's more closely related to your question .

The closest I know about load balancing in OpenFOAM is the redistributePar application, but this would require you to stop the solver once in a while for redistribution and then running it again to continue. You can have a look in the folder "application/test" for ideas.

Last but not least: I suspect that said capability is possible in OpenFOAM, but probably it's not implemented yet.

Best regards,
Bruno

Thanks, I will check it out.

Best regards,
Tomislav

[wyldckat]

Hi Tomislav,

Here you go, someone else is working on this subject as well: http://www.cfd-online.com/Forums/ope...tml#post384606
My guess is that you weren't lucky enough to find that thread in the first place...

Best regards,
Bruno

[tomislav_maric]

Originally Posted by wyldckat

Hi Tomislav,

Here you go, someone else is working on this subject as well: http://www.cfd-online.com/Forums/ope...tml#post384606
My guess is that you weren't lucky enough to find that thread in the first place...

Best regards,
Bruno

Thanks, we are already in contact with each other, it's time to bring the frustration to the Forum...

[jans]

Hi All,
Shouldn't the function ParMETIS _V3_PartMeshKway function, for example, be able to do decomposition of the mesh in parallel? This is based on a parallel multi-level graph-partitioning algorithm. The paper describing this algorithm can be found here: http://www.cs.ucsb.edu/~gilbert/cs29...mlevel_mpi.pdf. ptScotch also has parallel graph partitioning routines. Hence it seems to me xiaoweii has a point. It should be possible to parallelize decomposePar itself.

It seems to me from the discussion in this thread though that currently only a serial version of decomposePar is available. Am I right? Is there at least some near term plan to parallelize decomposePar using these parallel graph partioning algorithms?

Thanks
Anirban

[wyldckat]

Greetings Anirban,

AFAIK, as I've mentioned in one of the old posts:

• decomposePar is meant to be used for initial decomposition, which is why it's executed in serial mode only.
• redistributePar can be used in parallel and therefore it should be able to use "ptscotch" and similar algorithms.

If you check the file "applications/utilities/parallelProcessing/decomposePar/decomposeParDict" (see here online), you'll see that there is an interesting method named "multiLevel", which allows you to do multi-level decomposition.
Thinking a bit further on this last option, it gives the idea that we can do the following:

• Firstly we decompose with the N partitions we want, but using the "simple" or "hierarchical" methods. Don't forget to run decomposePar.
• Then we change in the file "system/decomposeParDict" from the method "simple" to "ptscotch".
• Then run redistributePar in parallel:

  foamJob -s -p redistributePar

  Or if you prefer the hardcore way of doing it (replace "N" with the number of cores):
  

  mpirun -np N redistributePar -parallel

For more information on how "ptscotch" is being used, check the file: src/parallel/decompose/ptscotchDecomp/ptscotchDecomp.C


Best regards,Bruno

[jans]

Hi Bruno,
Thanks much. Your suggestions are very interesting. I play with these and get back on what I discover.

I still do not entirely convinced why even the initial decomposition cannot be done in parallel. It seems like you are saying it is impossible algorithmically. I am not sure why. Oh well!

Best regards
Anirban

[wyldckat]

Hi Anirban,

AFAIK, the current OpenFOAM framework contemplates only a few parallel mesh paradigms:

• The mesh is loaded into a single process and decomposed into several sub-domains.
• Each sub-domain is loaded by a single process (standard OpenFOAM parallel operations);
  • and may or may not exchange cells with another sub-domain (snappyHexMesh and redistributePar do this).
• A single process reads each sub-domain and merges them into a single one, or it even manipulated each sub-domain on a single process.

Currently I'm not aware of any OpenFOAM application that runs in parallel and loads the same full mesh onto memory on all processes and then partition things among them.


And I think I forgot to mention in the previous posts a certain important aspect of how the official OpenFOAM version works: it only uses MPI for parallel processing. It does not use OpenMP nor is it prepared to work with multi-threaded processes.
The Extend variant of OpenFOAM on the other hand, is suppose to be able to work with multi-threaded capabilities, but I'm not familiar enough with it so I don't know what it can do with those multiple threads...


Now back on the multi-level topic: this option might be insinuating that the authors of OpenFOAM are already working on this kind of situation, namely a situation where they have hundreds of million cells in a mesh that need to be decomposed to thousands of cores, therefore the distribution has to be made in a multi-level manner, in order to propagate the mesh gradually throughout the whole system.
Nonetheless, I'm not aware of any specific documentation on this topic; and from what I know, it requires to start with at least with a seed decomposition of 2 sub-domains and then rely on redistributePar to take care of things and lead them to the thousands.


Either way, you do have some more possible ways to find more about this:

• By looking at the source code;
• By asking at the support section on openfoam.com
• By making a feature request on the bug reporting section at openfoam.org

Best regards and please do share with us what you may find out on this topic!
Bruno

[wyldckat]

Greetings to all!

@Anirban: Apparently redistributePar can redistribute from one processor to multiple processors, as somewhat explained on the following thread: http://www.cfd-online.com/Forums/ope...oundaries.html

It's even implied in the header documentation: https://github.com/OpenFOAM/OpenFOAM...istributePar.C

 Can also work like decomposePar:
\verbatim
# Create empty processor directories (have to exist for argList)
mkdir processor0
..
mkdir processorN

# Copy undecomposed polyMesh
cp -r constant processor0

# Distribute
mpirun -np ddd redistributePar -parallel
\endverbatim

Best regards,
Bruno

[landsber]

Hi All,
I tried to do use decomposePar in parallel as it is shown in the OF v3.0+ guide :
mpirun -np 2 redistributePar -decompose -parallel
but got an error that the argument decompose doesn't exist in redistributePar. Did someone already used it ? and specifically with the metis domain decomposition decomposition tool?

Thanks
Alex

[wyldckat]

Quick answer:

Originally Posted by landsber

I tried to do use decomposePar in parallel as it is shown in the OF v3.0+ guide :
mpirun -np 2 redistributePar -decompose -parallel
but got an error that the argument decompose doesn't exist in redistributePar.

If you are using OpenFOAM v3.0+, v1606+, v1612+ or v1706 (from http://openfoam.com), then that option should work.

If you are using any other version of OpenFOAM, namely the versions from http://openfoam.org - then that option does not exist.