How to run decomposePar with parMetis?
parMetis is a parallel decompose method, that means we can decompose the mesh using several processes.
I just don't know how to do this? After change the decomposeParDict, run "decomposePar" just executes in serial, and the command "mpirun -n 4 decomposePar -parallel" cannot work. |
Hello xiaow_g,
the decomposing is normally done after blockMesh to run snappyHexMesh in parallel (parallel meshing) or to decompose the mesh after meshing with snappyHexMesh to run the calculation in parallel (parallel calculation). The decomposing needs only the decomposeParDict so that you can run it by simply type decomposePar in the terminal. For the next steps parallel Meshing / calculation you need to use the mpirun -np ... (snappyHexMesh/rhoSimpleFoam) -parallel ... I hope my explanation is not to difficult :) MARTIN |
Quote:
I got your idea. You told me how to "parallel meshing" and "parallel calculation", but I am confused about how to "parallel decomposing". If running the decomposing in serial , I just need to run decomposePar in the terminal. but parMetis or ptScotch is parallel decomposing method, and the decomposing method can be set in the decomposeParDict. |
There is nothing like parallel decomposing. The only thing what the decomposer is doing is to decompose the mesh or field into processor-folders. This is accomplished in serial by running decomposePar. The methods how the field is beeing decomposed is described in the dictionary (hierarical, scotch, metis ...).
|
Greetings to all!
Martin is right. The only thing that the ptscotch and parmetis methods are good for is for parallel mesh manipulation. One example, already mentioned by Martin, is snappyHexMesh when executed in parallel, which allows mesh manipulation in parallel... although I don't know if cells are exchanged between processors in a load balancing system or if only the mesh structure is accessible between processors. If you study the script "incompressible/simpleFoam/turbineSiting/Allrun", you'll notice exactly this: snappyHexMesh must use ptscotch... ... or in other words, it can't use scotch, because I think I've been able to use simple and hierarchical with snappyHexMesh... Best regards, Bruno |
Quote:
I'm bumping this thread, because I need to use dynamic load balancing for the code I wrote. It's a two phase library with a geometrical transport of the volume fraction field, so the "action" is happening mostly near the interface. Is there a "recipe" for using dynamic load balancing in OpenFOAM in general? If there is none, can you at least name the ingredients? :) How do I use ptscotch and parmetis together with the dynamicRefineFvMesh? Tomislav |
Hi Tomislav,
I think it's best that you ask in a new thread in the "Running" sub-forum, because that's more closely related to your question ;). The closest I know about load balancing in OpenFOAM is the redistributePar application, but this would require you to stop the solver once in a while for redistribution and then running it again to continue. You can have a look in the folder "application/test" for ideas. Last but not least: I suspect that said capability is possible in OpenFOAM, but probably it's not implemented yet. Best regards, Bruno |
Quote:
Thanks, I will check it out. :) Best regards, Tomislav |
Hi Tomislav,
Here you go, someone else is working on this subject as well: http://www.cfd-online.com/Forums/ope...tml#post384606 My guess is that you weren't lucky enough to find that thread in the first place... Best regards, Bruno |
Quote:
|
Hi All,
Shouldn't the function ParMETIS _V3_PartMeshKway function, for example, be able to do decomposition of the mesh in parallel? This is based on a parallel multi-level graph-partitioning algorithm. The paper describing this algorithm can be found here: http://www.cs.ucsb.edu/~gilbert/cs29...mlevel_mpi.pdf. ptScotch also has parallel graph partitioning routines. Hence it seems to me xiaoweii has a point. It should be possible to parallelize decomposePar itself. It seems to me from the discussion in this thread though that currently only a serial version of decomposePar is available. Am I right? Is there at least some near term plan to parallelize decomposePar using these parallel graph partioning algorithms? Thanks Anirban |
Greetings Anirban,
AFAIK, as I've mentioned in one of the old posts:
Thinking a bit further on this last option, it gives the idea that we can do the following:
Best regards,Bruno |
Hi Bruno,
Thanks much. Your suggestions are very interesting. I play with these and get back on what I discover. I still do not entirely convinced why even the initial decomposition cannot be done in parallel. It seems like you are saying it is impossible algorithmically. I am not sure why. Oh well! Best regards Anirban |
Hi Anirban,
AFAIK, the current OpenFOAM framework contemplates only a few parallel mesh paradigms:
And I think I forgot to mention in the previous posts a certain important aspect of how the official OpenFOAM version works: it only uses MPI for parallel processing. It does not use OpenMP nor is it prepared to work with multi-threaded processes. The Extend variant of OpenFOAM on the other hand, is suppose to be able to work with multi-threaded capabilities, but I'm not familiar enough with it so I don't know what it can do with those multiple threads... Now back on the multi-level topic: this option might be insinuating that the authors of OpenFOAM are already working on this kind of situation, namely a situation where they have hundreds of million cells in a mesh that need to be decomposed to thousands of cores, therefore the distribution has to be made in a multi-level manner, in order to propagate the mesh gradually throughout the whole system. Nonetheless, I'm not aware of any specific documentation on this topic; and from what I know, it requires to start with at least with a seed decomposition of 2 sub-domains and then rely on redistributePar to take care of things and lead them to the thousands. Either way, you do have some more possible ways to find more about this:
Bruno |
Greetings to all!
@Anirban: Apparently redistributePar can redistribute from one processor to multiple processors, as somewhat explained on the following thread: http://www.cfd-online.com/Forums/ope...oundaries.html It's even implied in the header documentation: https://github.com/OpenFOAM/OpenFOAM...istributePar.C Quote:
Bruno |
running decomposePar in parallel
Hi All,
I tried to do use decomposePar in parallel as it is shown in the OF v3.0+ guide : mpirun -np 2 redistributePar -decompose -parallel but got an error that the argument decompose doesn't exist in redistributePar. Did someone already used it ? and specifically with the metis domain decomposition decomposition tool? Thanks Alex |
Quick answer:
Quote:
If you are using any other version of OpenFOAM, namely the versions from http://openfoam.org - then that option does not exist. |
All times are GMT -4. The time now is 15:37. |