Decomposing meshes
Hi all,
I have a few questions to the decompose process. The first question is about the decomposition output and general stuff:
Code:
Processor 0 The second question is about simple and hierarchical
If you split like (3 1 2) in xyz, you split first x (3) then y (1) and z (2). At least you also get 6 domains which should be (in my imagination) the same as above, or not? The third question is about scotch and metis
At least one more questions
Thanks in advance and for reading the topic. Any hints and experiences are welcome. |
Hi all,
today I made some tests with my college using decompose method hierarchical. Question two is not anymore necessary to explain. I miss understand something. For all who are interested here is the answer:
|
Hey Tobi,
could you please answer your questions from above if you know the answers now? Thanks and best wishes Sebastian |
Hello Sebastian,
I don't have all answers but I can give you more informations. Answer to question 1
Information about question 3
|
"Again what learned" as Lothar Matthäus would say :D!
Thank you very much for your quick answer... |
3 Attachment(s)
Let me share a recent observation of mine.
I simulated axi-symmetric gas flow in a long pipe. As the domain is quite large, the simulation was run using several parallel processes. The scotch decomposition created a processor boundary, which zic-zags through the pipe. The part of the processor boundary, which is parallel to the flow, seems to create some disturbance in the pressure field. Luckily, the disturbance does not blow up the simulation, yet it's quite interesting. The attached images show the pressure field. The involved subdomains are shown as white wireframes. |
There is also a topic of damn break in the German OpenFOAM forum. A guy decomposed the domain and got different results for different decomposition methods. That is clear if one is thinking about the fluxes which has to be shared at the processor boundaries.
Nice to get the proof again. Thank you Gerhard |
The images I posted are from a case, which I am not able to share. Unfortunately, I am not able to reproduce the issue using a simpler, distributable geometry.
A minimal working example (MWE) for the behaviour I observed in my case, would be quite interesting, since this is a quite odd behaviour. |
The influence of the decomposition should be stronger for free convection (if the flow can go anywhere). However, for forced convection - while the fluid has a fixed direction - the decomposition should not influence the fields too much.
|
Quote:
|
Decomposing Mesh for a multiregion domain
Dear Tobi and all
I am performing a heat transfer simulation in which I have four regions. When I use scotch method of decomposition for the regions, the mesh is decomposed correctly however, it doesnot move forward while performing faceAgglomeration. When I use simple method with the following coefficients for the two of the larger regions while using scotch method for the other two domains I get a singular matrix error. numberOfSubdomains 144; method simple; simpleCoeffs { n (16 9 1); // total must match numberOfSubdomains delta 0.001; } When I try using simple method for all regions with the above coefficients as before for the larger regions while the following coefficients for the other two smaller regions, numberOfSubdomains 144; method simple; simpleCoeffs { n (12 12 1); // total must match numberOfSubdomains delta 0.001; } I get the following warning and I also find 0 cells in some of the processors during decompostion FOAM Warning : From function Foam::polyMesh::polyMesh(const Foam::IOobject &) in file meshes/polyMesh/polyMesh.C at line 330 no points in mesh When I try running the solver, it terminates with the following message [57] --> FOAM FATAL ERROR: [57] (5 20) not found in table. Valid entries: 847 ( (98 103) (88 104) ....................... ........................... [57] From function T& Foam::HashTable<T, Key, Hash>::operator[](const Key&) [with T = double; Key = Foam::edge; Hash = Foam::Hash<Foam::edge>] [57] in file OpenFOAM-6/src/OpenFOAM/lnInclude/HashTableI.H at line 117. Can some one kindly help me to fix this issue |
I am a bit confused about the results shared here.
In my opinion, a domain decomposition/parallelization strategy should be designed in a way that it has no influence on the results whatsoever. That would be my first and most important item on a list of prerequisites for any parallelization. Is this a bug in OpenFOAM, or do other CFD packages just do a better job at hiding the influence of domain decomposition? |
Dear Flotus,
well, as I have no idea how ANSYS and other programs are solving the problem, I only can make a rough statement how I think FOAM is doing it. Maybe it is wrong here, probably as I don't now these things in detail as I never investigated into that. Without talking about parallelization, what I think we do is actually to divide the physical domain into closed single domains that - together - form the whole geometry again. Now, the problem is as follows:
Lets imagine that we have two cells that share a face for the one core case, it is easy to calculate the face value based on the two cells. Now, assume that the mesh is split at these cells. Therefore, as each processor domain is separated, the before mentioned cells don't know anything about each other anymore. They are only connected via the faces. Therefore, we calculate the value at the processor face by using only one cell (as we don't know the neighbor cell -> it is in another processor domain). This information is sent to the other mesh that is solved by the other processor and it is done until we reach the convergence criterion. So actually it is as follows:
Thus, for directed flows, this is not a big deal but for free convection it will influence your solution. Of course, if your decomposition strategy is not well defined and you get the processor boundary faces at - let say - really bad locations, this will also influence forced fluid flow cases. Nevertheless, I personally could tell that I have decomposition influences probably for free-convection cases using scotch, as it randomly decomposes your mesh. The error one introduces here depends on the number of decomposed regions, the way one decomposes (e.g., arbitrary (scotch) or aligned to the axis (simple, hierarchical)). Hope it got a bit clearer. If other software uses other strategies, no idea. PS: I might investigate into the processor boundary condition to ensure how it works. |
What you describe sounds like ordinary domain decomposition.
If a cell on a domain boundary needs information from an adjacent cell, which resides in a different domain, then the parallelization needs to provide this information. E.g. via MPI transfers. And that's what is usually done when parallelizing a code using domain decomposition. A very intuitive way to achieve this is to add the adjacent cells from the neighbor domain to the original domain, sometimes referred to as "ghost cells". They don't update their own values, they just provide values for updating the regular cells of each domain. I thought this was the standard way of handling domain decomposition, which avoids reverting back to lower order methods. |
Hi Alex,
well, I have to say, I don't know if foam is doing it like that. Here, one should investigate into the processor boundary condition in order to allow one to make a clear statement. I can't do that and I added a hint to my previous post. |
This slides provide some explanation of how parallization is done in OF https://www.google.com/url?sa=t&sour...a1AouhvqjcH3Ih
|
Thanks for the link.
Summary: I was wrong. We share the information of the neighbor cell values if I got the presentation correct. |
I understood the presentation that the processor patch is treated as boundary conditions.
If we look at the source code (https://www.openfoam.com/documentati...8C_source.html) we have an evaluate() function. This function is called to set the boundary conditions for the fields which are solved by the fvMatrix class. Actually it is called by the function correctBoundaryConditions(). For a deeper explanation see this thread: https://www.cfd-online.com/Forums/op...-evaluate.html. The correctBoundayCondition() function is directly called by the fvMatrix solver when solving the matrix. (see e.g. https://www.openfoam.com/documentati...ve_8C_source.c) so I guess depending on the operator (div or laplacian) the processor patch is responsible to evaluate the fluxes on the patch |
Dear Foamers,
I am recently working on the wide-gap Taylor-Couette flow (eta=0.5), the Reynolds number is 475, the number of vortices is varying according to the number of processors and time-steps. In the work of Razzak, they found the number of vortexes is 6 at Reynolds number is 475. (https://doi.org/10.1063/1.5125640) However, in my study, the number of vortexes is 6 when using 280 processors, the number of vortexes is 8 when using 240 processors, the number of vortexes is 10 when using 360 processors. The OpenFOAM version is openfoam5. The decompose method used here is scotch, similar results observed with simple and hierarchical methods to do decompose. So my question is whether the decomposing method in OF is able to such a lower Reynolds number? Have you ever met such an issue that the flow structure is varying according to the number of processors and time-steps? Maybe in turbulent flow, the numerical dissipation induced by the parallel decomposing will be less significant. Thanks in advance. https://www.cfd-online.com/Forums/op...ime-steps.html |
It is well established that the accuracy of the domain decomposition preconditioner decreases as the number of subdomains increases (see e.g. [1], [2]).
I am unaware of how this effects number of vortices. I can imagine, however, that there is some link. Do you monitor residuals in the simulations? Are you able to enforce same accuracy at each step of the segregated solver, independent of number of processors? [1]: https://books.google.nl/books?id=dxw...sition&f=false [2]: ddm.org |
All times are GMT -4. The time now is 20:12. |