CFD Online Discussion Forums

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)
-   OpenFOAM Running, Solving & CFD (https://www.cfd-online.com/Forums/openfoam-solving/)
-   -   OpenFOAM version 1.6 details (https://www.cfd-online.com/Forums/openfoam-solving/67629-openfoam-version-1-6-details.html)

lakeat August 21, 2009 05:28

OpenFOAM version 1.6 details
 
Dear all,

Have a nice day!

I have some questions,
  1. How to decide which solver should be used, GAMG or smoothSolver or PCG? I have read some papers and some threads, but still have no idea about the guidelines.
  2. What is the difference between pimplefoam and pisofoam?

Thank you.

chiven August 21, 2009 10:14

Hi, Daniel, nice to meet you herein.

About question 2,
Navier-Stokes equations solving methods are different, they are PISO algorithm and PIMPLE (merged PISO-SIMPLE) algorithm respectively. Maybe the following references can help you to understand more details:
1. J.H. Ferziger and M. Peric Computational Methods for Fluid Dynamics 3rd ed. Springer 2002.
2. Hrvoje Jasak, PhD 1996, PDF of thesis posted at: Error analysis and estimation in the Finite Volume method with applications to fluid flows.
3. Henrik Rusche, PhD 2002 Computational fluid dynamics of dispersed two-phase flows at high phase fractions.

about question 1, I also am not so clear, but for the p, I always use smoothsolver to get a faster convergence.

Best regards,
Chiven

sandy August 22, 2009 00:45

Quote:

Originally Posted by lakeat (Post 227049)
Dear all,

Have a nice day!

I have some questions,
  1. How to decide which solver should be used, GAMG or smoothSolver or PCG? I have read some papers and some threads, but still have no idea about the guidelines.
  2. What is the difference between pimplefoam and pisofoam?

Thank you.

:D:D PCG is the preconditioning CG method and can just be used to solve the symmetric matrix; smoothSolver is Guass Siedle method namely a very very old and slower method, but it can almost be used to solve any equation, I think. MG is multi-grids method. Many people think PBiCG, PCG and MG are very efficient methods, however, I think, maybe smoothSolver is most safest, and at least it is enough to solve U equation.

My advisor dislikes piso algorithm so that we are all not really familar with it. We all like SIMPLEC algorithm. But many people think Piso is good for unstable flows.

Am I wrong?

lakeat August 22, 2009 01:21

Thanks for your help.

Actually, What I am struggling with is Question 1, ie.
Guidelines for solver choosing

I am using pisoFoam simulating a cylinder flow, and my experience has shown that GAMG use much smaller time to do the same job than PCG.
  1. Is it correct to use GAMG for p?
  2. What if I use GAMG for U?
  3. How about the other turbulence variables?
  4. Prof. Jasak's SAMG sounds far better than the others, can I use it in my case?

Okay, let me add more questions here, I think the answers of these questions are very useful for every users, and should be put on top.

I have suffered the problem of parallel computing's speed-up, I remember once somebody talked a speed-up of 11.05 using 8 processors! That's impressive! How did he make it? The cylinder case I am solving is about 860000 grids, I found 32 processors or even 64 processors are much slower than what I thought. So, I want to know more detailedly,
Guidelines for decomposition:
  1. How many processors should be used for a given case like cylinder flow?
  2. What is the best grid numbers for each processor?
  3. Has OpenFOAM being used with more than 64 processors? How is it?

Thank you.

sandy August 22, 2009 04:44

Quote:

Originally Posted by lakeat (Post 227135)
Thanks for your help.

Actually, What I am struggling with is Question 1, ie.
Guidelines for solver choosing

I am using pisoFoam simulating a cylinder flow, and my experience has shown that GAMG use much smaller time to do the same job than PCG.
  1. Is it correct to use GAMG for p?
  2. What if I use GAMG for U?
  3. How about the other turbulence variables?
  4. Prof. Jasak's SAMG sounds far better than the others, can I use it in my case?
Thank you.

Hi Daniel, GAMG method can efficiently solve a positive definite, diagonally dominant matrix. However, in fact, as I known, p equation is actually a singular equation so that it is so difficult to be solved by most of current methods. You really think GAMG is efficient to solve it? Maybe PCG or PBiCG is better, I think. I seldom use GAMG. According to some papers, it is also very difficult to complex problems, if you don't want to choose Guass Seidel, I guess maybe you choose PCG or PBiCG more better. However, I don't know whether or not OpenFOAM will actually judge a matrix is really symmetric or asymmetric, before the PCG or PBiCG is implemented. Do you know where I can find the codes to judge a matrix in OpenFOAM?

I am so sorry I never hear of SAMG. Hope it is useful to you.

Sandy

alberto August 22, 2009 08:19

Hi Sandy,

Quote:

Originally Posted by sandy (Post 227138)
Hi Daniel, GAMG method can efficiently solve a positive definite, diagonally dominant matrix. However, in fact, as I known, p equation is actually a singular equation so that it is so difficult to be solved by most of current methods.

you are using the word "singular" in the wrong context. A "singular equation" cannot be solved at all, and in the case of the pressure equation you say it is "singular" if the matrix associated to the discretized equation is singular (its determinant is zero). This it clearly not the case in general, so you cannot say the pressure equation is singular.

Quote:

However, I don't know whether or not OpenFOAM will actually judge a matrix is really symmetric or asymmetric, before the PCG or PBiCG is implemented. Do you know where I can find the codes to judge a matrix in OpenFOAM?
Yes OpenFOAM checks if a matrix is symmetric or not. Just try to use a symmetric solver for an asymmetric matrix, and you'll notice that!

Best,
A.

alberto August 22, 2009 08:41

Hi lakeat,

Quote:

Originally Posted by lakeat (Post 227135)
Thanks for your help.
I am using pisoFoam simulating a cylinder flow, and my experience has shown that GAMG use much smaller time to do the same job than PCG.

That's exactly the goal of multi-grid methods: increase the convergence rate of the solution by using different grids, starting from a coarse one, mapping to a finer one and proceeding until the desired solution is obtained.

Quote:

  1. Is it correct to use GAMG for p?

Why not? It is a linear solver, and as such it can be used to solve any linear system. The criterion of choise is the performance to obtain the desired accuracy. Using GAMG (and in general AMG solvers) for the solution of the pressure equation makes sense because its convergence might be slow, and using a multi-grid method should make it faster. Keep in mind that it is convenient to use AMG-type solvers when the cost of the coarsening/remapping/refining process is outweighted by the increase in speed.

Quote:


2. What if I use GAMG for U?
See above.

Quote:


3. How about the other turbulence variables?
See above. Probably you would not notice much difference? Try and compare ;-)

Quote:

I have suffered the problem of parallel computing's speed-up, I remember once somebody talked a speed-up of 11.05 using 8 processors! That's impressive! How did he make it? The cylinder case I am solving is about 860000 grids, I found 32 processors or even 64 processors are much slower than what I thought. So, I want to know more detailedly,
How did they define the speedup?

Quote:

  1. How many processors should be used for a given case like cylinder flow?
  2. What is the best grid numbers for each processor?

This really depends on your case, on the solver you use, on what conditions you are considering. You can have general guidelines but they would be quite vague in my opinion.
Once you respect the basic rules of trying to decompose your domain minimizing the information exchange (in other words in the case of a symple cylinder try not to cut along the lenght of the pipe but make sections orthogonal to the pipe axis, for example), make some experiment and see what fits your needs.

Best,

lakeat August 22, 2009 09:13

Hi,

Thanks again. Your explanation is very clear and I will do the comparison.
I have chance now to access the fastest super-computer nationwide, I must compare and tailor out the number of processors needed for a given case.

Now apparently, I think simple and hierarchical are just for some academical comparison, it is now metis and scotch that are much superior to them. I have compared and seen a great advantages of scotch over the others, and it's easy for use too. I will use scoth henceforth.

FYI, my cylinder case has 860000 grids, o-type, using scoth, I did not see a better performance in using 32 cpus over 16 cpus over 8 cpus, I will give the pictures as soon as my hands are free. This is why I begin to think about the question:

"Is there any more choice to improve the parallel performance?"

I saw Prof. Jasak are recommending SAMG, In your opinion, will SAMG be better than GAMG and suitable for my case? (pisoFoam) Thank you.

Concerning the speedup, I get the info from thread -- http://www.cfd-online.com/Forums/ope...foam-13-a.html
The paper can be downloaded from:
(http://www.jncasr.ac.in/kirti/current_science.pdf)
That's astonishing, I have never experience that's high.


Thank you. :)

alberto August 22, 2009 09:17

Quote:

Originally Posted by sandy (Post 227132)
My advisor dislikes piso algorithm so that we are all not really familar with it. We all like SIMPLEC algorithm. But many people think Piso is good for unstable flows.

Well, maybe he should motivate why he dislikes PISO algorithms. It is not a question of taste when you select an algorithm, but a question of what works best for a given case. In other words, you should know PISO algorithms too and choose what fits your needs. :D

The SIMPLE algorithm in its original form (Caretto et al, 1972, Proceedings of the Third International Conference on Numerical Methods in Fluid Mechanics, Paris) is based on obtaining a pressure correction equation in which the velocity corrections are dropped because unknown (See, for example, Ferziger and Peric book). This is quite crude, and is one of the reasons why the method is not that efficient in terms of convergence rate. The effect of neglecting these terms is absent once the solution is converged however, and the method can be used "as is" for steady problems, where under-relaxation can be used to keep the solution stable.

An unsteady version of the SIMPLE algorithm can be written, by performing two loops: the external loop in time, and an internal loop to obtain a convergence solution at a given time step. In the internal loop, under-relaxation can be used, as long as a sufficient number of iterations is ensured, in order to reach the actual solution at that given time.

SIMPLEC is a version of SIMPLE where the velocity correction is introduced. However its convergence rate is very similar to what obtained with SIMPLE, using the correct under-relaxation: URF_p = 1 - URF_u (Again, see Ferziger and Peric book).

A more refined approach is the PISO one where you
  • ignore the velocity corrections in the pressure equation as in the SIMPLE algorithm at the first step
  • apply the velocity correction and then perform a number of other corrector steps, originating an interative procedure to solve the pressure equation treating the velocity correction term in it in an explicit manner.
as suggested by Issa (1986, Journal of Computational Physics). The PISO algorithm is considered particularly effective for unsteady flows.

Best,

lakeat August 22, 2009 09:26

Quote:

Originally Posted by alberto (Post 227163)
Well, maybe he should motivate why he dislikes PISO algorithms. It is not a question of taste when you selecting an algorithm, but a question of what works best for a given case. In other words, you should know PISO algorithms too and choose what fits your needs. :D

A more refined approach is the PISO one where you
  • ignore the velocity corrections in the pressure equation as in the SIMPLE algorithm at the first step
  • apply the velocity correction and then perform a number of other corrector steps, originating an interative procedure to solve the pressure equation treating the velocity correction term in it in an explicit manner.
as suggested by Issa (1986, Journal of Computational Physics). The PISO algorithm is considered particularly effective for unsteady flows.

Best,


Hi, Sandy, There are advisors and advisors ... :D:D:D

alberto August 22, 2009 09:32

Quote:

Originally Posted by lakeat (Post 227162)
Hi,

Now apparently, I think simple and hierarchical are just for some academical comparison, it is now metis and scotch that are much superior to them. I have compared and seen a great advantages of scotch over the others, and it's easy for use too. I will use scoth henceforth.

For a cylinder flow? A "simple" manual decomposition should do the job. In my experience metis is OK if you have complicated cases where finding a good decomposition by hand is tricky, but for simple cases like a channel flow or a pipe flow, a hand-made decomposition is far easier and might be more effective. I still have to try scotch. How does it decompose the pipe?

Quote:

FYI, my cylinder case has 860000 grids, o-type, using scoth, I did not see a better performance in using 32 cpus over 16 cpus over 8 cpus, I will give the pictures as soon as my hands are free. This is why I begin to think about the question:

"Is there any more choice to improve the parallel performance?"
Well, I don't know what processors you have handy. But with 860.000 cells, for a normal single-phase simulation, 8-12 processors are enough.

Quote:

I saw Prof. Jasak are recommending SAMG, In your opinion, will SAMG be better than GAMG and suitable for my case? (pisoFoam) Thank you.
I do not have any experience with SAMG. If he tested it in a similar case, I would say so.

Quote:

Concerning the speedup, I get the info from thread -- http://www.cfd-online.com/Forums/ope...foam-13-a.html
The paper can be downloaded from:
(http://www.jncasr.ac.in/kirti/current_science.pdf)
That's astonishing, I have never experience that's high.
Thanks for the links.

Best,

sandy August 22, 2009 10:49

Hi Alberto, I still care of two question:

1. Which code or file to judge a symmetric or asymmetric matrix in OpenFOAM - 1.5?
2. If you think p equation is not always a singular matrix, why we have to spend so much CPU time during the numerical simulation. Which step is the most expensive. I think, except for the grid generation, it is the solution of the linear equation in every iterative step. Am I wrong? The key is what kind of matrix is the p equation???

I am actually weak to analyze the matrix structure.

Have a nice weekend.
Sandy

alberto August 22, 2009 15:07

Quote:

Originally Posted by sandy (Post 227168)
Hi Alberto, I still care of two question:

1. Which code or file to judge a symmetric or asymmetric matrix in OpenFOAM - 1.5?

Solvers are defined (with some exception for CG methods) for both symmetric and asymmetric matrix. Check the matrix implementation and the solver implementation in OpenFOAM. You might start with a search in the Doxygen documentation for "matrix", "symmetric matrix" and also for the solver classes.

Quote:

2. If you think p equation is not always a singular matrix, why we have to spend so much CPU time during the numerical simulation. Which step is the most expensive. I think, except for the grid generation, it is the solution of the linear equation in every iterative step. Am I wrong? The key is what kind of matrix is the p equation???
I think you are confusing some basic concept of linear algebra. Let me try to clarify, but I suggest you to read a good book about the topic before proceeding with CFD.

Consider a square matrix A \in R^{n x n} (matrix with n rows and n columns, of real elements). You say that the matrix is invertible (or not singualar) if a matrix A^{-1} exists so that
A A^{-1} = A^{-1} A =  I,
where I is the identity matrix, and A^{-1} is the inverse matrix.

If such a matrix does not exist, the matrix is said to be singular, or not invertible.

You can easily understand if a matrix is invertible by computing its determinant, because a square matrix is singular if and only if its determinant is zero.

Now, let's consider a linear system in the form
A x = b,
where x is the unknown vector, b the known vector and A the matrix. Its solution is clearly given formally by
x = A^{-1} b
and exists only if A^{-1} is invertible, or, in other words, not singular.

If you have a linear system whose matrix has null determinant, you can still have solutions, but you need to evaluate the rank of the matrix, and compare it to the rank of the complete matrix (rectangular), obtained by adding to A the column b.
According to Rouché-Capelli theorem, you know that
  • Ax = b has solutions only if the rank of A (incomplete matrix) is equal to the rank of the complete matrix (A|b) obtained as said above.
  • The solution is unique only if rank(A) = rank(A|b) = n, where n is the dimension of the matrix
  • If rank(A) = rank(A|b) < n, the system has infinite solutions, with a moltiplicity equal to n - rank(A).
To go back to the pressure equation, keeping in mind what said above, you cannot say the matrix originating from its discretization is singular, since it is actually admit inverse (you indirectly determine it by solving the linear system itself).

The matrix of the linear system originating from the discretization of the pressure Poisson equation in standard form is tridiagonal or, more generally, block tridiagonal. There are specific algorithms for the solution of this kind of matrix (Thomas algorithm and its block version for example), which are very effective due to their low computational cost (Thomas algorithm costs O(n), while Gauss method costs O(n^3), for example).

I hope this helps.

Best,

sandy August 22, 2009 16:45

Quote:

Originally Posted by alberto (Post 227182)

According to Rouché-Capelli theorem, you know that
  • Ax = b has solutions only if the rank of A (incomplete matrix) is equal to the rank of the complete matrix (A|b) obtained as said above.
  • The solution is unique only if rank(A) = rank(A|b) = n, where n is the dimension of the matrix
  • If rank(A) = rank(A|b) < n, the system has infinite solutions, with a moltiplicity equal to n - rank(A).

Hi Alberto, if rank(A) = rank(A|b) < n, what is its determinant? And whether or not it is invertible and has solutions? This kind of matrix is not singular matrix, right? Maybe the pressure equation is this kind of matrix? Thanks.

alberto August 22, 2009 18:15

Quote:

Originally Posted by sandy (Post 227186)
Hi Alberto, if rank(A) = rank(A|b) < n, what is its determinant? And whether or not it is invertible and has solutions? This kind of matrix is not singular matrix, right? Maybe the pressure equation is this kind of matrix? Thanks.

Hi Sandy,

if the rank of A is less than n, the determinant of A is zero, and A is singular. Let's consider an example I took from my linear algebra book (Greco, Valabrega, Lezioni di Algebra e Geometria, Levrotto & Bella, 1992).

Let's consider A=

1 -2 1 -1
1 -2 0 -2
-2 4 -1 3
-2 4 -2 2

and a known vector b = {2, -3, 1, -4}, so that the complete matrix is (A|b) =

1 -2 1 -1 | 2
1 -2 0 -2 | -3
-2 4 -1 3 | 1
-2 4 -2 2 | -4

If you careful look at the matrix A, you notice that the first row and the last row are linearly dependent, because you can obtain the last row by multiplying the first one by -2. This leads to a null determinant of the matrix, for a known property of matrix determinants (you can calculate it to check). This means that the rank of the matrix is not 4.

Now, let's remind one practical definition of matrix rank: A matrix has rank r if the number of linearly independent rows (or column) is r, and r+1 rows (or columns) are always linearly dependent.

According to this definition, the rank of matrix A is 2, since also the second row can be obtained as a linear combination of the first and the third rows, so the rank cannot be 3.

Similarly, you can consider the complete matrix, and show its rank is 2.

According to Rouche-Capelli theorem you have that the system admits solutions, which are infinite, with multiplicity equal to 4 - 2 = 2. In practice, you find these solutions by taking two linearly independent equations from the system and use them to define the solution, which will have two variables free to vary.

Notice that you can show that saying a matrix has zero determinant is equivalent to say it has two (or more) linearly dependent rows (or columns).

Back to the pressure equation, I guess at this point it is clear that the matrix of the linear system obtained from the discretization of the Poisson equation has nothingto do with a singular matrix.

Usually the difficulty in the numerical solution (analytically it's not a problem clearly) of a linear system is related to the condition number of its matrix (the highest this number is, the hardest it is to numerically solve the system accurately because of the limited machine precision).

Now, please refer to a good linear algebra book. You need to know these elements of linear algebra if you want to properly understand how CFD codes and numerical algorithms work. It is really in your interest, and if I got it right you are a student, so take advantage of that and attend algebra classes :D

I hope this helps.

Best,

sandy August 22, 2009 20:21

Quote:


Back to the pressure equation, I guess at this point it is clear that the matrix of the linear system obtained from the discretization of the Poisson equation has nothingto do with a singular matrix.

Hi Alberto, I am a student. I am appreciating so much for your kind help. Could you further explain how to derive the pressure equation is singular matrix, when giving normal velocity BC and the first class boundary condition namely p = constant? Why the pressure equation has solutions in Nuemann boundary condition?

Thanks again.
Sandy

sandy August 22, 2009 20:24

In addition, I actually need to know how to analyze the matrix structure in OpenFOAM. In my case, an "implicitly" solved source term was added into this Poisson equation. I want to know whether the new pressure equation still keep symmetric or not??? NO! right? Is it singular either?? What boundary conditions can make it well convergence. It is the first time I meet with this kind of pressure equation. Please help me out. Thanks a lot.

alberto August 23, 2009 09:06

Quote:

Originally Posted by sandy (Post 227193)
Hi Alberto, I am a student. I am appreciating so much for your kind help. Could you further explain how to derive the pressure equation is singular matrix, when giving normal velocity BC and the first class boundary condition namely p = constant? Why the pressure equation has solutions in Nuemann boundary condition?

Thanks again.
Sandy

Hi Sandy,

why don't you derive the matrix yourself in a simple case like a rectangle, with these conditions, assuming the flow incompressible to make it simpler:

top: fixed pressure, zero velocity gradient
bottom: fixed velocity, zero pressure gradient
sides: zero velocity, zero pressure gradient

which is the typical case in a channel? Why do you think that matrix is singular?

Note that in the case you use Neumann BC's on all boundaries for the pressure, you obtain an ill posed problem since you never specify a value explicitly for the pressure, and you generally need some addtional information to solve it.

Best,

alberto August 23, 2009 09:17

Quote:

Originally Posted by sandy (Post 227194)
In addition, I actually need to know how to analyze the matrix structure in OpenFOAM. In my case, an "implicitly" solved source term was added into this Poisson equation. I want to know whether the new pressure equation still keep symmetric or not??? NO! right? Is it singular either?? What boundary conditions can make it well convergence. It is the first time I meet with this kind of pressure equation. Please help me out. Thanks a lot.

I would suggest to derive the discretized equation for your case and check yourself, so you will learn how it works. You can use a good linear algebra book and some basic CFD books as a starting point to build your base:
  • Ferziger and Peric, Computational methods for fluid dynamics, Springer.
  • Tannehill, Anderson, Pletcher, Computational fluid mechanics and heat transfer, Taylor & Francis
Best,

sandy August 23, 2009 09:44

Quote:


Note that in the case you use Neumann BC's on all boundaries for the pressure, you obtain an ill posed problem since you never specify a value explicitly for the pressure, and you generally need some addtional information to solve it.

Hi Alberto, if I give the pdRefcell and pdRefvalue, is it still an ill posed problem?? I feel grey. Now I am actually in the Nuemann BC's CFD, namely zero Gradient in the academic informations. Thank you very much. :)

alberto August 23, 2009 09:54

Quote:

Originally Posted by sandy (Post 227216)
Hi Alberto, if I give the pdRefcell and pdRefvalue, is it still an ill posed problem?? I feel grey. Now I am actually in the Nuemann BC's CFD, namely zero Gradient in the academic informations. Thank you very much. :)

That's what you do for example in the cavity tutorial in OpenFOAM. Setting a pressure reference allows you to find a unique solution for the pressure.

Best,

sandy August 23, 2009 10:15

Quote:

Originally Posted by lakeat (Post 227164)
Hi, Sandy, There are advisors and advisors ... :D:D:D

YES, lakeat, Alberto is I said he said ... :D

alberto August 23, 2009 10:30

Quote:

Originally Posted by sandy (Post 227219)
YES, lakeat, Alberto is I said he said ... :D

I'm not an advisor, which is not easy if you want to do it right.
I was very very lucky during my studies and now in my post-doc to have good advisors though! :cool:

sandy August 24, 2009 07:45

Quote:

Originally Posted by lakeat (Post 227135)

I am using pisoFoam simulating a cylinder flow...
...
The cylinder case I am solving is about 860000 grids, I found 32 processors or even 64 processors ...

Hi lakeat, why you need so many grids to simulate a cylinder flow. I ever read a paper to just use 50000 grids to simulate a hydraufoil with the Kunz's cavitation model. How could they get it? You think. :confused:

alberto August 24, 2009 07:54

Hi Sandy,

I'm not sure how Lakeat is trying to simulate the flow, but if the Reynolds number is high enough (and it does not need to be very high), and he is doing LES, that number of cells does not seem huge to me. Keep in mind that in LES you must resolve scales until well inside the inertial subrange of the spectrum.

Best,

sandy August 24, 2009 08:02

WOW~, in 2D, it still needs so huge grids? If that, I should choose a good rest but work so hard.

lakeat August 24, 2009 08:08

Quote:

Originally Posted by alberto (Post 227266)
Hi Sandy,

I'm not sure how Lakeat is trying to simulate the flow, but if the Reynolds number is high enough (and it does not need to be very high), and he is doing LES, that number of cells does not seem huge to me. Keep in mind that in LES you must resolve scales until well inside the inertial subrange of the spectrum.

Best,

Yes, I am performing a LES simulation, this is what my thesis focuses.
I have never do an extensive survey, but I am not sure how good it will be to use RANS for a cylinder flow??????????

Will RANS give the Cd and Cl correct enough????

But Apparently, I will not expect good results from 2D simulation. The energy cascade is completely wrong in 2D simulation.

The Re (3900) is not high though, but already costs me a lot. This is LES.

I am expecting a much larger mesh for my next case Re=140000, the grid number would be 7 million.

You know what, someone told me that the "Bird Nest" in Beijing, certain man use just 80000 grids to do the simulation, and I was shocked, "how could he manage to do that?" "How could he get the content of TURBULENCE?" :confused::eek:

alberto August 24, 2009 08:24

Quote:

Originally Posted by lakeat (Post 227269)
But Apparently, I will not expect good results from 2D simulation. The energy cascade is completely wrong in 2D simulation.

In 2D you should not even do LES, since eddies are definetly 3D structures. :D

Quote:

The Re (3900) is not high though, but already costs me a lot. This is LES.
Re = 3900 (referred to the mean velocity and the diameter I assume) is not high at all, and it should be possible to do that in 3D with a very high resolution. Is your domain a simple cylinder (pipe)? Can you describe it with periodic conditions?

Quote:

You know what, someone told me that the "Bird Nest" in Beijing, certain man use just 80000 grids to do the simulation, and I was shocked, "how could he manage to do that?" "How could he get the content of TURBULENCE?" :confused::eek:
Modelling it in some way, probably, as done in many cases for practical applications. Does he claim he is doing LES? You could check what kind of resolution he gets with the simple estimation formulas provided in turbulence books (Pope, for example).

Best,

lakeat August 24, 2009 09:17

2 Attachment(s)
First, I want to say thank you, Alberto, and also thank you Sandy.

Okay, let me make myself clearer,

the cylinder case mesh is an O-type mesh, with Nx*Ny*Nz=165*165*32, Re=3900, with benchmark of LES simulation already published that can be easily followed.

To achieve Re=3900, I simply set inlet velocity 1m/s, and cylinder diameter 1m. Flow region is a circle with diameter of 15m, which means standing at the cylinder center, the distance to the inlet and out let are both 7.5m.

The timeStep is set to 0.0025 to keep Courant number no larger than unity.


Of course, this is full LES, a wall resolved LES, completely 3D simulation for a 2D geometry, as we can imagine from a wind tunnel section model. Periodic B.C. are applied in the front and back, I do not use convectiveOutlet for the moment.

Then I need to consider:
1. To achieve the highest efficiency, how many CPUs do I need for this case?
2. To achieve the highest efficiency and accuracy, which solver should I use for p and for U?

The 1st one, your experience is 8~16 processors are enough, right?
The 2nd one, you recommend GAMG for p and smoothsolver for U, right?

Here are some findings, some of which make me headache now,

No. 1
My teacher strongly doubt my simulation result, he said the simulation time is too long, and is unacceptable.
Code:

Time = 397.925

Courant Number mean: 0.0201989 max: 0.821767
smoothSolver:  Solving for Ux, Initial residual = 0.000263646, Final residual = 9.52965e-08, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.00125789, Final residual = 4.36729e-07, No Iterations 2
smoothSolver:  Solving for Uz, Initial residual = 0.00347622, Final residual = 1.45703e-06, No Iterations 2
GAMG:  Solving for p, Initial residual = 0.0120711, Final residual = 0.000402602, No Iterations 2
time step continuity errors : sum local = 2.96528e-09, global = 7.05016e-12, cumulative = -3.29373e-09
GAMG:  Solving for p, Initial residual = 0.000504671, Final residual = 4.92814e-05, No Iterations 3
time step continuity errors : sum local = 3.62961e-10, global = 3.00537e-12, cumulative = -3.29072e-09
ExecutionTime = 164607 s  ClockTime = 167103 s

Time = 397.927

Courant Number mean: 0.0202005 max: 0.821096
smoothSolver:  Solving for Ux, Initial residual = 0.000263663, Final residual = 9.53374e-08, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.00125653, Final residual = 4.36351e-07, No Iterations 2
smoothSolver:  Solving for Uz, Initial residual = 0.00347678, Final residual = 1.45956e-06, No Iterations 2
GAMG:  Solving for p, Initial residual = 0.0120541, Final residual = 0.000401538, No Iterations 2
time step continuity errors : sum local = 2.95737e-09, global = 6.11715e-12, cumulative = -3.2846e-09
GAMG:  Solving for p, Initial residual = 0.000502906, Final residual = 4.82624e-05, No Iterations 3
time step continuity errors : sum local = 3.5545e-10, global = 2.07349e-12, cumulative = -3.28253e-09
ExecutionTime = 164610 s  ClockTime = 167106 s

So what do you think, is there something wrong? Why my simulation is that long?

No. 2
Based on your experience, can you give a judge, are these figures correct? (I use pyFoam to get these figures)
I mean time spent.
Attachment 877Attachment 878
Note also the question in the picture. Will you meet the problem of unstable simulation time (ie. ExecutionTime) since the flow is unsteady?


Waiting for you,

alberto August 24, 2009 10:23

Quote:

Originally Posted by lakeat (Post 227283)
First, I want to say thank you, Alberto, and also thank you Sandy.

Okay, let me make myself clearer,

the cylinder case mesh is an O-type mesh, with Nx*Ny*Nz=165*165*32, Re=3900, with benchmark of LES simulation already published that can be easily followed.

To achieve Re=3900, I simply set inlet velocity 1m/s, and cylinder diameter 1m. Flow region is a circle with diameter of 15m, which means standing at the cylinder center, the distance to the inlet and out let are both 7.5m.

If you want a constant mean velocity of 1 m/s in your pipe, this is not correct, since the dissipation will slow your flow down. Furthermore, how can you fix the velocity at the inlet if the domain is periodic? You should use something like channelFoam (OpenFOAM 1.6) or channelOodles (OpenFOAM 1.5), where the pressure gradient is adapted per each time step so that the mean flow rate is kept constant.

In addition, what is your initial condition? If you use a uniform, not perturbed, condition it is going to take a long time to actually develop turbulence structures. Eugene de Villiers published a code to initialize a perturbed flow in a cylinder: search the forum for it.

Moreover, the lenght of the system has to be enough not to feel the effect of periodic conditions (I would say this was checked in the original work).

Quote:

Of course, this is full LES, a wall resolved LES, completely 3D simulation for a 2D geometry, as we can imagine from a wind tunnel section model. Periodic B.C. are applied in the front and back, I do not use convectiveOutlet for the moment.
I don't get it again. A cylinder has one wall BC and two sections as boundary conditions. If you specify periodic conditions in the flow direction, you cannot have other form of inlets/outlets.

Quote:

The 1st one, your experience is 8~16 processors are enough, right?
Well, with ~800,000 cells, and a single-phase flow, yes, definetly.

Quote:

My teacher strongly doubt my simulation result, he said the simulation time is too long, and is unacceptable.
Usually, if you start from a good initial condition, with fully developed turbulence, you need at least 20 characteristic times to extract statistics.
In other words, you initialize a perturbed flow as said above, you run until the flow is completely developed (and this takes time, a lot!), and then you reset the averages and start averaging for a sufficient number of characteristic times to get your statistics.

What does your teacher takes as reference to say the computational time is unacceptable?

P.S. What are you using as pressure residual?
Best,
A.

sandy August 24, 2009 10:26

I never think that GAMG should be chosen in parallel because it is so complex. U equation does not spend your CPU time too much, whatever which solver is chose. The key is P equation forever.

alberto August 24, 2009 13:02

Quote:

Originally Posted by sandy (Post 227292)
I never think that GAMG should be chosen in parallel because it is so complex.

Are you stating GAMG solvers are suitable only for serial calculations? I would not think so.

Simon Lapointe August 24, 2009 22:00

Hi guys,

I feel there is some miscommunication between you. From what I understood of Lakeat's problem (correct me if I'm wrong), he is simulating the external flow around a cylinder, not an internal pipe flow. When he says he uses periodic conditions at front and back I think he means at both ends of the cylinder.

Concerning Lakeat's concerns,

1) Many factors can affect your parallel performance.
First, it depends on the computer you're using. A poor connection between the computing nodes can really slow down the calculation, causing a big difference in ExcutionTime and ClockTime.
Second, if the connection is good, I've found that the OpenFOAM compilation can greatly influence the computational time. Are you using a pre-compiled OpenFOAM ? If you do, I strongly suggest you compile OpenFOAM locally on the computer you're using for your parallel computations, preferably with the local compiler (such as an Intel compiler).
Concerning the decomposition, a "simple" decomposition can be used on simple geometries. If you aren't so sure how to decompose it, you should use MeTis.
From my personal experience of unsteady simulations of external flows, I've seen good speedup (linear or better) with as less as 10-15k cells per processor. But that depends on the computer and the case.

2) I've used the GAMG solver for the pressure equation in many parallel cases and I think it performs well, generally faster than PCG. I don't agree with sandy. How the does fact that it is more complex make it slower ? Did you compare it with other solvers in parallel cases ?

Hope this helps

alberto August 24, 2009 22:53

Quote:

Originally Posted by Simon Lapointe (Post 227359)
Hi guys,

I feel there is some miscommunication between you. From what I understood of Lakeat's problem (correct me if I'm wrong), he is simulating the external flow around a cylinder, not an internal pipe flow. When he says he uses periodic conditions at front and back I think he means at both ends of the cylinder.

Oops. My fault.:o I should have thought to that when he described his system. Sorry about the confusion, please ignore my comment about the pipe flow simulation, the initialization of a perturbed flow and such.

About the domain decomposition, in the case of a flow around a cylinder, I agree with Simon when he suggests to use METIS, since it becomes more complicated to do by hand. I'm a bit skeptical about 10.000 cells/processor. In my experience a good trade-off is something between 40.000 and 80.000 cells per processor (meaning core), especially if you are not in a hurry to get the results and you share resources with others ;-)

Quote:

2) I've used the GAMG solver for the pressure equation in many parallel cases and I think it performs well, generally faster than PCG. I don't agree with sandy. How the does fact that it is more complex make it slower ? Did you compare it with other solvers in parallel cases ?
I agree on GAMG. Also in my experience it is generally faster than PCG. The only side note I have is that sometime (I'm thinking to some multi-phase case), PCG are stabler, while with GAMG I had crashes in some runs. However this should not be the case for a simple single phase flow.

Best,

lakeat August 24, 2009 22:54

Quote:

Originally Posted by Simon Lapointe (Post 227359)
Hi guys,

I feel there is some miscommunication between you. From what I understood of Lakeat's problem (correct me if I'm wrong), he is simulating the external flow around a cylinder, not an internal pipe flow. When he says he uses periodic conditions at front and back I think he means at both ends of the cylinder.

Yes, I'm very sorry for the miscommunication, my case is not a internal flow but an external flow past a circular cylinder. :D:D:D:D And thank you Simon for your patience to read so long a post.

Quote:

Concerning Lakeat's concerns,

1) Many factors can affect your parallel performance.
First, it depends on the computer you're using. A poor connection between the computing nodes can really slow down the calculation, causing a big difference in ExcutionTime and ClockTime.
"big difference", no, I did not see big difference between the two.
You can see from my last post:

Quote:

Time = 397.927
..................................
ExecutionTime = 164610 s ClockTime = 167106 s
ClockTime-ExecutionTime=167106-164610=2496, and 2496/164610~1.516%.

And 1.516% is not that big, right?

Quote:

A poor connection between the computing nodes
Yes, but this is hardly the case of me, the super-computer I am using have 32 processors per node, even between different nodes, they are connected by what so called infiniband. So I guess this won't be a problem for me, right. But the question is raised here:

Will different mpi implementations like openmpi, mpich, mvapich, make great differences on the speed? Say, if I am decomposing the case to several nodes, and if I am using just one node.

And I remember wiki writes:
Quote:

In computing, wall clock time is the actual time taken by a computer to complete a task. It is the sum of three terms: CPU time, I/O time, and the communication channel delay (e.g. if data are scattered on multiple machines). In contrast to CPU time, which measures only the time during which the processor is actively working on a certain task, wall time measures the total time for the process to complete. The difference between the two consists of time that passes due to programmed delays or waiting for resources to become available.

Quote:

Second, if the connection is good, I've found that the OpenFOAM compilation can greatly influence the computational time. Are you using a pre-compiled OpenFOAM ? If you do, I strongly suggest you compile OpenFOAM locally on the computer you're using for your parallel computations, preferably with the local compiler (such as an Intel compiler).
Really? You know, in fact, I have compared the two, one is pre-compiled (I guess it was compiled on opensuse), the other is compiled one (The super-computer is installed with SUSE), and currently, I did not see compiled one superior to the pre-compiled one. Because I have also recompiled the openmpi, so I am not sure now, why that I did not see the "superior" to the pre-compiled OpenFOAM? Is it because my recompiled openmpi did not work very good...... I have no idea now.

Quote:

Concerning the decomposition, a "simple" decomposition can be used on simple geometries. If you aren't so sure how to decompose it, you should use MeTis.
I hope someone will clear this up, you can see from the official website, new features of version-1.6, it seems scotch is obviously superior to metis, please correct me if I was wrong. Second, for a O-type grid, unless, you decompose it into 4 regions (2*2*1), I still cannot see why simple will work better than metis and scotch method. I one decomposed them into 8 and 16 and 32, and here are the output log files.

simple one:
Quote:

Number of processor faces = 102046
Max number of processor patches = 6
Max number of faces between processors = 10888
metis one:
Quote:

Number of processor faces = 68674
Max number of processor patches = 7
Max number of faces between processors = 5904
You see there is a lot of differences.

Quote:

From my personal experience of unsteady simulations of external flows, I've seen good speedup (linear or better) with as less as 10-15k cells per processor. But that depends on the computer and the case.
Okay, this experience is very helpful, you know.
And for a first try, my cylinder case mesh, 860000/15000~57 processors??!!!! Is this what you meant??


Quote:

I've used the GAMG solver for the pressure equation in many parallel cases and I think it performs well, generally faster than PCG. I don't agree with sandy. How the does fact that it is more complex make it slower ? Did you compare it with other solvers in parallel cases?

Hope this helps
Yes, your information is very very helpful.

PS: pls call me Daniel...

Thank you all.

alberto August 24, 2009 23:14

Hi,

if you do not use a hardware-optimized implementation of the parallel libraries (HP, Intel, ...), you won't notice much difference switching between OpenMPI (which comes from LAM) and MPICH generic libraries. Performance mainly depends on how you compile them and your code.

Quote:

Originally Posted by lakeat (Post 227363)
I hope someone will clear this up, you can see from the official website, new features of version-1.6, it seems scotch is obviously superior to metis, please correct me if I was wrong. Second, for a O-type grid, unless, you decompose it into 4 regions (2*2*1), I still cannot see why simple will work better than metis and scotch method. I one decomposed them into 8 and 16 and 32, and here are the output log files.

A "simple" decomposition works OK if the geometry is not complicated and the mesh is quite uniform. In this way you can decompose the domain by hand in quite a precise way and control the decomposition easily. If the domain is more complicated and the mesh not simple either, an automatic decomposition saves you a lot of time and effort, and gives better results.

Quote:

Okay, this experience is very helpful, you know.
And for a first try, my cylinder case mesh, 860000/15000~57 processors??!!!! Is this what you meant??
Hehe, your sysadmin and colleagues will be happy when you'll run bigger cases. :D:D:D

Best,

Simon Lapointe August 24, 2009 23:37

Quote:

Originally Posted by lakeat (Post 227363)

Yes, but this is hardly the case of me, the super-computer I am using have 32 processors per node, even between different nodes, they are connected by what so called infiniband. So I guess this won't be a problem for me, right. But the question is raised here:

Will different mpi implementations like openmpi, mpich, mvapich, make great differences on the speed? Say, if I am decomposing the case to several nodes, and if I am using just one node.

I did not mean to say a poor interconnection was your problem, but I mentioned it because it is a common cause of slow parallel computations. Since it seems your system has a good interconnect this is probably not the case here.

Concerning the mpi implementations I'm not so sure, since with OpenFOAM I've only used OpenMPI. The best way to answer your question about the number of nodes is to try different combinations of nodes and processors per nodes. The best ratio will depend on the connection speed between the nodes, the connection speed between the processors and the usage of the processors on the different nodes you're using.

Quote:

Originally Posted by lakeat (Post 227363)
Really? You know, in fact, I have compared the two, one is pre-compiled (I guess it was compiled on opensuse), the other is compiled one (The super-computer is installed with SUSE), and currently, I did not see compiled one superior to the pre-compiled one. Because I have also recompiled the openmpi, so I am not sure now, why that I did not see the "superior" to the pre-compiled OpenFOAM? Is it because my recompiled openmpi did not work very good...... I have no idea now.

In my case, it made a huge difference (note I'm using 1.5-dev). With the pre-compiled version, there was a small speedup at 4 or 8 processors, but far from a linear speed. When using more than 8 processors, it was slower and the clocktime was much larger than the cpu time. After being compiled on the supercomputer with the Intel compiler, the speedup is much better.

Which compiler did you use ? Are there different compilers available ? You could try using a different compiler.

Quote:

Originally Posted by lakeat (Post 227363)
I hope someone will clear this up, you can see from the official website, new features of version-1.6, it seems scotch is obviously superior to metis, please correct me if I was wrong. Second, for a O-type grid, unless, you decompose it into 4 regions (2*2*1), I still cannot see why simple will work better than metis and scotch method.

I've never used Scotch so I can't comment on its performances. I didn't say simple would be better than metis or scotch, just that it works well on simple meshes. I suggest you try both MeTis and Scotch and compare your results. Others would be interested in such a comparison. Where did you see that Scotch was "obviously superior to metis" ?

Quote:

Originally Posted by lakeat (Post 227363)
Okay, this experience is very helpful, you know.
And for a first try, my cylinder case mesh, 860000/15000~57 processors??!!!! Is this what you meant??

Yes, this is what I meant. I've split meshes of 750k to 1M cells in 48 or 64 processors and the speedup was very good. But, this is very dependent of the computer used and the case, so I can't promise you anything...Also, as Alberto said, unless the usage of your system is low, it is much more reasonable to use a smaller number of processors (maybe 50k cells/processors). This way you'll still get decent speed and you could use a higher number of processors for cases that really need them.

sandy August 25, 2009 00:21

Quote:

Originally Posted by alberto (Post 227316)
Are you stating GAMG solvers are suitable only for serial calculations? I would not think so.

Hi Alberto, I appreciate so much for your patiently and repeatly directing.

In addtion, maybe I will try GAMG for p equation. In fact, GAMG is solved by ICCG or Block-ICCG (Why not is BiCG?!!! In coarse grid, all matrices are symmetric??!!! Who can explain it??) in coarse grids, and maybe solved by Guass Seidel in the fine grid? So, it should not be difficult to uderstand this method maybe is faster than PCG.

WOW~, really?? GAMG just can be used in solving symmetrical matrix namely p equation?? I just find it!:)

If that, what kinds of Mutil-Grids methods can be used to solve the asymmetrical matrix in OpenFOAM?

@Sorry Daniel, I actually know nothing to parallel. :(

braennstroem August 25, 2009 05:13

Hi Daniel,

another way to accelerate flow development is, to map fluctuations of a 'boxTurb' generated velocity field onto your initial RANS velocities.

Fabian

alberto August 25, 2009 08:51

Quote:

Originally Posted by sandy (Post 227368)
Hi Alberto, I appreciate so much for your patiently and repeatly directing.

WOW~, really?? GAMG just can be used in solving symmetrical matrix namely p equation?? I just find it!:)

There is not such a restriction. Take a look at this document, where in one example (page 15), GAMG is used also for U: http://www.tfd.chalmers.se/~hani/kur...report-fin.pdf

Best,


All times are GMT -4. The time now is 08:15.