To parallel or not ...
 

I have been thinking about this issue and I need some opinions about it.

The issue is this, lets say that i have a unstructured grid based segregated solver. This solver solves for turbulence and heat transfer problems.

The issue to decide is this, this solver is single processor solver and i need to make it parallel version of it. But at the same time I could spend my time adding more interesting features to it. For example lets say chimera type of mesh support or immersed boundary type of support to it.

I can spend my time doing one of these tasks and i could not decide where to put my efforts.

So the question is what do you guys think is parallel solver is too much needed for people who use cfd or they would rather have a solver that provide something that other solvers do not provide.

For example Fluent CFX, starCCM , openFOAM etc do not provide immersed boundary type of solver. So i think it would be great to have.

Any input would be helpful.

Re: To parallel or not ...
 

You should do what you need...

I think that this is the moment to invest in parallel capability rather than other stuff. The reason is that multicore machines are rapidly becoming common and cheaper and cheaper.

You can always spend some time later in adding more nice features as immersed boundary or chimera capability (or whatelse) but i think that a parallel capability is no more avoidable in CFD (maybe also in students projects).

If you spend your time in something different...the more the serial stuff the more will be the work to go in the parallel direction.

But, as stated before, you should do what you need...

Re: To parallel or not ...
 

One thought to consider here is that if you are definitely going to add overset capability, it might make sense to do that first, and add parallel later. There is scope for relatively simple parallelisation, where each overset domain logically runs on its own processor.

Re: To parallel or not ...
 

rather than adding an overset capablity i have added immersed boundary type solver. And which is working fine. (I have compared results with polyflow and so far results are good).

So the solver (at the moment) works in two modes:

1: simple unstructured grid solver with algo same as usual segregated solver.

2: coupled with immersed boundary method - in this mode, one creates a base cartesian type block. (this could have non uniform and irregular distribution of nodes) and on the top of it, user can put solids regions, which can move freely, so in a way one can solve solids moving in fluid easily.

(i have been checking turbulence modules in second mode, and last thing to add is heat transfer module in 2 mode).

The whole issue is since it was not written thinking a parallel code, i need to start again separately and write new code using part of this already written code.

When i say add new features in already written code, i mean IBM, and other modules like VOF, noise, structure fluid interaction etc etc.

One thing to remember is the most difficult part of parallel code is to get AMG solver working well. Which works well in single processor mode. Plus how to handle sliding interfaces in parallel mode. All this is available in single processor version.

Sometimes i think shared memory based parallisation would make more sense.

Re: To parallel or not ...
 

Hi,

I believe there is also the speed concern. If your simulation takes a long time, then u should do it, and asap since modification later on as u add more capabilities will be less.

However, if it only takes a few hr per runtime, I don't think it is worth the effort

Re: To parallel or not ...
 

Well, I think there is a lot to be said for SMP. The fact is that it is not expensive now to put together an 8-way SMP machine.

Re: To parallel or not ...
 

Why not use existing linear algebra libraries. PETSc would give you a uniform interface (change solvers and all parameters on the command line) to lots of iterative methods and preconditioners/direct solvers (including ML and BoomerAMG which work very well in parallel).

Shared memory parallel isn't nearly what it's cracked up to be for scientific codes. For instance sparse matrix-vector product is an important kernel which can dominate the runtime of a linear solve, but it is entirely memory bandwidth limited. The FPU is normally at about 20% which means that adding more cores won't help unless thay have their own bus.

Parallel coding isn't much harder than sequential, there are just different things to think about.

With regard to immersed boundary, how do you find the conditioning compares to fixed boundaries? Are you using domain decomposition/Schur complement preconditioners or is a normal preconditioner sufficient?

Re: To parallel or not ...
 

hi, thanks for posting.

Yes, the idea of using existing liberaries is also possible, but for me, there are two issues, one if i use external solver than i have to change data structure that suits that solver. Second, matrix solver is just a part of program. (albeit most important). There are many tasks those have to be done in parallel, for me the basic structure of code is intgrated with solver i have written. So i am most reluctant to change it.

Further, i want to avoid using external liberaries as possible, at the moment you can compile the code with any c++ compiler, it won't complain that this lib is not there or that is not there. (I tried installing openfoam and could not make it work, it always tells me this is not working or there or that has issues, i just hate this thing).

I have no experience with shared memory based parallel program, so i think i would go with what you said. (the most important thing about shared memory programming is it is easier).

Having said all this, i think you are correct that mpi based programming is not so difficult. It just need more planning. And this is what i have done in last few days. So i can implement it now (in fact already started).

About immersed boundary method, since i have control over matrix solver it is very easy to take care.

In immersed boundary , we force u,v,w to follow values of solid region. It is very easy to implement, first replace, variable values in matrix (X vector), now modify source for these terms so that u = (sum al u) /ap , gives you u = u-solid. Since we have already replaced, original u by solid boundaries, this shall give you zero residual, and hence this portion is already done. Solve linear equations in a usual way. (preconditioing does not make much difference, because of reason i said ). Finally when come out of matrix solver, just replace the solid region values again by values to be forced. This keeps solution very stable.

However, pressure equation shall be solved, without doing any modifications. Since for pressure equation, source is due to mass flux and it shall adjust itself to immersed boundaries. (Since pressure equation is not aware of any solid regions, the preconditioner does not affect here).

(I have not read about this way of doing it in any paper, but this is the most stable implentation of immersed boundary method so far i have witnessed).