CFD Online URL
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

AMG Solver in parallel

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   December 5, 2006, 06:15
Default Hi all! I've tried to use the
  #1
Senior Member
 
Francesco Del Citto
Join Date: Mar 2009
Location: Zürich Area, Switzerland
Posts: 209
Rep Power: 8
fra76 is on a distinguished road
Hi all!
I've tried to use the AMG solver in a parallel, simpleFoam case single precision, but it seems to be really very slow, with a slow load for CPU during pressure solving, and "501" iteration for each SIMPLE iteration.
I used "AMG 1.e-6 0 100".
I switched to ICCG and everything seems ok, now.

Furthermore, is it possible to use a fast network instead of the ethernet? If yes, shoud I recompile lamport MPI or what?

Francesco
fra76 is offline   Reply With Quote

Old   December 5, 2006, 06:29
Default It probably means you've messe
  #2
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,748
Rep Power: 20
hjasak will become famous soon enough
It probably means you've messed up your discretisation or boundary conditions. The solver is "slow" because it is doing a maximum number of iterations without converging, meaning that something in your mesh or discretisation setp is bothering it.

Since I wrote the solver, I wouldn't mind trying it out (if the case is public).

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   December 5, 2006, 06:49
Default Sorry, a few more questions:
  #3
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,748
Rep Power: 20
hjasak will become famous soon enough
Sorry, a few more questions:
- can you plot the residual history for the solver (go to ~/.OpenFOAM-1.3/controlDict and set the debug switch for lduMatrix to 2. This will give you a residual for every iteration). It may be useful to show this for ICCG as well
- how big is this case?
- you are running single precision and converging to 1e-6. The round-off error at single precision will be around 1e-7 (times the number of equations for the residual). Is that your problem?

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   December 5, 2006, 09:05
Default Hi Hrvoje! Thanks for your re
  #4
Senior Member
 
Francesco Del Citto
Join Date: Mar 2009
Location: Zürich Area, Switzerland
Posts: 209
Rep Power: 8
fra76 is on a distinguished road
Hi Hrvoje!
Thanks for your reply!
At first, I've tried to increase the tolerance of the solver (up to 1e-5). Seems to be better.
First iteration: 501 + 17 (1 non-orthogonal corrector)
Second: 501 15
Third: 21 11
Fourth: 24 15
...
It came back to 501 during 6th and 7th iteration, but I'm letting it go.

I'll make some tests, single and double precision, with debug activated, and I'll let you know something!

The size of the mesh, however, is a few millions cells, on 16 processes.
fra76 is offline   Reply With Quote

Old   December 5, 2006, 09:10
Default P.S. What it's strange, in te
  #5
Senior Member
 
Francesco Del Citto
Join Date: Mar 2009
Location: Zürich Area, Switzerland
Posts: 209
Rep Power: 8
fra76 is on a distinguished road
P.S.
What it's strange, in terms of performances, is this (after 10 iterations):
ExecutionTime = 398.07 s ClockTime = 1321 s
fra76 is offline   Reply With Quote

Old   December 5, 2006, 09:13
Default I am just writing a paper of v
  #6
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,748
Rep Power: 20
hjasak will become famous soon enough
I am just writing a paper of very fast solvers, containing some considerable new work. :-)

Incidentally, do you have a particularly bad communications on your parallel machine? BTW, I would still like to see a residual graph if possible.

Thanks,

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   December 6, 2006, 05:47
Default The inefficiency seems to be r
  #7
Senior Member
 
Francesco Del Citto
Join Date: Mar 2009
Location: Zürich Area, Switzerland
Posts: 209
Rep Power: 8
fra76 is on a distinguished road
The inefficiency seems to be related to the network interface.
In fact, running the same simulation on 4 processors (all on the same computational node, so that network is not used at all), the difference between executionTime and clockTime is almost zero.

BTW, the solver seems to be much more robust in double precision than in single. I started from the solution provided by potentialFoam, with these settings in fvSolution:

solvers
{
p AMG 1e-09 0 100;
U BICCG 1e-09 0.1;
k BICCG 1e-09 0.1;
epsilon BICCG 1e-09 0.1;
R BICCG 1e-09 0.1;
nuTilda BICCG 1e-09 0.1;
}

SIMPLE
{
nNonOrthogonalCorrectors 1;
pRefCell 0;
pRefValue 0;
}

And I got:
Selecting incompressible transport model Newtonian
Selecting turbulence model realizableKE

Starting time loop

Time = 1

BICCG: Solving for Ux, Initial residual = 0.20082417, Final residual = 0.0090950718, No Iterations 1
BICCG: Solving for Uy, Initial residual = 0.2947727, Final residual = 0.010673377, No Iterations 1
BICCG: Solving for Uz, Initial residual = 0.29221236, Final residual = 0.011575026, No Iterations 1
AMG: Solving for p, Initial residual = 1, Final residual = 9.012947e-10, No Iterations 44
AMG: Solving for p, Initial residual = 0.28980045, Final residual = 6.1152146e-10, No Iterations 35
time step continuity errors : sum local = 1.13472e-08, global = -4.5350984e-10, cumulative = -4.5350984e-10
BICCG: Solving for epsilon, Initial residual = 0.00020144714, Final residual = 1.2768634e-06, No Iterations 1
BICCG: Solving for k, Initial residual = 0.99999999, Final residual = 0.008439019, No Iterations 1
ExecutionTime = 186.14 s ClockTime = 190 s

Time = 2

BICCG: Solving for Ux, Initial residual = 0.17720949, Final residual = 0.013158181, No Iterations 1
BICCG: Solving for Uy, Initial residual = 0.25730573, Final residual = 0.011324291, No Iterations 1
BICCG: Solving for Uz, Initial residual = 0.090200218, Final residual = 0.0075883116, No Iterations 1
AMG: Solving for p, Initial residual = 0.51558964, Final residual = 4.655188e-10, No Iterations 44
AMG: Solving for p, Initial residual = 0.17652435, Final residual = 9.4423534e-10, No Iterations 33
time step continuity errors : sum local = 1.455791e-08, global = -7.7483077e-10, cumulative = -1.2283406e-09
BICCG: Solving for epsilon, Initial residual = 0.00014428334, Final residual = 8.4038709e-07, No Iterations 1
BICCG: Solving for k, Initial residual = 0.029816968, Final residual = 0.00022590526, No Iterations 1
ExecutionTime = 348.5 s ClockTime = 352 s


I'll generate the residuals for the single precision case as soon as possible.

BTW, there is a way of use the fast network interconnection I have, instead of the standard ethernet, so that I can speedup the parallel AMG solver?
fra76 is offline   Reply With Quote

Old   December 6, 2006, 06:37
Default This is good news. Incidental
  #8
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,748
Rep Power: 20
hjasak will become famous soon enough
This is good news. Incidentally, round-off error pollution will be a problem in your case with single precision. What should be done is to keep x and residual in double precision; the rest of the software can be kept single precision. Since you've got the full source, you should be able to do this on your own.

For my personal pleasure, I would always run in double precision and not worry about round-off.

By the way, you are running SIMPLE and converging the pressure equation to 1e-10 every time, which is a massive waste of time. You can get away with converging the pressure equation to 0.05 or even 0.1 and you will save 80% in CPU time. Definitely worth playing with. Also, there's no point in keeping the solver tolerance at 1e-9 - 1e-6 will almost certainly do:

p AMG 1e-06 0.05 100;

Please keep me posted - it would be nice to hear you say all is well with the solver for future generations to see. :-)

Enjoy,

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   December 13, 2006, 09:31
Default Hi, Reading Hrvoje's commen
  #9
Member
 
Ola Widlund
Join Date: Mar 2009
Location: Sweden
Posts: 87
Rep Power: 7
olwi is on a distinguished road
Hi,

Reading Hrvoje's comment he has a paper brewing with new solver algorithms, I get very curious... I look forward to reading the full paper in due time, I'm sure it will be good stuff.

Just a few general questions, concerning speed and solver efficiency.
1. If I do a profiling of a standard high-level solver, like turbFoam, how much of the time would be spent in the linear solvers? Is the setting up of the matrices (the code in the high-level solvers) a big part?
2. Is there a potential for making any of the linear solver's more efficient, or implementing other solvers than those in the code today? (I'm suspecting that Hrvoje's new exciting stuff is probably not simply a new linear solver, but more on the solution algorithm as a whole)
3. Is there a big overhead due to the data structures, and the polyhedral capabilities?

Best regards,
Ola
olwi is offline   Reply With Quote

Old   December 13, 2006, 09:46
Default Heya, 1) You should be spen
  #10
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,748
Rep Power: 20
hjasak will become famous soon enough
Heya,

1) You should be spending 50-80% of total execution time in the solvers. When you do a profile, the linear solvers should be first by a long way, followed by the velocity gradient and then other minor operations. The first four items ofn the list should bring you over 90% of total time - tells you a lot about the code and algorithm.

2) I just did it. The new solver is just that - a solver: it's only that it's three times faster than anything I've ever seem before. No cheating, no being selective in the test cases, no being economical with the truth or similar. The paper has been submitted to a conference, we'll see what will come out of it. If you wish to test it and are serious about using it, drop me a line.

3) Polyhedral mesh handling actually reduces execution time - it is the most beautiful (read: efficient) way of dealing with an FVM mesh. As for the rest, have a look at run-time and memory consumption comparisons vs other CFD software and "make your own judgement".

Enjoy,

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   January 18, 2007, 10:57
Default Hi prof. Jasak, Today I qu
  #11
Senior Member
 
Frank Bos
Join Date: Mar 2009
Location: The Netherlands
Posts: 338
Rep Power: 8
lr103476 is on a distinguished road
Hi prof. Jasak,

Today I quickly read your papers (from your site) about extrapolated / preconditioned iterative solvers and I am impressed. Will those new solver variants also be present in the new OpenFOAM 1.4?
If so, when do you plan to release the new version? Since I am doing very computational intensive simulations, 3D moving meshes, a big gain in calculation time can be obtained.

Looking forward to it...

Regards, Frank
__________________
Frank Bos
lr103476 is offline   Reply With Quote

Old   January 18, 2007, 15:21
Default Yeah, it does look pretty cool
  #12
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,748
Rep Power: 20
hjasak will become famous soon enough
Yeah, it does look pretty cool, doesn't it? :-) I haven't been expecting such a great improvement in performance but it looks like surprises do indeed exist.

The work will be presented on 15th Annual Conference of the CFD Society of Canada, Toronto, Ontario, Canada, May 27-31, 2007. Why don't you come over to Zagreb to the OpenFOAM Workshop, Jun/2007 and we can talk about it - it is just after the Toronto meeting. In any case, it would be really nice to see some of your work because you've been quite busy over the last year and there will be a session dedicated to fluid-structure interaction.

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   January 18, 2007, 18:48
Default Yes, it looks really cool and
  #13
Senior Member
 
Frank Bos
Join Date: Mar 2009
Location: The Netherlands
Posts: 338
Rep Power: 8
lr103476 is on a distinguished road
Yes, it looks really cool and promising. I need to study the theories in more detail to really understand what's happening.

I was already considering a visit to the next OpenFOAM workshop. If I will come is more of a time issue. It depends on my simulations and I hope to have some respectable results by then. Your new solver techniques may improve the speed of my simulations:-)

Could you also please give some comments on my problems concerning parallel computations with dynamicBodyFvMesh using more than 2 processors in the other tread.

Regards, Frank
__________________
Frank Bos
lr103476 is offline   Reply With Quote

Old   February 22, 2007, 03:38
Default Hi prof. Jasak, Just to keep
  #14
Senior Member
 
Francesco Del Citto
Join Date: Mar 2009
Location: Zürich Area, Switzerland
Posts: 209
Rep Power: 8
fra76 is on a distinguished road
Hi prof. Jasak,
Just to keep you updated, I was able to recompile the communication library so that I can run OpenFOAM on the extremely fast interconnection we have.
Everything is now astonishing quick, and I've measured non linear speedup, on a 3.7 mil cells mesh, double precision, up to 64 processors (efficiency=2.19, for sake of precision)!
I'm still playing around with single and double precision. Single is faster, but it seems that you need more iteration to converge. However, it's hard to say, as the solvers tolerances have to be different.

What about the new solvers you mentioned? Will you distribute them with OF 1.4?

Thanks a lot for all the suggestions!
Francesco
fra76 is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
"Define_Profile" UDF for parallel solver Antoine Fluent UDF and Scheme Programming 7 February 26, 2013 05:22
Events and Parallel Solver Duarte Albuquerque CD-adapco 4 July 20, 2008 15:46
UDF problem with Parallel Solver manu FLUENT 0 January 24, 2008 15:31
Parallel solver with 3.2 robbie CD-adapco 1 August 10, 2004 08:29
3D PARALLEL SOLVER (MPI Lib) Saud Khashan Main CFD Forum 8 July 14, 1999 09:52


All times are GMT -4. The time now is 14:01.