CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

IcoFoam in parallel Issue with speed up

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By skabilan

Reply
 
LinkBack Thread Tools Display Modes
Old   July 24, 2008, 03:28
Default Heya, Your setup looks OK.
  #1
Senior Member
 
Hrvoje Jasak
Join Date: Mar 2009
Location: London, England
Posts: 1,758
Rep Power: 21
hjasak will become famous soon enough
Heya,

Your setup looks OK. Try playing with the pressure solver, starting from ICCG and then varying the AMG parameters if that does not help. You should get good performance at least up to 100 CPUs.

Please keep us (The Forum) posted in case there's a real problem.

Hrv
__________________
Hrvoje Jasak
Providing commercial FOAM/OpenFOAM and CFD Consulting: http://wikki.co.uk
hjasak is offline   Reply With Quote

Old   July 24, 2008, 04:23
Default Hi Senthil! Of course the o
  #2
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,915
Rep Power: 40
gschaider will become famous soon enoughgschaider will become famous soon enough
Hi Senthil!

Of course the other interesting question is: how big is your case?
And how are your CPU's connected.

If for instance you're doing the damBreak over 100MBit ethernet, I'd be surprised if you see any speedup on 6 CPUs

Bernhard
__________________
Note: I don't use "Friend"-feature on this forum out of principle. Ah. And by the way: I'm not on Facebook either. So don't be offended if I don't accept your invitation/friend request
gschaider is offline   Reply With Quote

Old   July 24, 2008, 12:05
Default Hi Hrv, Thanks for the help
  #3
Senior Member
 
Senthil Kabilan
Join Date: Mar 2009
Posts: 113
Rep Power: 8
skabilan is on a distinguished road
Hi Hrv,

Thanks for the help! Ill try your suggestions and keep the forum posted.

Bernhard: The mesh has 2.7 million elements (hybrid). Its a shared memory machine.

Regards,
Senthil
skabilan is offline   Reply With Quote

Old   July 24, 2008, 12:26
Default What kind of shared memory mac
  #4
Senior Member
 
Eugene de Villiers
Join Date: Mar 2009
Posts: 725
Rep Power: 12
eugene is on a distinguished road
What kind of shared memory machine?
eugene is offline   Reply With Quote

Old   July 24, 2008, 12:44
Default Hi Eugene, Its Altix from S
  #5
Senior Member
 
Senthil Kabilan
Join Date: Mar 2009
Posts: 113
Rep Power: 8
skabilan is on a distinguished road
Hi Eugene,

Its Altix from SGI (http://www.sgi.com/products/servers/altix/)

The system runs a single copy of Linux over 128 Intel Itanium2 processors running at 1.5GHz. The system has 256GB of RAM of shared memory.

Thanks
Senthil
skabilan is offline   Reply With Quote

Old   July 24, 2008, 13:13
Default Hi Senthil, I have some bad e
  #6
Senior Member
 
Francesco Del Citto
Join Date: Mar 2009
Location: Zürich Area, Switzerland
Posts: 215
Rep Power: 9
fra76 is on a distinguished road
Hi Senthil,
I have some bad experience with some SGI shared memory machine, where the memory is distributed on different motherboards and shared via kernel on top of an Infiniband network. Is this the case? I think so, as they write that the Altix has a "Modular blade design".
If so, you can experience very poor performance if the computer is not configure for the software you are running...

Francesco
fra76 is offline   Reply With Quote

Old   July 24, 2008, 15:42
Default Hi Francesco, The same mach
  #7
Senior Member
 
Senthil Kabilan
Join Date: Mar 2009
Posts: 113
Rep Power: 8
skabilan is on a distinguished road
Hi Francesco,

The same machine did demonstrate excellent speed up in case of a steady state simulation with simpleFoam. I got good speed up with 64 processors.

Senthil
skabilan is offline   Reply With Quote

Old   July 25, 2008, 05:25
Default When running on Itanium it is
  #8
Senior Member
 
Eugene de Villiers
Join Date: Mar 2009
Posts: 725
Rep Power: 12
eugene is on a distinguished road
When running on Itanium it is crucial that you use the Intel Icc compiler and the SGI native MPI otherwise performance will be poor.
eugene is offline   Reply With Quote

Old   July 25, 2008, 18:17
Default Hi All, When I increase the
  #9
Senior Member
 
Senthil Kabilan
Join Date: Mar 2009
Posts: 113
Rep Power: 8
skabilan is on a distinguished road
Hi All,

When I increase the number of processors, the solver stalls while calculating the pressure equation. I think it is the problem with GAMG. Any suggestions changing the following parameters?

p GAMG
{
agglomerator faceAreaPair;
nCellsInCoarsestLevel 100;
cacheAgglomeration true;
directSolveCoarsest true;
nPreSweeps 1;
nPostSweeps 2;
nFinestSweeps 2;
tolerance 1e-05;
relTol 0.1;
smoother GaussSeidel;
mergeLevels 1;
minIter 0;
maxIter 10;
};

Thanks
Senthil
skabilan is offline   Reply With Quote

Old   August 13, 2008, 17:45
Default Hi All, I have tried variou
  #10
Senior Member
 
Senthil Kabilan
Join Date: Mar 2009
Posts: 113
Rep Power: 8
skabilan is on a distinguished road
Hi All,

I have tried various permutations and combinations for the pressure solver but I could not attain any speed up. Any Suggestions?

Thanks
Senthil
skabilan is offline   Reply With Quote

Old   October 3, 2011, 00:09
Default
  #11
Senior Member
 
Senthil Kabilan
Join Date: Mar 2009
Posts: 113
Rep Power: 8
skabilan is on a distinguished road
Hi All,

This issue has been resolved! The key to this problem was changing nCellsInCoarsestLevel in GAMG setting.

Thanks
Senthil
skabilan is offline   Reply With Quote

Old   October 3, 2011, 05:03
Default
  #12
Member
 
Tibor Nyers
Join Date: Jul 2010
Location: Hungary
Posts: 91
Rep Power: 8
Toorop is on a distinguished road
Hi Senthil,

could you elaborate a bit on the matter please? I have made some parallel tests and used icoFoam with GAMG and the scale up was far from perfect.

Thanks in advance!
Toorop is offline   Reply With Quote

Old   October 3, 2011, 13:04
Default
  #13
Senior Member
 
Senthil Kabilan
Join Date: Mar 2009
Posts: 113
Rep Power: 8
skabilan is on a distinguished road
Hi,

First, it is impossible to get linear speedup with any code. That said, icoFoam, performed relatively good on multiple processors. When you increase the number of processors, each processor gets a smaller chunk of the mesh. Therefore, you need to decrease the nCellsInCoarsestLevel settings in GAMG (fvSolution file). You can think of this number as the number of common cells shared between two processors.
Also, OpenFOAM performed better with GNU OpenMPI 1.5 compared to the previous version (1.4).

Thanks
wyldckat likes this.
skabilan is offline   Reply With Quote

Old   October 10, 2011, 11:06
Default
  #14
Member
 
Tibor Nyers
Join Date: Jul 2010
Location: Hungary
Posts: 91
Rep Power: 8
Toorop is on a distinguished road
Thx Senthil for the tip, but it wont change anything in my case.

My test was a simple cavity3D created with blockMesh with 10M cells. Default icoFoam solver with GAMG for pressure on a Intel Xeon E5430. Iteration count 200.
parallel thread - runtime
1 - 32,2 h
2 - 27.7 h
4 - 28.9 h
8 - 23.8 h
Toorop is offline   Reply With Quote

Old   October 10, 2011, 13:10
Default
  #15
Senior Member
 
Senthil Kabilan
Join Date: Mar 2009
Posts: 113
Rep Power: 8
skabilan is on a distinguished road
Hi,

Which version of OpenFOAM are you using?
skabilan is offline   Reply With Quote

Old   October 11, 2011, 02:41
Default
  #16
Senior Member
 
akidess's Avatar
 
Anton Kidess
Join Date: May 2009
Location: Delft, Netherlands
Posts: 919
Rep Power: 17
akidess will become famous soon enough
Quote:
Originally Posted by skabilan View Post
Hi,
First, it is impossible to get linear speedup with any code.
This is commonly true, but NOT impossible. See superlinear speedup.

Quote:
Therefore, you need to decrease the nCellsInCoarsestLevel settings in GAMG (fvSolution file). You can think of this number as the number of common cells shared between two processors.
I believe this is not true, because processor coupling still happens on the finest level, not the coarsest (please correct me if I'm wrong)! Also, if you set nCellsInCoarsestLevel too low, you'll spend more time on interpolation and restriction than you gain from the coarse level solution.
akidess is offline   Reply With Quote

Old   October 11, 2011, 03:14
Default
  #17
Member
 
Tibor Nyers
Join Date: Jul 2010
Location: Hungary
Posts: 91
Rep Power: 8
Toorop is on a distinguished road
Hi,
OS: Ubuntu 11.04
OF: a month old 2.0.x
Toorop is offline   Reply With Quote

Old   October 11, 2011, 13:02
Default
  #18
Senior Member
 
Senthil Kabilan
Join Date: Mar 2009
Posts: 113
Rep Power: 8
skabilan is on a distinguished road
Anton,

You are correct! I need to rephrase my wordings. I meant we cannot get linear speedup with OpenFoam utilities (at least the ones I have used). We will run into the interpolation issues if we have the nCellsInCoarsestLevel set to a very small number. It has to appropriate with the total number of elements on each processor.
skabilan is offline   Reply With Quote

Old   October 11, 2011, 13:20
Default
  #19
Senior Member
 
akidess's Avatar
 
Anton Kidess
Join Date: May 2009
Location: Delft, Netherlands
Posts: 919
Rep Power: 17
akidess will become famous soon enough
Personally I've always encountered lower than linear speedup as well, but some people have claimed to get close to linear or better speedup using OpenFOAM:
http://www.hpcadvisorycouncil.com/pd...M_at_Scale.pdf

http://web.student.chalmers.se/group...SlidesOFW5.pdf
akidess is offline   Reply With Quote

Old   October 11, 2011, 17:07
Default
  #20
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,301
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all!

I would like to add to the knowledge being shared here and point you all to the following post: Parallel processing of OpenFOAM cases on multicore processor??? post #11 - it's a bit of a rant, but several scalability notions I've gotten with OpenFOAM and have gathered over time are written there.

As for super-linear scalability, AFAIK it's only a half-truth: it's only super linear when we don't take into account the timings in serial mode! In other words, we can observe something similar on the "Report" I mention on that post above, where scalability sky-rockets when we don't take into account the serial run and the real theoretical speedup that should be expected.

I've been trying to gather as much information on this subject as possible and have been keeping track of stuff on the blog post mentioned at the end of the post I mention above, namely: Notes about running OpenFOAM in parallel

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with IcoFoam in parallel skabilan OpenFOAM Running, Solving & CFD 12 April 1, 2008 05:55
IcoFoam parallel woes msrinath80 OpenFOAM Running, Solving & CFD 9 July 22, 2007 02:58
Parallel with Windoze, speed difference between PV Charles CFX 3 March 10, 2005 02:25
Parallel speed up Soren CFX 18 May 31, 2002 12:26
Parallel speed up for CFX 5 on PC's Roued CFX 6 November 28, 2001 19:02


All times are GMT -4. The time now is 16:38.