CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

High system load running OF-2.2 on cluster, but not OF-2.1

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   September 4, 2013, 12:10
Default High system load running OF-2.2 on cluster, but not OF-2.1
  #1
Member
 
Roland
Join Date: Mar 2009
Location: Netherlands
Posts: 62
Rep Power: 8
sylvester is on a distinguished road
Recently we updated our compute cluster from CentOS 5.5 to Debian 7, mostly because we had trouble compiling OpenFOAM 2.2.x with the GCC version available.

However, we are now seeing a large drop-off in performance, but only for cases running OF-2.2.x; OF-2.1.x performs, roughly, as before. We suspect there is a communication problem between the nodes, as jobs using only one (multi-cpu, multi-core) node do not show this behaviour. Gigabit ethernet is used for the interconnect.

Is anyone else experiencing this?

Thanks,
Sylvester

ps. For both OF versions the system GCC was used (version 4.7.2) and the third party openmpi (version 1.5.3 for OF-2.1.x and version 1.6.3 for OF-2.2.x). No changes were made to any OpenFOAM settings or compile flags.
sylvester is offline   Reply With Quote

Old   January 21, 2014, 08:01
Default
  #2
Senior Member
 
Joachim Herb
Join Date: Sep 2010
Posts: 291
Rep Power: 9
jherb is on a distinguished road
Hello Sylvester,

did you solve this problem? Have you tried other MPI libraries?
jherb is offline   Reply With Quote

Old   January 21, 2014, 08:54
Default
  #3
Member
 
Roland
Join Date: Mar 2009
Location: Netherlands
Posts: 62
Rep Power: 8
sylvester is on a distinguished road
Hi Joachim,

Unfortunately we have, despite considerable effort, not been able to solve our problems.

Besides the OpenFOAM supplied MPI, we have tried several system MPI's. None of them showed a difference in performance.

Good luck in your troubles and please let us know if you find a solution.

Best regards,
Sylvester
sylvester is offline   Reply With Quote

Old   January 22, 2014, 07:54
Default
  #4
Senior Member
 
Joachim Herb
Join Date: Sep 2010
Posts: 291
Rep Power: 9
jherb is on a distinguished road
Hi Sylvester,

here is my setup:
Code:
Intel Compiler: icc (ICC) 12.1.0 20111011
SuSE Linux Enterprise 11 SP2 (x86_64)
mvapich2-1.5.1p1-0.6.4
OpenFOAM 2.1.1 and 2.2.2

To test the runtime I used the motorbike tutorial:
OpenFOAM/OpenFOAM-2.2.1/tutorials/incompressible/simpleFoam/motorBike/
(the version of 2.1.1 to test with both OpenFOAM versions, because the newer one does not work with the 2.1.1 version). The solver was simpleFoam (different version for 2.1.1 and 2.2.2)

For both version of OpenFOAM the test case was decomposed into 6 processor domains. The test run was on two nodes of a cluster with each node using 3 processors. The nodes are connected with gigabit ethernet and Infiniband. I think mvapich2 is automatically using Infiniband.

The jobs were started from the queuing system using mpiexec 0.84 from https://www.osc.edu/~djohnson/mpiexec/:
Code:
mpiexec -verbose -comm pmi simpleFoam -parallel
The runtime for OF 2.1.1 was:
ExecutionTime = 232.49 s ClockTime = 243 s

The runtime for OF 2.2.2 was:
ExecutionTime = 229.14 s ClockTime = 239 s

So it looks like everything is ok.

In the past I also used gcc and it seemed to work. But on the other hand, I also noticed sometimes that OpenFOAM hang to whole network of the cluster. Using top a lot of "kworker" and "migration" processes appear using lots of CPU time (on the head and the nodes of the cluster).

I would guess, that in your case there is a problem with the configuration of the networking hardware in combination with configuration problems of MPI. In my case, the above mentioned mpiexec with the ability to specify the communication channel seemed to help. (ok, everything very vague).
jherb is offline   Reply With Quote

Reply

Tags
cluster, debian, mpi, parallel, system load

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
CFX11 + Fortran compiler ? Mohan CFX 20 March 30, 2011 18:56
What is the best cluster file system to go with OF sampaio OpenFOAM Installation 4 March 17, 2011 05:10
High velocity grinding and classification system Sri FLUENT 3 July 30, 2008 10:35
running Fluent on a DUAL CORE system Ralf Schmidt FLUENT 3 June 20, 2006 10:21
Installation on a cluster running a queueing system PBS sampaio OpenFOAM Installation 12 June 20, 2005 17:16


All times are GMT -4. The time now is 12:09.