CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Installation

OpenFOAM and infiniband

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   October 13, 2008, 12:47
Default Hi all, I need to get OpenFO
  #1
Member
 
Michael Rangitsch
Join Date: Mar 2009
Location: Midland, Michigan, USA
Posts: 31
Rep Power: 8
mrangitschdowcom is on a distinguished road
Hi all,
I need to get OpenFOAM running with our infiniband systems, and can't quite figure out what to do next. I've successfully rebuild the openmpi version in the ThirdParty directory to use our OFED libraries to use the Infiniband connections. What do I need to do next? There is no separate make structure in the Pstream directory in the sources, and I can't quite figure out what else I need to do...

Thanks!


Mike Rangitsch
mrangitschdowcom is offline   Reply With Quote

Old   October 13, 2008, 14:09
Default There is a bit in http://op
  #2
Super Moderator
 
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,416
Rep Power: 16
mattijs is on a distinguished road
There is a bit in

http://openfoamwiki.net/index.php/HowTo_Pstream
mattijs is offline   Reply With Quote

Old   October 24, 2008, 10:30
Default Hi again all, I did a make
  #3
Member
 
Michael Rangitsch
Join Date: Mar 2009
Location: Midland, Michigan, USA
Posts: 31
Rep Power: 8
mrangitschdowcom is on a distinguished road
Hi again all,
I did a make distclean in the OpenMPI directory of the ThirdParty area, then reconfigured using
./configure --with-openib=/usr/local/ofed to point to our ofed installation. The configure and compile worked just fine, as far as I could tell. I then went and recompiled the Pstream directory with Allwmake. interFoam runs on my infiniband nodes now, but I can't tell if it's using infiniband transport or ip over infiniband. It runs slower than running on our Gigabit connected machines (I select the machines just by changing the nodes file), so I think something is still wrong. Will the Pstream still work on both ethernet and infiniband?

Any ideas?


Mike
mrangitschdowcom is offline   Reply With Quote

Old   October 30, 2008, 05:40
Default Hi Michael, did you resolve
  #4
New Member
 
Thomas Gallinger
Join Date: Mar 2009
Posts: 28
Rep Power: 8
thomas is on a distinguished road
Hi Michael,

did you resolve your problem?

I'm struggled with the same - I'm not sure, if Foam takes the infiniband or ethernet connection. I also compiled openmpi with infiniband support.

Best regards
Thomas
thomas is offline   Reply With Quote

Old   October 30, 2008, 07:40
Default Hi, Some hopefully helpful
  #5
Senior Member
 
Jens Klostermann
Join Date: Mar 2009
Posts: 117
Rep Power: 8
jens_klostermann is on a distinguished road
Hi,

Some hopefully helpful coments from my side:

1. Make you sure openmpi is using the infinband interconnect, e.g. with ping-pong test (search for: osu-benchmark) infiniband should be about 3-5 Ás for mesage sizes up to 1024 bytes (Ethernet usually ten times slower)
If openmpi is not running with infiniband look here OpenMPI FAQ for help.
2. OpenMPI uses by default the fastest interconnect at least up to v 1.2.6, so if everything is set up right it should use infiniband by default.
3. It think was Pstream is compiled and linked against OpenMPI it has not to be recompiled as long as you still use OpenMPI as MPI.

Jens
jens_klostermann is offline   Reply With Quote

Old   October 30, 2008, 08:47
Default Jens & Thomas, Still no lu
  #6
Member
 
Michael Rangitsch
Join Date: Mar 2009
Location: Midland, Michigan, USA
Posts: 31
Rep Power: 8
mrangitschdowcom is on a distinguished road
Jens & Thomas,
Still no luck with the infiniband. I get exactly the same timing with infiniband connected nodes as with gigabit connected ones. Fluent seems to run OK on the infiniband nodes, so it looks like ib is working. Also, when I start the job, there is infiniband traffic. It might be using ip over infiniband, I just can't tell. I've been trying to recompile (again) and have run into different problems. It doesn't seem to find the mpi.h headers any more.
I also tried using mpirun with -mca btl ib,self as a parameter, but it wouldn't start up. If I used -mca btl ib,tcp,self it did, because it found the tcp connections (I think).
I haven't run the tests in the openmpi directory, I will this morning.

Mike
mrangitschdowcom is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
HPMPI Infiniband problem carsten OpenFOAM Bugs 2 January 25, 2009 16:36
Infiniband alexander_rudert OpenFOAM 20 January 14, 2009 09:01
infiniband interconnect help rjh FLUENT 0 February 4, 2008 12:41
OpenFOAM 141 parallel results infiniband vs gigabit vs SMP msrinath80 OpenFOAM Running, Solving & CFD 10 November 30, 2007 19:11
Interesting behavior when using LAM without ib rpi on an infiniband cluster OF 141 msrinath80 OpenFOAM Running, Solving & CFD 0 October 2, 2007 16:31


All times are GMT -4. The time now is 02:24.