CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

how to confirm that I have already use infiniband in OpenFOAM?

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By GDTech

Reply
 
LinkBack Thread Tools Display Modes
Old   September 5, 2013, 01:49
Default how to confirm that I have already use infiniband in OpenFOAM?
  #1
New Member
 
Detian Liu
Join Date: Mar 2013
Location: Beijing, P.R.China
Posts: 20
Rep Power: 5
Detian Liu is on a distinguished road
Dear all,
Recently, I am wroking on a case with LES for about 4M mesh, I try to install the infiniband on Ubuntu 12.04 in our cluster, but when I run the case in 48 cores with infiniband connections, I found that it is seems to the be same as I run the case with Ethernet, so is that means I do not use the infiniband at all, or I do not really use ib protocol?
In addition, I installed OpenFOAM use the Ubuntu Deb Pack Installation, the openmpi was also installed automatically, so is that means I should recompile the openmpi again to support infiniband, if so, how can I should do?
Or weather there is some tools to monitor the flux of infiniband, so that I can ensure that I have already use infiniband, because the monitor of Ubuntu could only monitor TCP/IP.
Thanks for your help~
Detian Liu is offline   Reply With Quote

Old   September 5, 2013, 05:42
Default
  #2
Member
 
Laurent Fitschy
Join Date: May 2011
Posts: 39
Rep Power: 7
GDTech is on a distinguished road
Hi,

Quote:
In addition, I installed OpenFOAM use the Ubuntu Deb Pack Installation, the openmpi was also installed automatically, so is that means I should recompile the openmpi again to support infiniband, if so, how can I should do?
Default options for compiling openmpi in thirdparty do not include infiniband support. (see options in ThirdParty-x.x.x/Allwmake)
Check if your openmpi installation has infiniband support with the following command :
Code:
# ompi_info | grep openib
                 MCA btl: openib (MCA v2.0, API v2.0, Component v1.6.3)
If the previous command shows nothing, You have to recompile openmpi. Here is what I added to Allwmake before "./configure " line to get infiniband support :
Code:
configOpt="$configOpt --with-openib --with-psm
Quote:
Or weather there is some tools to monitor the flux of infiniband, so that I can ensure that I have already use infiniband, because the monitor of Ubuntu could only monitor TCP/IP.
It is possible to force mpirun to use a specific protocol for communication.
To force openib protocol :
mpirun --mca btl openib,self -n 4 icoFoam -parallel

To force ethernet protocol
mpirun --mca btl tcp,self -n 4 icoFoam -parallel

Best Regards.
GDTech is offline   Reply With Quote

Old   September 5, 2013, 06:24
Default
  #3
New Member
 
Detian Liu
Join Date: Mar 2013
Location: Beijing, P.R.China
Posts: 20
Rep Power: 5
Detian Liu is on a distinguished road
Dear Laurent,
Thanks for your reply!
As to the problem,when I check the openmpi with: # ompi_info | grep openib
I get the result:
MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.3)
Is that means there is no problem with the openmpi?
Then, the two command:
mpirun --mca btl openib,self -n 4 icoFoam -parallel
mpirun --mca btl tcp,self -n 4 icoFoam -parallel
can both run the case, but the efficiency seems to be the same, is that means for a 4M case in 48 cores, both openib and tcp seem to be similar? So do you have some experience that for how many cores, infiniband could emerge its advantage?
Thanks again for your patient reply~
Best regards!
Detian Liu is offline   Reply With Quote

Old   September 5, 2013, 08:34
Default
  #4
Member
 
Laurent Fitschy
Join Date: May 2011
Posts: 39
Rep Power: 7
GDTech is on a distinguished road
Quote:
I get the result:
MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.3)
yes it means you have infiniband support.

Quote:
Then, the two command:
mpirun --mca btl openib,self -n 4 icoFoam -parallel
mpirun --mca btl tcp,self -n 4 icoFoam -parallel
can both run the case, but the efficiency seems to be the same
It is possible to configure infiniband network in order to make tcp/ip communication passing trough infiniband (called IP over IB). In that case, you won't see any significant difference between the two previous commands. You can check if IP over IB is enabled on your cluster with this command :
Code:
lsmod | grep ipoib
If that command gives you an output, IP over IB is enabled.
jploz likes this.
GDTech is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
OpenFOAM - Validation of Results Ahmed OpenFOAM Running, Solving & CFD 9 June 22, 2011 18:59
Issues with OpenFoam sanjibdsharma OpenFOAM 0 August 14, 2009 08:41
64bitrhel5 OF installation instructions mirko OpenFOAM Installation 2 August 12, 2008 18:07
How to confirm that OpenFoam runs in 64 bit jdk OpenFOAM Running, Solving & CFD 2 February 5, 2008 05:55
The OpenFOAM extensions project mbeaudoin OpenFOAM 16 October 9, 2007 09:33


All times are GMT -4. The time now is 13:38.