CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM (http://www.cfd-online.com/Forums/openfoam/)
-   -   how to confirm that I have already use infiniband in OpenFOAM? (http://www.cfd-online.com/Forums/openfoam/123127-how-confirm-i-have-already-use-infiniband-openfoam.html)

Detian Liu September 5, 2013 01:49

how to confirm that I have already use infiniband in OpenFOAM?
 
Dear all,
Recently, I am wroking on a case with LES for about 4M mesh, I try to install the infiniband on Ubuntu 12.04 in our cluster, but when I run the case in 48 cores with infiniband connections, I found that it is seems to the be same as I run the case with Ethernet, so is that means I do not use the infiniband at all, or I do not really use ib protocol?
In addition, I installed OpenFOAM use the Ubuntu Deb Pack Installation, the openmpi was also installed automatically, so is that means I should recompile the openmpi again to support infiniband, if so, how can I should do?
Or weather there is some tools to monitor the flux of infiniband, so that I can ensure that I have already use infiniband, because the monitor of Ubuntu could only monitor TCP/IP.
Thanks for your help~

GDTech September 5, 2013 05:42

Hi,

Quote:

In addition, I installed OpenFOAM use the Ubuntu Deb Pack Installation, the openmpi was also installed automatically, so is that means I should recompile the openmpi again to support infiniband, if so, how can I should do?
Default options for compiling openmpi in thirdparty do not include infiniband support. (see options in ThirdParty-x.x.x/Allwmake)
Check if your openmpi installation has infiniband support with the following command :
Code:

# ompi_info | grep openib
                MCA btl: openib (MCA v2.0, API v2.0, Component v1.6.3)

If the previous command shows nothing, You have to recompile openmpi. Here is what I added to Allwmake before "./configure " line to get infiniband support :
Code:

configOpt="$configOpt --with-openib --with-psm
Quote:

Or weather there is some tools to monitor the flux of infiniband, so that I can ensure that I have already use infiniband, because the monitor of Ubuntu could only monitor TCP/IP.
It is possible to force mpirun to use a specific protocol for communication.
To force openib protocol :
mpirun --mca btl openib,self -n 4 icoFoam -parallel

To force ethernet protocol
mpirun --mca btl tcp,self -n 4 icoFoam -parallel

Best Regards.

Detian Liu September 5, 2013 06:24

Dear Laurent,
Thanks for your reply!
As to the problem,when I check the openmpi with: # ompi_info | grep openib
I get the result:
MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.3)
Is that means there is no problem with the openmpi?
Then, the two command:
mpirun --mca btl openib,self -n 4 icoFoam -parallel
mpirun --mca btl tcp,self -n 4 icoFoam -parallel
can both run the case, but the efficiency seems to be the same, is that means for a 4M case in 48 cores, both openib and tcp seem to be similar? So do you have some experience that for how many cores, infiniband could emerge its advantage?
Thanks again for your patient reply~
Best regards!

GDTech September 5, 2013 08:34

Quote:

I get the result:
MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.3)
yes it means you have infiniband support.

Quote:

Then, the two command:
mpirun --mca btl openib,self -n 4 icoFoam -parallel
mpirun --mca btl tcp,self -n 4 icoFoam -parallel
can both run the case, but the efficiency seems to be the same
It is possible to configure infiniband network in order to make tcp/ip communication passing trough infiniband (called IP over IB). In that case, you won't see any significant difference between the two previous commands. You can check if IP over IB is enabled on your cluster with this command :
Code:

lsmod | grep ipoib
If that command gives you an output, IP over IB is enabled.


All times are GMT -4. The time now is 00:46.