[Detian Liu]

Dear all,
Recently, I am wroking on a case with LES for about 4M mesh, I try to install the infiniband on Ubuntu 12.04 in our cluster, but when I run the case in 48 cores with infiniband connections, I found that it is seems to the be same as I run the case with Ethernet, so is that means I do not use the infiniband at all, or I do not really use ib protocol?
In addition, I installed OpenFOAM use the Ubuntu Deb Pack Installation, the openmpi was also installed automatically, so is that means I should recompile the openmpi again to support infiniband, if so, how can I should do?
Or weather there is some tools to monitor the flux of infiniband, so that I can ensure that I have already use infiniband, because the monitor of Ubuntu could only monitor TCP/IP.
Thanks for your help~

[GDTech]

Hi,

In addition, I installed OpenFOAM use the Ubuntu Deb Pack Installation, the openmpi was also installed automatically, so is that means I should recompile the openmpi again to support infiniband, if so, how can I should do?

Default options for compiling openmpi in thirdparty do not include infiniband support. (see options in ThirdParty-x.x.x/Allwmake)
Check if your openmpi installation has infiniband support with the following command :

# ompi_info | grep openib
                 MCA btl: openib (MCA v2.0, API v2.0, Component v1.6.3)

If the previous command shows nothing, You have to recompile openmpi. Here is what I added to Allwmake before "./configure " line to get infiniband support :

configOpt="$configOpt --with-openib --with-psm

Or weather there is some tools to monitor the flux of infiniband, so that I can ensure that I have already use infiniband, because the monitor of Ubuntu could only monitor TCP/IP.

It is possible to force mpirun to use a specific protocol for communication.
To force openib protocol :
mpirun --mca btl openib,self -n 4 icoFoam -parallel

To force ethernet protocol
mpirun --mca btl tcp,self -n 4 icoFoam -parallel

Best Regards.

[Detian Liu]

Dear Laurent,
Thanks for your reply!
As to the problem,when I check the openmpi with: # ompi_info | grep openib
I get the result:
MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.3)
Is that means there is no problem with the openmpi?
Then, the two command:
mpirun --mca btl openib,self -n 4 icoFoam -parallel
mpirun --mca btl tcp,self -n 4 icoFoam -parallel
can both run the case, but the efficiency seems to be the same, is that means for a 4M case in 48 cores, both openib and tcp seem to be similar? So do you have some experience that for how many cores, infiniband could emerge its advantage?
Thanks again for your patient reply~
Best regards!

[GDTech]

I get the result:
MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.3)

yes it means you have infiniband support.

Then, the two command:
mpirun --mca btl openib,self -n 4 icoFoam -parallel
mpirun --mca btl tcp,self -n 4 icoFoam -parallel
can both run the case, but the efficiency seems to be the same

It is possible to configure infiniband network in order to make tcp/ip communication passing trough infiniband (called IP over IB). In that case, you won't see any significant difference between the two previous commands. You can check if IP over IB is enabled on your cluster with this command :

lsmod | grep ipoib

If that command gives you an output, IP over IB is enabled.

[lilifeng]

Originally Posted by GDTech

yes it means you have infiniband support.


It is possible to configure infiniband network in order to make tcp/ip communication passing trough infiniband (called IP over IB). In that case, you won't see any significant difference between the two previous commands. You can check if IP over IB is enabled on your cluster with this command :

lsmod | grep ipoib

If that command gives you an output, IP over IB is enabled.

Now I use Infiniband to connect two computers, And I use the following command:mpirun --mca btl openib,self -hostfile hostfile -np 64 pisofoam -parallel,I use 32 cpus on each node,However, I found that the computation speed of the IB network card connection is, similar to that of the ordinary network cable connection.