how to confirm that I have already use infiniband in OpenFOAM?
 

Dear all,
Recently, I am wroking on a case with LES for about 4M mesh, I try to install the infiniband on Ubuntu 12.04 in our cluster, but when I run the case in 48 cores with infiniband connections, I found that it is seems to the be same as I run the case with Ethernet, so is that means I do not use the infiniband at all, or I do not really use ib protocol?
In addition, I installed OpenFOAM use the Ubuntu Deb Pack Installation, the openmpi was also installed automatically, so is that means I should recompile the openmpi again to support infiniband, if so, how can I should do?
Or weather there is some tools to monitor the flux of infiniband, so that I can ensure that I have already use infiniband, because the monitor of Ubuntu could only monitor TCP/IP.
Thanks for your help~

Hi,

Default options for compiling openmpi in thirdparty do not include infiniband support. (see options in ThirdParty-x.x.x/Allwmake)
Check if your openmpi installation has infiniband support with the following command :
If the previous command shows nothing, You have to recompile openmpi. Here is what I added to Allwmake before "./configure " line to get infiniband support :
It is possible to force mpirun to use a specific protocol for communication.
To force openib protocol :
mpirun --mca btl openib,self -n 4 icoFoam -parallel

To force ethernet protocol
mpirun --mca btl tcp,self -n 4 icoFoam -parallel

Best Regards.

Dear Laurent,
Thanks for your reply!
As to the problem,when I check the openmpi with: # ompi_info | grep openib
I get the result:
MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.3)
Is that means there is no problem with the openmpi?
Then, the two command:
mpirun --mca btl openib,self -n 4 icoFoam -parallel
mpirun --mca btl tcp,self -n 4 icoFoam -parallel
can both run the case, but the efficiency seems to be the same, is that means for a 4M case in 48 cores, both openib and tcp seem to be similar? So do you have some experience that for how many cores, infiniband could emerge its advantage?
Thanks again for your patient reply~
Best regards!

yes it means you have infiniband support.

It is possible to configure infiniband network in order to make tcp/ip communication passing trough infiniband (called IP over IB). In that case, you won't see any significant difference between the two previous commands. You can check if IP over IB is enabled on your cluster with this command :
If that command gives you an output, IP over IB is enabled.

Hello,I would like to ask if you have solved the problem of infiniband speed increase
 

Now I use Infiniband to connect two computers, And I use the following command:mpirun --mca btl openib,self -hostfile hostfile -np 64 pisofoam -parallel,I use 32 cpus on each node,However, I found that the computation speed of the IB network card connection is, similar to that of the ordinary network cable connection.:confused: