Infiniband How-to for OpenSuse?
Looking to install infiniband on a 7 node, 20 socket, 80 core cluster, all running OpenSuse 12.3 with Openfoam 2.3. The IB is posited to be 4X SDR using old Voltaire kit badged HP.
e.g. 376166-B22 Voltaire InfiniBand 4X SDR 24-Port Internally Managed Switch 380299-B21 10GB INFINIBAND PCI-X 4X 2 PORT HCA See any problems with this approach? Do have any info on how to do install OFED and drivers on OpenSuse? Any tips on configuring the switch? |
|
In the middle of trying to get infiniband going can any one telp me which RPM the OpenIBD script is in?
|
Quote:
what a thrutch! OFED 3.52 officially supported up to kernel 3.5 but OpenSUSE 12.3 is 3.7. However OpS12.3 has the infiniband kernel modules but nothing else. The journey so far means using the OpS12.3 default kernel then modifying the install perl script to let it proceed with 3.7 kernel e.g Code:
if ($package =~ /compat-rdma|ib-bonding/) { Code:
} Recover the build fragments from /var/tmp to another directory. Create a script to remove any modules that get installed. (rpm -e xxx) Now run the install script in custom build mode, but dont build compat-rdma dapl ibacm Now retrieve openib.conf from the build fail of compat-rdma and put it in /etc/infiniband and from the same place put openibd where opS12.3 will use it at boot time to load the modules. edit openib.conf so as to select the right modules. Code:
openibd start simples ... NOT |
Its working all the way to Openfoam 2.3
Getting Qlogic cards to work turned out to be too complex (intellipath PSM ) compared to buying few more mellanox based cards. A tangled web of OFED, OPENMPI and updating firmware was navigated. The combination OFED 3.5, Openmpi 1.6.5 was used. Opensuse 11.4 was used to update the mellanox cards along with obscure versions mellanox software sourced from an old mellanox site. Various issues with documentation were encountered such as the "pinning of memory" in Bash login scripts. If anyone is interested I will go into the details |
Infiniband on Leap 42.1 is so easy, no compilation needed as its in the kernel and the distro and can be installed with Yast. The openmpi in the distro has infiniband included. A doddle compared to 12.3
One tip: use the same infiniband card on all nodes. |
Hi derekm,
have you made infiniband work with Leap 42.1 out of the box? I got some troubles with 42.2: http://serverfault.com/questions/841...connectx-3-hca |
i did this for my cards VOLTAIRE 410-4EX (mthca)
Code:
|
Solved with installing libmlx4-rdmav2 form Factory repo and downgrading every other infiniband package to Factory version.
|
After the extreme complexity of installing infiniband on OpenSuse 12.3, the simplicity of infiniband with leap 42 takes a bit of getting used to. I made a similar mistake and had to redo the installation remembering to do the absolute minimum.
|
All times are GMT -4. The time now is 15:46. |