|
[Sponsors] |
May 21, 2014, 09:22 |
Infiniband How-to for OpenSuse?
|
#1 |
Senior Member
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 172
Rep Power: 13 |
Looking to install infiniband on a 7 node, 20 socket, 80 core cluster, all running OpenSuse 12.3 with Openfoam 2.3. The IB is posited to be 4X SDR using old Voltaire kit badged HP.
e.g. 376166-B22 Voltaire InfiniBand 4X SDR 24-Port Internally Managed Switch 380299-B21 10GB INFINIBAND PCI-X 4X 2 PORT HCA See any problems with this approach? Do have any info on how to do install OFED and drivers on OpenSuse? Any tips on configuring the switch?
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET |
|
May 21, 2014, 17:22 |
|
#2 |
Senior Member
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 172
Rep Power: 13 |
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET |
|
May 27, 2014, 11:35 |
|
#3 |
Senior Member
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 172
Rep Power: 13 |
In the middle of trying to get infiniband going can any one telp me which RPM the OpenIBD script is in?
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET |
|
May 28, 2014, 16:54 |
|
#4 | |
Senior Member
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 172
Rep Power: 13 |
Quote:
what a thrutch! OFED 3.52 officially supported up to kernel 3.5 but OpenSUSE 12.3 is 3.7. However OpS12.3 has the infiniband kernel modules but nothing else. The journey so far means using the OpS12.3 default kernel then modifying the install perl script to let it proceed with 3.7 kernel e.g Code:
if ($package =~ /compat-rdma|ib-bonding/) { if (not $packages_info{$package}{'rpm_exist'}) { # Check that required kernel is supported #if ($kernel !~ /2.6.16.60-[A-Za-z0-9.]*-[A-Za-z0-9.]*|2.6.1[8-9]|2.6.2[0-9]|2.6.3[0-9]|2.6.40|3.[0-5]/) { if ($kernel !~ /2.6.16.60-[A-Za-z0-9.]*-[A-Za-z0-9.]*|2.6.1[8-9]|2.6.2[0-9]|2.6.3[0-9]|2.6.40|3.[0-7]/) { print RED "Kernel $kernel is not supported.", RESET "\n"; print BLUE "For the list of Supported Platforms and Operating Systems see", RESET "\n"; print BLUE "$CWD/docs/OFED_release_notes.txt", RESET "\n"; exit 1; Code:
} print BLUE "Detected Linux Distribution: $DISTRO", RESET "\n" if ($verbose3); # Uninstall the previous installations # DMM uninstall(); my $vendor_ret; if (length($vendor_pre_install) > 0) { Recover the build fragments from /var/tmp to another directory. Create a script to remove any modules that get installed. (rpm -e xxx) Now run the install script in custom build mode, but dont build compat-rdma dapl ibacm Now retrieve openib.conf from the build fail of compat-rdma and put it in /etc/infiniband and from the same place put openibd where opS12.3 will use it at boot time to load the modules. edit openib.conf so as to select the right modules. Code:
openibd start simples ... NOT
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET |
||
July 9, 2014, 09:38 |
|
#5 |
Senior Member
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 172
Rep Power: 13 |
Its working all the way to Openfoam 2.3
Getting Qlogic cards to work turned out to be too complex (intellipath PSM ) compared to buying few more mellanox based cards. A tangled web of OFED, OPENMPI and updating firmware was navigated. The combination OFED 3.5, Openmpi 1.6.5 was used. Opensuse 11.4 was used to update the mellanox cards along with obscure versions mellanox software sourced from an old mellanox site. Various issues with documentation were encountered such as the "pinning of memory" in Bash login scripts. If anyone is interested I will go into the details
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET |
|
December 31, 2015, 19:24 |
|
#6 |
Senior Member
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 172
Rep Power: 13 |
Infiniband on Leap 42.1 is so easy, no compilation needed as its in the kernel and the distro and can be installed with Yast. The openmpi in the distro has infiniband included. A doddle compared to 12.3
One tip: use the same infiniband card on all nodes.
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET |
|
April 1, 2017, 05:53 |
|
#7 |
New Member
Dmitry Glyzin
Join Date: Apr 2017
Location: Yaroslavl, Russia
Posts: 2
Rep Power: 0 |
Hi derekm,
have you made infiniband work with Leap 42.1 out of the box? I got some troubles with 42.2: http://serverfault.com/questions/841...connectx-3-hca |
|
April 1, 2017, 10:51 |
|
#8 |
Senior Member
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 172
Rep Power: 13 |
i did this for my cards VOLTAIRE 410-4EX (mthca)
Code:
zypper install opensm ibutils ibutils-devel infiniband-diags infiniband-diags-devel libibcm1 libibverbs-devel libibverbs-runtime ibacm libibcm1 libmthca-rdmav2 rdma tvflash libibnetdisc5 ibsim qperf #enable OPENIBD service on one node systemctl enable openibd systemctl start openibd
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET |
|
April 1, 2017, 16:41 |
|
#9 |
New Member
Dmitry Glyzin
Join Date: Apr 2017
Location: Yaroslavl, Russia
Posts: 2
Rep Power: 0 |
Solved with installing libmlx4-rdmav2 form Factory repo and downgrading every other infiniband package to Factory version.
|
|
April 2, 2017, 10:25 |
|
#10 |
Senior Member
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 172
Rep Power: 13 |
After the extreme complexity of installing infiniband on OpenSuse 12.3, the simplicity of infiniband with leap 42 takes a bit of getting used to. I made a similar mistake and had to redo the installation remembering to do the absolute minimum.
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET |
|
|
|