[derekm]

Looking to install infiniband on a 7 node, 20 socket, 80 core cluster, all running OpenSuse 12.3 with Openfoam 2.3. The IB is posited to be 4X SDR using old Voltaire kit badged HP.

e.g.
376166-B22 Voltaire InfiniBand 4X SDR 24-Port Internally Managed Switch
380299-B21 10GB INFINIBAND PCI-X 4X 2 PORT HCA

See any problems with this approach?
Do have any info on how to do install OFED and drivers on OpenSuse?
Any tips on configuring the switch?

[derekm]

found this https://www.openfabrics.org/images/d...n_Linux_R1.pdf

[derekm]

In the middle of trying to get infiniband going can any one telp me which RPM the OpenIBD script is in?

[derekm]

Originally Posted by derekm

In the middle of trying to get infiniband going can any one telp me which RPM the OpenIBD script is in?

got it going today at least to ip over infiniband...

what a thrutch!

OFED 3.52 officially supported up to kernel 3.5 but OpenSUSE 12.3 is 3.7. However OpS12.3 has the infiniband kernel modules but nothing else.
The journey so far means using the OpS12.3 default kernel
then modifying the install perl script to let it proceed with 3.7 kernel e.g

       if ($package =~ /compat-rdma|ib-bonding/) {
            if (not $packages_info{$package}{'rpm_exist'}) {
                # Check that required kernel is supported
                #if ($kernel !~ /2.6.16.60-[A-Za-z0-9.]*-[A-Za-z0-9.]*|2.6.1[8-9]|2.6.2[0-9]|2.6.3[0-9]|2.6.40|3.[0-5]/) {
 
                if ($kernel !~ /2.6.16.60-[A-Za-z0-9.]*-[A-Za-z0-9.]*|2.6.1[8-9]|2.6.2[0-9]|2.6.3[0-9]|2.6.40|3.[0-7]/) {

                print RED "Kernel $kernel is not supported.", RESET "\n";
                    print BLUE "For the list of Supported Platforms and Operating Systems see", RESET "\n";
                    print BLUE "$CWD/docs/OFED_release_notes.txt", RESET "\n";
                    exit 1;

and prevent it doing a uninstall cos the uninstall is buggy.

   }
    print BLUE "Detected Linux Distribution: $DISTRO", RESET "\n" if ($verbose3);
    
    # Uninstall the previous installations
   # DMM uninstall();
    my $vendor_ret;
    if (length($vendor_pre_install) > 0) {

Then run it once to get the compat-Rdma module attempt to build but fail.
Recover the build fragments from /var/tmp to another directory.
Create a script to remove any modules that get installed. (rpm -e xxx)
Now run the install script in custom build mode, but dont build compat-rdma dapl ibacm

Now retrieve openib.conf from the build fail of compat-rdma and put it in /etc/infiniband
and from the same place put openibd where opS12.3 will use it at boot time to load the modules. edit openib.conf so as to select the right modules.

openibd start

then follow the standard OFED instructions

simples ... NOT

[derekm]

Its working all the way to Openfoam 2.3
Getting Qlogic cards to work turned out to be too complex (intellipath PSM ) compared to buying few more mellanox based cards.
A tangled web of OFED, OPENMPI and updating firmware was navigated.
The combination OFED 3.5, Openmpi 1.6.5 was used.
Opensuse 11.4 was used to update the mellanox cards along with obscure versions mellanox software sourced from an old mellanox site.
Various issues with documentation were encountered such as the "pinning of memory" in Bash login scripts.
If anyone is interested I will go into the details

[derekm]

Infiniband on Leap 42.1 is so easy, no compilation needed as its in the kernel and the distro and can be installed with Yast. The openmpi in the distro has infiniband included. A doddle compared to 12.3
One tip: use the same infiniband card on all nodes.

[dglyzin]

Hi derekm,
have you made infiniband work with Leap 42.1 out of the box?

I got some troubles with 42.2:

http://serverfault.com/questions/841...connectx-3-hca

[derekm]

i did this for my cards VOLTAIRE 410-4EX (mthca)

zypper install opensm ibutils ibutils-devel infiniband-diags infiniband-diags-devel libibcm1 libibverbs-devel libibverbs-runtime ibacm libibcm1 libmthca-rdmav2 rdma tvflash libibnetdisc5 ibsim qperf
#enable OPENIBD service on one node 
systemctl enable openibd
systemctl start  openibd

[dglyzin]

Solved with installing libmlx4-rdmav2 form Factory repo and downgrading every other infiniband package to Factory version.

[derekm]

After the extreme complexity of installing infiniband on OpenSuse 12.3, the simplicity of infiniband with leap 42 takes a bit of getting used to. I made a similar mistake and had to redo the installation remembering to do the absolute minimum.