CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Hardware

Infiniband How-to for OpenSuse?

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By derekm

Reply
 
LinkBack Thread Tools Display Modes
Old   May 21, 2014, 09:22
Default Infiniband How-to for OpenSuse?
  #1
Member
 
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 95
Rep Power: 4
derekm is on a distinguished road
Looking to install infiniband on a 7 node, 20 socket, 80 core cluster, all running OpenSuse 12.3 with Openfoam 2.3. The IB is posited to be 4X SDR using old Voltaire kit badged HP.

e.g.
376166-B22 Voltaire InfiniBand 4X SDR 24-Port Internally Managed Switch
380299-B21 10GB INFINIBAND PCI-X 4X 2 PORT HCA

See any problems with this approach?
Do have any info on how to do install OFED and drivers on OpenSuse?
Any tips on configuring the switch?
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET
derekm is offline   Reply With Quote

Old   May 21, 2014, 17:22
Default
  #2
Member
 
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 95
Rep Power: 4
derekm is on a distinguished road
found this https://www.openfabrics.org/images/d...n_Linux_R1.pdf
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET
derekm is offline   Reply With Quote

Old   May 27, 2014, 11:35
Default
  #3
Member
 
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 95
Rep Power: 4
derekm is on a distinguished road
In the middle of trying to get infiniband going can any one telp me which RPM the OpenIBD script is in?
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET
derekm is offline   Reply With Quote

Old   May 28, 2014, 16:54
Default
  #4
Member
 
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 95
Rep Power: 4
derekm is on a distinguished road
Quote:
Originally Posted by derekm View Post
In the middle of trying to get infiniband going can any one telp me which RPM the OpenIBD script is in?
got it going today at least to ip over infiniband...

what a thrutch!

OFED 3.52 officially supported up to kernel 3.5 but OpenSUSE 12.3 is 3.7. However OpS12.3 has the infiniband kernel modules but nothing else.
The journey so far means using the OpS12.3 default kernel
then modifying the install perl script to let it proceed with 3.7 kernel e.g
Code:
       if ($package =~ /compat-rdma|ib-bonding/) {
            if (not $packages_info{$package}{'rpm_exist'}) {
                # Check that required kernel is supported
                #if ($kernel !~ /2.6.16.60-[A-Za-z0-9.]*-[A-Za-z0-9.]*|2.6.1[8-9]|2.6.2[0-9]|2.6.3[0-9]|2.6.40|3.[0-5]/) {
 
                if ($kernel !~ /2.6.16.60-[A-Za-z0-9.]*-[A-Za-z0-9.]*|2.6.1[8-9]|2.6.2[0-9]|2.6.3[0-9]|2.6.40|3.[0-7]/) {

                print RED "Kernel $kernel is not supported.", RESET "\n";
                    print BLUE "For the list of Supported Platforms and Operating Systems see", RESET "\n";
                    print BLUE "$CWD/docs/OFED_release_notes.txt", RESET "\n";
                    exit 1;
and prevent it doing a uninstall cos the uninstall is buggy.
Code:
   }
    print BLUE "Detected Linux Distribution: $DISTRO", RESET "\n" if ($verbose3);
    
    # Uninstall the previous installations
   # DMM uninstall();
    my $vendor_ret;
    if (length($vendor_pre_install) > 0) {
Then run it once to get the compat-Rdma module attempt to build but fail.
Recover the build fragments from /var/tmp to another directory.
Create a script to remove any modules that get installed. (rpm -e xxx)
Now run the install script in custom build mode, but dont build compat-rdma dapl ibacm

Now retrieve openib.conf from the build fail of compat-rdma and put it in /etc/infiniband
and from the same place put openibd where opS12.3 will use it at boot time to load the modules. edit openib.conf so as to select the right modules.

Code:
openibd start
then follow the standard OFED instructions

simples ... NOT
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET
derekm is offline   Reply With Quote

Old   July 9, 2014, 09:38
Default
  #5
Member
 
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 95
Rep Power: 4
derekm is on a distinguished road
Its working all the way to Openfoam 2.3
Getting Qlogic cards to work turned out to be too complex (intellipath PSM ) compared to buying few more mellanox based cards.
A tangled web of OFED, OPENMPI and updating firmware was navigated.
The combination OFED 3.5, Openmpi 1.6.5 was used.
Opensuse 11.4 was used to update the mellanox cards along with obscure versions mellanox software sourced from an old mellanox site.
Various issues with documentation were encountered such as the "pinning of memory" in Bash login scripts.
If anyone is interested I will go into the details
ghost82 likes this.
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET
derekm is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 17:58.