Infiniband switch OS

Allen_ · June 10, 2020, 21:27

I’ve built a small homelab cluster that for now consists of two nodes with just a direct connection between the infiniband cards, but I’m about ready to expand so I’m shopping around for a switch. One thing that is causing some doubt is rebranded mellanox switches. Specifically I’m looking at the mellanox sx6015, and there are a few that are affordable for a homelab cluster. One that’s available is rebranded as an HPE switch, and another is loaded with an “EMC” operating system. Can I expect either of these switches to work properly with standard single port FDR cards that have the regular mellanox firmware on them, or would I need to flash my cards with matching firmware? I know with an Ethernet switch the switching functions are standardized and I wouldn’t have to worry about it, but I’m new to infiniband and not really sure.

Thanks,

Allen

ssyadav · July 30, 2020, 01:32

Dear Allen

I was wondering whether you were successful in connecting the nodes with infiniband or gigabit network?

I have three conventional workstations (dual processor xeons) and I want to join them via Infiniband or Gigabit network. I request you to please help me in this regard.

Particularly what is switch you are using? What cards are you using for connecting with the motherboards etc.

I want to run simulations with OpenFOAM and Ansys Fluent & CFX.

Thanks in advance.

Allen_ · October 15, 2020, 21:41

Ok, sorry for such a late response. Hopefully you found an acceptable solution

I am using QDR Infiniband. The cards are Mellanox ConnectX-3. They come in a few varieties and with one or two ports. It doesn’t make any difference how many ports you get. Unless you’re setting up a more complicated network topology that uses subgroups of nodes all interconnected age different levels with multiple switches, you’ll only be using one port anyway. I have both two-port and one-port cards because I expanded my cluster slowly and purchased them at different times, and my only selection criteria at the time of purchase was price and shipping time (eBay). Looking back I think I’d prefer to have just had all the cards be single port just for simplicity, but functionally it makes no difference at all. All that said, the model numbers of the cards are “MCX-353A” and “MCX-354A”. The “353” is the single port model. The second thing to pay attention to is the four letter suffix. The “QCBT” is the QDR model, and “FCBT” is the FDR model. So the whole card model number reads like “MCX353A-QCBT” Like I said, I went with QDR since the cheapest used FDR switches I could find were still over $500. I got all of my cards from eBay, and the prices ranged from $30 to $55, including shipping.

As for the switch, I went with the Mellanox IS5023. Again from eBay it was $125 including shipping. As for the managed vs unmanaged, I still don’t really know if I’d prefer managed or not. I want to eventually set up a little more automation in my cluster so I thought I’d buy a managed switch just in case. However, I accidentally bought an unmanaged switch. As it happens, in Mellanox parlance, the phrase “externally managed” just means “unmanaged”. Anyway, it works fine. I just have to remember to run the OpenSM manager whenever I start the head node, so it’s not a huge deal. I’m still not sure of what limitations it might have though, when I try to build in some more advanced features to the cluster.

Hope that helped, though I think it’s likely you’ll probably have already moved forward with your build. But anyway, this is what I did with mine.

June 10, 2020, 21:27	Infiniband switch OS	#1
Allen_ New Member Allen Join Date: Jan 2020 Posts: 8 Rep Power: 6	I’ve built a small homelab cluster that for now consists of two nodes with just a direct connection between the infiniband cards, but I’m about ready to expand so I’m shopping around for a switch. One thing that is causing some doubt is rebranded mellanox switches. Specifically I’m looking at the mellanox sx6015, and there are a few that are affordable for a homelab cluster. One that’s available is rebranded as an HPE switch, and another is loaded with an “EMC” operating system. Can I expect either of these switches to work properly with standard single port FDR cards that have the regular mellanox firmware on them, or would I need to flash my cards with matching firmware? I know with an Ethernet switch the switching functions are standardized and I wouldn’t have to worry about it, but I’m new to infiniband and not really sure. Thanks, Allen

July 30, 2020, 01:32	Please post the details of your cluster	#2
ssyadav New Member Shyam Sunder Join Date: Sep 2015 Posts: 27 Rep Power: 10	Dear Allen I was wondering whether you were successful in connecting the nodes with infiniband or gigabit network? I have three conventional workstations (dual processor xeons) and I want to join them via Infiniband or Gigabit network. I request you to please help me in this regard. Particularly what is switch you are using? What cards are you using for connecting with the motherboards etc. I want to run simulations with OpenFOAM and Ansys Fluent & CFX. Thanks in advance.

October 15, 2020, 21:41		#3
Allen_ New Member Allen Join Date: Jan 2020 Posts: 8 Rep Power: 6	Ok, sorry for such a late response. Hopefully you found an acceptable solution I am using QDR Infiniband. The cards are Mellanox ConnectX-3. They come in a few varieties and with one or two ports. It doesn’t make any difference how many ports you get. Unless you’re setting up a more complicated network topology that uses subgroups of nodes all interconnected age different levels with multiple switches, you’ll only be using one port anyway. I have both two-port and one-port cards because I expanded my cluster slowly and purchased them at different times, and my only selection criteria at the time of purchase was price and shipping time (eBay). Looking back I think I’d prefer to have just had all the cards be single port just for simplicity, but functionally it makes no difference at all. All that said, the model numbers of the cards are “MCX-353A” and “MCX-354A”. The “353” is the single port model. The second thing to pay attention to is the four letter suffix. The “QCBT” is the QDR model, and “FCBT” is the FDR model. So the whole card model number reads like “MCX353A-QCBT” Like I said, I went with QDR since the cheapest used FDR switches I could find were still over $500. I got all of my cards from eBay, and the prices ranged from $30 to $55, including shipping. As for the switch, I went with the Mellanox IS5023. Again from eBay it was $125 including shipping. As for the managed vs unmanaged, I still don’t really know if I’d prefer managed or not. I want to eventually set up a little more automation in my cluster so I thought I’d buy a managed switch just in case. However, I accidentally bought an unmanaged switch. As it happens, in Mellanox parlance, the phrase “externally managed” just means “unmanaged”. Anyway, it works fine. I just have to remember to run the OpenSM manager whenever I start the head node, so it’s not a huge deal. I’m still not sure of what limitations it might have though, when I try to build in some more advanced features to the cluster. Hope that helped, though I think it’s likely you’ll probably have already moved forward with your build. But anyway, this is what I did with mine.