[DungPham]

Hi everyone,

I am now trying to find a solution to run parallel Fluent on four workstations by using a configuration with IB switch. I know that a member (Ghost84?) has already posted a tutorial for interconnecting 2 workstations directly without a switch. But now it is 4 workstations, please share your solutions. So far what I know is we need IB cards (DDR CX4-SFF8470 20Gb/s is good or not?), IB switch (DDR? or any suggestion for a suitable switch), and 20Gb/s CX4 to CX4 cables.
Thank you for your reading and sharing.

My Dell T5610 specs are: Dual E5-2697 v2 12core 2.7ghz, quad channel 64GB DDR3-10600R.

[wkernkamp]

It is possible to run without switch. I did it with two ConnectX-3 pcie3 cards from Mellanox +/- $40 on ebay. The copper IB cables cost also about $40. Your speed will be 56 Gb/s when you have QDR FDR cards. Hardware cost total: $120.

Be careful that you:
1. get the IB cards and not the EN cards. The latter does not support IB, but the former does support Ethernet over IB.
2. The HP cards have a non-standard shape that does not match your "normal" pcie slot.

The mellanox cards typically need a firmware update. This happens automatically with ofed install, but not when it is a mellanox card from ibm or oracle. However, you can turn an Oracle or IBM card back to a regular configuration with the following command:

sudo flint --no_flash_verify -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin --allow_psid_change burn

replace mt4099_pci_cr0 with your applicable device; and download firmware "fw-...bin" as appropriate.

Caution: firmware burning on your own can brick your card. Did not happen to me, but I don't want to be responsible for your card.

You need opensm running on one of the machines. (That is the IB network manager)

I ran this on ubuntu 18.10. For my application the speed-up was better than 2x, because the problem memory access is reduced on the individual machine (bottleneck), while the IB with direct memory access (rdma) is very fast.

There are IB switches available on ebay for $300. Got one, because I am linking more than two machines. It works well but is very noisy.

Note that you could also build a chain or ring with the individual cards if they are dual port. Each segment requires it own setup. I did not go that route, because it is difficult to add/subtract machines. I am also not sure whether the fabric remains as fast if you chain nodes in this way. If you try this, let me know!

Good luck.

[zhangqin200000]

Originally Posted by wkernkamp

It is possible to run without switch. I did it with two ConnectX-3 pcie3 cards from Mellanox +/- $40 on ebay. The copper IB cables cost also about $40. Your speed will be 56 Gb/s when you have QDR FDR cards. Hardware cost total: $120.

Be careful that you:
1. get the IB cards and not the EN cards. The latter does not support IB, but the former does support Ethernet over IB.
2. The HP cards have a non-standard shape that does not match your "normal" pcie slot.

The mellanox cards typically need a firmware update. This happens automatically with ofed install, but not when it is a mellanox card from ibm or oracle. However, you can turn an Oracle or IBM card back to a regular configuration with the following command:

sudo flint --no_flash_verify -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin --allow_psid_change burn

replace mt4099_pci_cr0 with your applicable device; and download firmware "fw-...bin" as appropriate.

Caution: firmware burning on your own can brick your card. Did not happen to me, but I don't want to be responsible for your card.

You need opensm running on one of the machines. (That is the IB network manager)

I ran this on ubuntu 18.10. For my application the speed-up was better than 2x, because the problem memory access is reduced on the individual machine (bottleneck), while the IB with direct memory access (rdma) is very fast.

There are IB switches available on ebay for $300. Got one, because I am linking more than two machines. It works well but is very noisy.

Note that you could also build a chain or ring with the individual cards if they are dual port. Each segment requires it own setup. I did not go that route, because it is difficult to add/subtract machines. I am also not sure whether the fabric remains as fast if you chain nodes in this way. If you try this, let me know!

Good luck.

Hi,
is there possible for you to share the models of your IB card and IB switch? Therefore we can have a reference to setup our own. Is there necessarily to go for 56G? What are you currently using? 10G,20G,40G or 56G?

Many thanks

Qin