[hsnyder]

Purely out of curiosity, does anybody here use more than one node, on hardware they own themselves? I'd be curious for thoughts/experiences regarding whether getting (possibly used) infiniband cards is worth it over regular ethernet...


What got me thinking about this is, while there's a lot of discussion here on Epyc builds, old quad-channel Xeons are sometimes very very cheap by comparison... If hardware cost and not electricity cost were a prime factor, I'm curious if a small cluster of old haswell or broadwell hardware could make sense.

[gnwt4a]

The answer has to be infiniband because the number
of nodes must be greater than 4.

i've given up (at least 4 now) on gpu compute
(4 dns) as the number of new cryptocurrencies
increases faster than the compute-gpu supply.

u can get a 2697v2 for about 100 euros (including
vat) from aliexpress so building a two-node 2p system
will be about the price of a used radeon vii.
four node system ex-infiniband looks totally
feasible. used infiniband cards (50-100 gb/s) and
router will be of the order of several hundred
euros.
==


ps; 10gb/s cards are not in the running.

[hsnyder]

Thanks for your input!

Originally Posted by gnwt4a

The answer has to be infiniband because the number
of nodes must be greater than 4.

What do you mean by this? Must be greater than 4 to have worthwhile performance?