Choose motherboard for cluster building for CFD application
 

Hi,

I'm building my cluster for running Fluent. I have already fixed the CPU that I'm gonna to choose. And I have found several motherboards which are compatible with that CPU. But I have no idea about how to compare these motherboards to select the one which can achieve better CFD speed.

Which motherboard factors shall I look at? I heard about motherboard is important for the data exchange speed between the components installed on it. Which speed is important for CFD by Fluent?

By the way, the cluster will be composed of 8 motherboards. The CPU I chose is Intel Xeon E5-1620. I'm not bothering you by asking which motherboard is good. I'm just asking the way to choose more suitable motherboard.

I don't think that there is a great varieties of chipset that support Xeon E5 CPU's. So it is unlikely that you will find a substantial difference in performance, especially because the memory controller is part of the CPU anyway. So you need to choose your motherboard on features, not performance. The features that you would need to evaluate are things like form factor, number of DIMM slots, number of PCI-X slots, number of network ports and weigh up the cost-implication of a built-in Infiniband NIC vs. plug in cards.

Hi CapSizer,

Could you comment on my CPU selection and motherboard selection for CFD application? This will be my first time to purchase and assemble a cluster. I read the discussions on this forum quite carefully but I’m still not confident about my choice. Hope someone could let me know if I select something wrong.

I’m going to setup a 32 cores cluster for only CFD application. The solver will be Fluent.

I select CPU Intel Xeon E5-1620 v2 which has four 3.7 GHz cores and max memory bandwidth 59.7 GB/s. Each of it only costs 370 US dollars in U.S. and less than 300 US dollars in China. I think its price in Europe won't be too different. I will connect 8 or 16 E5-1620 v2 CPUs to have 32 cores. This is the best maximum memory bandwidth (GB/s) per dollar I found.

I select motherboard ASUS P9X79 to match this CPU. This motherboard is 400 US dollars for each and is mainly for desktop application. It only has one socket for CPU. I will have to buy 8 these expensive motherboards in order to have 32 cores in total. But I didn’t find a cheaper choice for this. The CPU Intel Xeon E5-1620 gives too good bandwidth (GB/s) per dollar and 3.7 GHz. So I have to use large number of this expensive motherboard to support the selected CPU.

I will choose 1866 GHz memory sticks as the motherboard supports max 2133 GHz and CPU supports only 1866 GHz. The memory brand and version is constraint by the motherboard’s memory suggestion list. This is my headache. The list is too short and I can’t find the suggested memory in the market unless I import them from U.S. Could I select out of motherboard suggestion list memory but of a big brand like Kinston? It seems currently the compatibility of memory is always very good. Btw, someone told me that 1600 GHz memory is much more stable than 1833 GHz memory. Is this true? I’m confusing about that. That means I have to degrade the performance of my cluster by choosing a weaker memory for stability reason. Any thought or suggestion on that?

Meimei, you are asking some good questions! First of all though, if you are using a single socket motherboard, it is not sensible to use the Xeon E5 CPU, which is intended for dual socket boards. You will do better I think with Core i7 CPU's, either the 4-core 4820 K, which looks like good value for money, or perhaps the 6-core 4930 K. The LGA 2011 i7's have access to very good memory bandwidth from the 4 memory channels, so I expect that you will see a worthwhile performance improvement from the extra 2 cores, although it sounds as if you are constrained by software licensing to 32 cores overall? But with 6-core CPU's, you would only need 6 nodes instead of 8, which will mean that you save on motherboards, chassis and network cards, to make up for the more expensive CPU's.

As for memory, in your position I would look for Kingston 2400 MHz memory. If it is unstable you can always reduce the memory speed a bit, but if it works you will have a worthwhile amount of extra performance.

However, to get this cluster performing anywhere near its potential, using Infiniband is essential, despite the expense. Gb ethernet is not good enough. If you cannot afford new cards, cables and switch (small IB switches are rare), take a look at e-bay. You can get older 20 Gbps DDR IB hardware for relatively little, but you will need to invest some effort to make it all work.

I chose Intel Xeon rather than i7 for two reasons.

1) Fluent suggests Xeon. I think Fluent is somehow designed for Xeon series.
2) i7 is a desktop CPU serie. And Xeon is for server (and cluster?). It looks like i7 is not designed for connected nodes.

Besides changing the CPU, another way shall be changing the motherboard. Some 'per CPU cheap' motherboard declares that it may not support some features of the CPU which makes the motherboard selection a difficult task. Shall I theoretically study how to select motherboard for a certain CPU rather than just look at the recommendation from the motherboard website which tells not enough information about how much the motherboard compatible with the CPU?

Your suggestion on memory selection is very interesting to me. If the CPU only supports at most 1866 GHz and motherboard supports at most 2133 GHz. Won't be a waste to use 2400 GHz memory? Do you mean overclocking? How to achieve that?

By the way, I saw you have a lot of very helpful discussions in this forum. But I didn't find a clear answer this question (someone has asked it). If I have 8 places for memory cards, shall I go for 4*8 GB or 8*4 GB?

OK, lotta questions in one post:

1. No, there is very little difference between Xeon and i7. Basically the same processor, but one intended for multi-socket server, and the other for desktop workstation. The most important difference is the QPI architecture, which allows the Xeon to communicate at very high speed with another CPU on the same motherboard.
2. Correct, but i7 will be fine for a cluster, you just cannot use it in a multi-socket motherboard.
3. Look for the motherboard that is compatible with the CPU, in this case LGA 2011, and supports the required memory speed.
4. You can "tune" the overclocking settings in the BIOS. Get an enthusiastic kid to help you. i7 boards make provision for overclocking, Xeon boards don't. You're not trying to squeeze the last bit of speed out of the CPU, you just want to activate the best memory speed that your system will support.
5. For the single socket board 4X8 and 8X4 will be more or less the same, but I think you will find that the very fast memory modules will only come as 4 GB units. On the single socket LGA 2011 i7 system, you have 4 parallel memory channels. 8 DIMMS will not give you more speed.

1. I look at the i7 list but didn't find any i7 CPU with comparable performance/price to Xeon E5-1620 v2. The Xeon E5-1620 v2 only costs less than 360 U.S. dollars. The only comparable i7-4960X with 6 cores, same bandwidth, 3.6 GHz (E5-1620 v2 is 3.7 GHz) costs more than 1200 U.S. dollars. It is much weaker than Xeon E5-1620 v2 for CFD application. Having two more cores with constant memory bandwidth is not very meaningful for CFD, right?

2. I look at the dual CPU motherboard and found that normally they limit the memory speed to 1600 GHz or even lower. The ASUS Z9PE-D8 dual motherboard is attractable. But when I realize that it only has 4 memory channel in total (2 memory channel per CPU), I don't think it is nice for CFD application. I guess the dual CPU motherboard is designed for servers, not for CFD.

For these reasons, probably it's better to stay with the single CPU mother board ASUS P979 with Xeon E5-1620 v2.

PS: When I read the discussions in the forum, I see that people here usually agree on the fact that the memory bandwidth per dollar is an important factor to CFD hardware performance. But the number of cores per dollar is also an important factor. And also the data exchange rate between the motherboards is important. We have to also minimize the number of motherboards to improve the CFD hardware performance by reducing the data transfer time between the motherboards. It seems that we can’t use single criteria to decide the hardware selection (correct me if that's not correct). How to balance these 4 factors?

For my case, we share licenses with another 3 companies in the same building. The licenses number is not a concern to us. The main concern is hardware price.

3. Thanks for your answer about memory. I would follow your advice and go for Kinston 4g 2400 bar, even though they are not in the motherboard suggestion list. They are almost the same price as 1866 bars. But I guess I can only get the speed until 1866 as it is limited by the CPU (The motherboard limits it to 2133). I use company's cluster. I don't dare to use overclocking with Xeon processor.

4. The Infiniband is an area quite fresh to me. I read a lot of discussions about Infiniband for CFD in this forum. But I haven't figured out how fast shall it be. The Infiniband switch is a very expensive stuff. The price of it varies from 1000 U.S. dollars to 20000 U.S. dollars, so does for the Infiniband cards. The type of it could be 2x, 4x and 12x. Is there any indication or any paper or discussions about the selection of the infiniband?

The XEON e5-1600 series are single socket processors. They will not work in a dual socket motherboard. That is what the 1 stands for. Xeon E5-2600 series are for two sockets. and E5-4600 are for 4 sockets.
Regarding XEON vs.i7
E5-1620 = i7-3820
E5-1650 = i7-3930K
E5-1660 = i7-3960X
E5-1620v2 = i7-4820K
E5-1650v2 = i7-4930K
E5-1660v2 = i7-4960X

The only real difference is XEON support ECC memory, and higher RAM GB capacity. The i7s have unlocked multipliers (except the i7-3820) so you can run higher CPU clock frequencies with the i7's than the XEONs.

Your memory speed will be dependent on your motherboards capabilities. You say you don't want to overclock, but if you get anything higher than 1866MHz, that is overclocked memory, as the CPU's do not officially support frequencies higher than 1600MHz or 1866 MHz for the v2s (and i7-4 series).

Dual socket motherboards will support all memory channels on all CPUs. The ASUS one you mentioned probably meant 4 channel for each CPU, not 4 channels total.

Get used infiniband cards and a switch off ebay, new stuff is crazy expensive.


Whatever you pick, I would suggest building one of these and testing it out the stability and compatibility before building 6 or 8 of them.

Remember, you will have to build, test, load software and infiniband drivers, and ensure the compatibility and stability of 8 of these things. I wouldn't recommend going for super-high speed over-clocked, non-official memory speed configurations, or you may have a heck of a time getting everything running properly. It sounds like a difficult enough project even if your computers behave perfectly.

Could I just put 2 or 3 E5-4xxx serie CPU in a 4 socket motherboard?

If I only connect 2 workstation by infiniband, do I still have to buy an infiniband switch? Could I just do a port to port connecting for 2 workstations with infiniband port?

You can put 4 E5-4600 CPU's in a 4 socket motherboard, I'm not sure if they will work with only 2 or 3 in the motherboard. Scaling says "4S only" so that may mean it has to be 4.

You don't need to buy an infiniband switch if you are only using 2 machines. Card to Card will work perfectly.

agreed there is a non trivial amount of work in this. Do able but it will take a bit of time

Erik is right, the asus z9ped8 ws has 8 memory slots, one for memory channel (2xcpus), so the memory will work in quad channel for each cpu.
I have 2 of these motherboards, I think they are good.
Take care to ram: initially I bought 64gb corsair non ecc ram (not overclocked) and I had a lot of problems because fluent crashed every 3-4 hours: switching to samsung ecc solved the problem..I couldn't understanx what was the real problem, maybe it was related to the motherboard-ram compatibility.
About infiniband cards, if you go for used stuff, I suggest to buy mellanox cards.
If you choose other brands you could have problems to make them work.
With old mellanox 20 gbs cards you can find easily drivers (I was able to make the cards to work without problems in win7 and sles 11 sp2).
I have posted a tutorial on how to setup a mini cluster (2 workstation connected by infiniband, without switch) in the fluent section (for win7), maybe it will be usefull for you.

http://www.cfd-online.com/Forums/flu...band-win7.html

I'm switching to sles now and hopefully I will write a similar tutorial.

EDIT: here is new tutorial about 2-nodes cluster with SLES:
http://www.cfd-online.com/Forums/flu...les11-sp2.html

Daniele