CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Hardware

Choose motherboard for cluster building for CFD application

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree4Likes
  • 1 Post By Anna Tian
  • 1 Post By CapSizer
  • 1 Post By evcelica
  • 1 Post By ghost82

Reply
 
LinkBack Thread Tools Display Modes
Old   July 16, 2014, 09:26
Question Choose motherboard for cluster building for CFD application
  #1
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 8
Anna Tian is on a distinguished road
Hi,

I'm building my cluster for running Fluent. I have already fixed the CPU that I'm gonna to choose. And I have found several motherboards which are compatible with that CPU. But I have no idea about how to compare these motherboards to select the one which can achieve better CFD speed.

Which motherboard factors shall I look at? I heard about motherboard is important for the data exchange speed between the components installed on it. Which speed is important for CFD by Fluent?

By the way, the cluster will be composed of 8 motherboards. The CPU I chose is Intel Xeon E5-1620. I'm not bothering you by asking which motherboard is good. I'm just asking the way to choose more suitable motherboard.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   July 17, 2014, 09:28
Default
  #2
Senior Member
 
Charles
Join Date: Apr 2009
Posts: 184
Rep Power: 10
CapSizer is on a distinguished road
I don't think that there is a great varieties of chipset that support Xeon E5 CPU's. So it is unlikely that you will find a substantial difference in performance, especially because the memory controller is part of the CPU anyway. So you need to choose your motherboard on features, not performance. The features that you would need to evaluate are things like form factor, number of DIMM slots, number of PCI-X slots, number of network ports and weigh up the cost-implication of a built-in Infiniband NIC vs. plug in cards.
CapSizer is offline   Reply With Quote

Old   July 18, 2014, 03:30
Question
  #3
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 8
Anna Tian is on a distinguished road
Quote:
Originally Posted by CapSizer View Post
I don't think that there is a great varieties of chipset that support Xeon E5 CPU's. So it is unlikely that you will find a substantial difference in performance, especially because the memory controller is part of the CPU anyway. So you need to choose your motherboard on features, not performance. The features that you would need to evaluate are things like form factor, number of DIMM slots, number of PCI-X slots, number of network ports and weigh up the cost-implication of a built-in Infiniband NIC vs. plug in cards.
Hi CapSizer,

Could you comment on my CPU selection and motherboard selection for CFD application? This will be my first time to purchase and assemble a cluster. I read the discussions on this forum quite carefully but Iím still not confident about my choice. Hope someone could let me know if I select something wrong.

Iím going to setup a 32 cores cluster for only CFD application. The solver will be Fluent.

I select CPU Intel Xeon E5-1620 v2 which has four 3.7 GHz cores and max memory bandwidth 59.7 GB/s. Each of it only costs 370 US dollars in U.S. and less than 300 US dollars in China. I think its price in Europe won't be too different. I will connect 8 or 16 E5-1620 v2 CPUs to have 32 cores. This is the best maximum memory bandwidth (GB/s) per dollar I found.

I select motherboard ASUS P9X79 to match this CPU. This motherboard is 400 US dollars for each and is mainly for desktop application. It only has one socket for CPU. I will have to buy 8 these expensive motherboards in order to have 32 cores in total. But I didnít find a cheaper choice for this. The CPU Intel Xeon E5-1620 gives too good bandwidth (GB/s) per dollar and 3.7 GHz. So I have to use large number of this expensive motherboard to support the selected CPU.

I will choose 1866 GHz memory sticks as the motherboard supports max 2133 GHz and CPU supports only 1866 GHz. The memory brand and version is constraint by the motherboardís memory suggestion list. This is my headache. The list is too short and I canít find the suggested memory in the market unless I import them from U.S. Could I select out of motherboard suggestion list memory but of a big brand like Kinston? It seems currently the compatibility of memory is always very good. Btw, someone told me that 1600 GHz memory is much more stable than 1833 GHz memory. Is this true? Iím confusing about that. That means I have to degrade the performance of my cluster by choosing a weaker memory for stability reason. Any thought or suggestion on that?
Freewill1 likes this.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   July 18, 2014, 03:56
Default
  #4
Senior Member
 
Charles
Join Date: Apr 2009
Posts: 184
Rep Power: 10
CapSizer is on a distinguished road
Meimei, you are asking some good questions! First of all though, if you are using a single socket motherboard, it is not sensible to use the Xeon E5 CPU, which is intended for dual socket boards. You will do better I think with Core i7 CPU's, either the 4-core 4820 K, which looks like good value for money, or perhaps the 6-core 4930 K. The LGA 2011 i7's have access to very good memory bandwidth from the 4 memory channels, so I expect that you will see a worthwhile performance improvement from the extra 2 cores, although it sounds as if you are constrained by software licensing to 32 cores overall? But with 6-core CPU's, you would only need 6 nodes instead of 8, which will mean that you save on motherboards, chassis and network cards, to make up for the more expensive CPU's.

As for memory, in your position I would look for Kingston 2400 MHz memory. If it is unstable you can always reduce the memory speed a bit, but if it works you will have a worthwhile amount of extra performance.

However, to get this cluster performing anywhere near its potential, using Infiniband is essential, despite the expense. Gb ethernet is not good enough. If you cannot afford new cards, cables and switch (small IB switches are rare), take a look at e-bay. You can get older 20 Gbps DDR IB hardware for relatively little, but you will need to invest some effort to make it all work.
Anna Tian likes this.
CapSizer is offline   Reply With Quote

Old   July 18, 2014, 04:31
Question
  #5
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 8
Anna Tian is on a distinguished road
Quote:
Originally Posted by CapSizer View Post
Meimei, you are asking some good questions! First of all though, if you are using a single socket motherboard, it is not sensible to use the Xeon E5 CPU, which is intended for dual socket boards. You will do better I think with Core i7 CPU's, either the 4-core 4820 K, which looks like good value for money, or perhaps the 6-core 4930 K. The LGA 2011 i7's have access to very good memory bandwidth from the 4 memory channels, so I expect that you will see a worthwhile performance improvement from the extra 2 cores, although it sounds as if you are constrained by software licensing to 32 cores overall? But with 6-core CPU's, you would only need 6 nodes instead of 8, which will mean that you save on motherboards, chassis and network cards, to make up for the more expensive CPU's.

As for memory, in your position I would look for Kingston 2400 MHz memory. If it is unstable you can always reduce the memory speed a bit, but if it works you will have a worthwhile amount of extra performance.

However, to get this cluster performing anywhere near its potential, using Infiniband is essential, despite the expense. Gb ethernet is not good enough. If you cannot afford new cards, cables and switch (small IB switches are rare), take a look at e-bay. You can get older 20 Gbps DDR IB hardware for relatively little, but you will need to invest some effort to make it all work.
I chose Intel Xeon rather than i7 for two reasons.

1) Fluent suggests Xeon. I think Fluent is somehow designed for Xeon series.
2) i7 is a desktop CPU serie. And Xeon is for server (and cluster?). It looks like i7 is not designed for connected nodes.

Besides changing the CPU, another way shall be changing the motherboard. Some 'per CPU cheap' motherboard declares that it may not support some features of the CPU which makes the motherboard selection a difficult task. Shall I theoretically study how to select motherboard for a certain CPU rather than just look at the recommendation from the motherboard website which tells not enough information about how much the motherboard compatible with the CPU?

Your suggestion on memory selection is very interesting to me. If the CPU only supports at most 1866 GHz and motherboard supports at most 2133 GHz. Won't be a waste to use 2400 GHz memory? Do you mean overclocking? How to achieve that?

By the way, I saw you have a lot of very helpful discussions in this forum. But I didn't find a clear answer this question (someone has asked it). If I have 8 places for memory cards, shall I go for 4*8 GB or 8*4 GB?
__________________
Best regards,
Meimei

Last edited by Anna Tian; July 18, 2014 at 05:45.
Anna Tian is offline   Reply With Quote

Old   July 18, 2014, 14:31
Default
  #6
Senior Member
 
Charles
Join Date: Apr 2009
Posts: 184
Rep Power: 10
CapSizer is on a distinguished road
OK, lotta questions in one post:

  1. No, there is very little difference between Xeon and i7. Basically the same processor, but one intended for multi-socket server, and the other for desktop workstation. The most important difference is the QPI architecture, which allows the Xeon to communicate at very high speed with another CPU on the same motherboard.
  2. Correct, but i7 will be fine for a cluster, you just cannot use it in a multi-socket motherboard.
  3. Look for the motherboard that is compatible with the CPU, in this case LGA 2011, and supports the required memory speed.
  4. You can "tune" the overclocking settings in the BIOS. Get an enthusiastic kid to help you. i7 boards make provision for overclocking, Xeon boards don't. You're not trying to squeeze the last bit of speed out of the CPU, you just want to activate the best memory speed that your system will support.
  5. For the single socket board 4X8 and 8X4 will be more or less the same, but I think you will find that the very fast memory modules will only come as 4 GB units. On the single socket LGA 2011 i7 system, you have 4 parallel memory channels. 8 DIMMS will not give you more speed.
CapSizer is offline   Reply With Quote

Old   July 19, 2014, 06:41
Question
  #7
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 8
Anna Tian is on a distinguished road
Quote:
Originally Posted by CapSizer View Post
OK, lotta questions in one post:
  1. No, there is very little difference between Xeon and i7. Basically the same processor, but one intended for multi-socket server, and the other for desktop workstation. The most important difference is the QPI architecture, which allows the Xeon to communicate at very high speed with another CPU on the same motherboard.
  2. Correct, but i7 will be fine for a cluster, you just cannot use it in a multi-socket motherboard.
  3. Look for the motherboard that is compatible with the CPU, in this case LGA 2011, and supports the required memory speed.
  4. You can "tune" the overclocking settings in the BIOS. Get an enthusiastic kid to help you. i7 boards make provision for overclocking, Xeon boards don't. You're not trying to squeeze the last bit of speed out of the CPU, you just want to activate the best memory speed that your system will support.
  5. For the single socket board 4X8 and 8X4 will be more or less the same, but I think you will find that the very fast memory modules will only come as 4 GB units. On the single socket LGA 2011 i7 system, you have 4 parallel memory channels. 8 DIMMS will not give you more speed.

1. I look at the i7 list but didn't find any i7 CPU with comparable performance/price to Xeon E5-1620 v2. The Xeon E5-1620 v2 only costs less than 360 U.S. dollars. The only comparable i7-4960X with 6 cores, same bandwidth, 3.6 GHz (E5-1620 v2 is 3.7 GHz) costs more than 1200 U.S. dollars. It is much weaker than Xeon E5-1620 v2 for CFD application. Having two more cores with constant memory bandwidth is not very meaningful for CFD, right?

2. I look at the dual CPU motherboard and found that normally they limit the memory speed to 1600 GHz or even lower. The ASUS Z9PE-D8 dual motherboard is attractable. But when I realize that it only has 4 memory channel in total (2 memory channel per CPU), I don't think it is nice for CFD application. I guess the dual CPU motherboard is designed for servers, not for CFD.

For these reasons, probably it's better to stay with the single CPU mother board ASUS P979 with Xeon E5-1620 v2.

PS: When I read the discussions in the forum, I see that people here usually agree on the fact that the memory bandwidth per dollar is an important factor to CFD hardware performance. But the number of cores per dollar is also an important factor. And also the data exchange rate between the motherboards is important. We have to also minimize the number of motherboards to improve the CFD hardware performance by reducing the data transfer time between the motherboards. It seems that we canít use single criteria to decide the hardware selection (correct me if that's not correct). How to balance these 4 factors?

For my case, we share licenses with another 3 companies in the same building. The licenses number is not a concern to us. The main concern is hardware price.

3. Thanks for your answer about memory. I would follow your advice and go for Kinston 4g 2400 bar, even though they are not in the motherboard suggestion list. They are almost the same price as 1866 bars. But I guess I can only get the speed until 1866 as it is limited by the CPU (The motherboard limits it to 2133). I use company's cluster. I don't dare to use overclocking with Xeon processor.

4. The Infiniband is an area quite fresh to me. I read a lot of discussions about Infiniband for CFD in this forum. But I haven't figured out how fast shall it be. The Infiniband switch is a very expensive stuff. The price of it varies from 1000 U.S. dollars to 20000 U.S. dollars, so does for the Infiniband cards. The type of it could be 2x, 4x and 12x. Is there any indication or any paper or discussions about the selection of the infiniband?
__________________
Best regards,
Meimei

Last edited by Anna Tian; July 19, 2014 at 13:14.
Anna Tian is offline   Reply With Quote

Old   July 21, 2014, 09:04
Default
  #8
Senior Member
 
Erik
Join Date: Feb 2011
Location: Earth (Land portion)
Posts: 577
Rep Power: 11
evcelica is on a distinguished road
The XEON e5-1600 series are single socket processors. They will not work in a dual socket motherboard. That is what the 1 stands for. Xeon E5-2600 series are for two sockets. and E5-4600 are for 4 sockets.
Regarding XEON vs.i7
E5-1620 = i7-3820
E5-1650 = i7-3930K
E5-1660 = i7-3960X
E5-1620v2 = i7-4820K
E5-1650v2 = i7-4930K
E5-1660v2 = i7-4960X

The only real difference is XEON support ECC memory, and higher RAM GB capacity. The i7s have unlocked multipliers (except the i7-3820) so you can run higher CPU clock frequencies with the i7's than the XEONs.

Your memory speed will be dependent on your motherboards capabilities. You say you don't want to overclock, but if you get anything higher than 1866MHz, that is overclocked memory, as the CPU's do not officially support frequencies higher than 1600MHz or 1866 MHz for the v2s (and i7-4 series).

Dual socket motherboards will support all memory channels on all CPUs. The ASUS one you mentioned probably meant 4 channel for each CPU, not 4 channels total.

Get used infiniband cards and a switch off ebay, new stuff is crazy expensive.


Whatever you pick, I would suggest building one of these and testing it out the stability and compatibility before building 6 or 8 of them.

Remember, you will have to build, test, load software and infiniband drivers, and ensure the compatibility and stability of 8 of these things. I wouldn't recommend going for super-high speed over-clocked, non-official memory speed configurations, or you may have a heck of a time getting everything running properly. It sounds like a difficult enough project even if your computers behave perfectly.
evcelica is offline   Reply With Quote

Old   July 23, 2014, 23:58
Question
  #9
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 8
Anna Tian is on a distinguished road
Quote:
Originally Posted by evcelica View Post
The XEON e5-1600 series are single socket processors. They will not work in a dual socket motherboard. That is what the 1 stands for. Xeon E5-2600 series are for two sockets. and E5-4600 are for 4 sockets.
Regarding XEON vs.i7
E5-1620 = i7-3820
E5-1650 = i7-3930K
E5-1660 = i7-3960X
E5-1620v2 = i7-4820K
E5-1650v2 = i7-4930K
E5-1660v2 = i7-4960X

The only real difference is XEON support ECC memory, and higher RAM GB capacity. The i7s have unlocked multipliers (except the i7-3820) so you can run higher CPU clock frequencies with the i7's than the XEONs.

Your memory speed will be dependent on your motherboards capabilities. You say you don't want to overclock, but if you get anything higher than 1866MHz, that is overclocked memory, as the CPU's do not officially support frequencies higher than 1600MHz or 1866 MHz for the v2s (and i7-4 series).

Dual socket motherboards will support all memory channels on all CPUs. The ASUS one you mentioned probably meant 4 channel for each CPU, not 4 channels total.

Get used infiniband cards and a switch off ebay, new stuff is crazy expensive.


Whatever you pick, I would suggest building one of these and testing it out the stability and compatibility before building 6 or 8 of them.

Remember, you will have to build, test, load software and infiniband drivers, and ensure the compatibility and stability of 8 of these things. I wouldn't recommend going for super-high speed over-clocked, non-official memory speed configurations, or you may have a heck of a time getting everything running properly. It sounds like a difficult enough project even if your computers behave perfectly.
Could I just put 2 or 3 E5-4xxx serie CPU in a 4 socket motherboard?

If I only connect 2 workstation by infiniband, do I still have to buy an infiniband switch? Could I just do a port to port connecting for 2 workstations with infiniband port?
__________________
Best regards,
Meimei

Last edited by Anna Tian; July 24, 2014 at 03:51.
Anna Tian is offline   Reply With Quote

Old   July 28, 2014, 19:57
Default
  #10
Senior Member
 
Erik
Join Date: Feb 2011
Location: Earth (Land portion)
Posts: 577
Rep Power: 11
evcelica is on a distinguished road
You can put 4 E5-4600 CPU's in a 4 socket motherboard, I'm not sure if they will work with only 2 or 3 in the motherboard. Scaling says "4S only" so that may mean it has to be 4.

You don't need to buy an infiniband switch if you are only using 2 machines. Card to Card will work perfectly.
Anna Tian likes this.
evcelica is offline   Reply With Quote

Old   August 4, 2014, 18:12
Default
  #11
Senior Member
 
Derek Mitchell
Join Date: Mar 2014
Location: UK, Reading
Posts: 134
Rep Power: 5
derekm is on a distinguished road
Quote:
Originally Posted by evcelica View Post
...

Remember, you will have to build, test, load software and infiniband drivers, and ensure the compatibility and stability of 8 of these things. I wouldn't recommend going for super-high speed over-clocked, non-official memory speed configurations, or you may have a heck of a time getting everything running properly. It sounds like a difficult enough project even if your computers behave perfectly.

agreed there is a non trivial amount of work in this. Do able but it will take a bit of time
__________________
A CHEERING BAND OF FRIENDLY ELVES CARRY THE CONQUERING ADVENTURER OFF INTO THE SUNSET
derekm is offline   Reply With Quote

Old   August 5, 2014, 10:11
Default
  #12
Senior Member
 
ghost82's Avatar
 
Daniele
Join Date: Oct 2010
Location: Italy
Posts: 993
Rep Power: 17
ghost82 will become famous soon enough
Erik is right, the asus z9ped8 ws has 8 memory slots, one for memory channel (2xcpus), so the memory will work in quad channel for each cpu.
I have 2 of these motherboards, I think they are good.
Take care to ram: initially I bought 64gb corsair non ecc ram (not overclocked) and I had a lot of problems because fluent crashed every 3-4 hours: switching to samsung ecc solved the problem..I couldn't understanx what was the real problem, maybe it was related to the motherboard-ram compatibility.
About infiniband cards, if you go for used stuff, I suggest to buy mellanox cards.
If you choose other brands you could have problems to make them work.
With old mellanox 20 gbs cards you can find easily drivers (I was able to make the cards to work without problems in win7 and sles 11 sp2).
I have posted a tutorial on how to setup a mini cluster (2 workstation connected by infiniband, without switch) in the fluent section (for win7), maybe it will be usefull for you.

NEW TUTORIAL: setting 2-node cluster with infiniband (WIN7)

I'm switching to sles now and hopefully I will write a similar tutorial.

EDIT: here is new tutorial about 2-nodes cluster with SLES:
NEW TUTORIAL: setting 2-node cluster with infiniband (SLES11 SP2)

Daniele
Anna Tian likes this.

Last edited by ghost82; August 12, 2014 at 10:49.
ghost82 is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
4 cpu motherboard for CFD petrile_83 Hardware 11 April 16, 2012 16:49
HTC: How to choose reference temperature value halcyon STAR-CCM+ 3 April 12, 2012 11:46
How to choose the mean diameter value for dispersed fluid? creddy_trddc CFX 1 October 30, 2011 05:30
Solver and geometry choose for drag coefficient calculation around circular cylinder at large Re lin OpenFOAM Running, Solving & CFD 3 April 16, 2009 10:50
Where can I choose RANS and RNG!!! maoasis FLUENT 4 May 2, 2006 17:19


All times are GMT -4. The time now is 12:58.