CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > General Forums > Hardware

Replacing cluster server with a workstation

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree4Likes
  • 2 Post By flotus1
  • 2 Post By evcelica

Reply
 
LinkBack Thread Tools Display Modes
Old   May 17, 2017, 07:50
Default Replacing cluster server with a workstation
  #1
New Member
 
Kalle Tiitinen
Join Date: Jan 2010
Location: Finland
Posts: 7
Rep Power: 10
KalleT is on a distinguished road
Greetings fellow CFD enthusiasts,

We are running CFD on ANSYS CFX. We have licenses to utilize 32 cores. The most common use is to have four solvers running each using 8 cores. From time to time we also run larger simulations utilizing all 32 cores for one solve. We currently have a cluster with two nodes having 16 cores each (2 * Xeon E5-2670 @ 2.6GHz) connected with gigabit ethernet. We are now planning to replace this hardware as it is now quite old.

Would it make sense of buying a workstation instead of a new cluster server? It seems that I could get 32 cores into one machine using two E5-2683 v4 processors (16 cores each). Will this be slower than the current cluster as the clock speed is smaller (2.1 GHz vs. 2.6 GHz) and memory bandwidth/core is also lower. If you think that we should stick to updating to a new cluster what would be recommendation of processor bearing in mind that we are limited to 32 cores by CFX licenses.

KalleT is offline   Reply With Quote

Old   May 17, 2017, 09:25
Default
  #2
Senior Member
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 1,692
Rep Power: 26
flotus1 will become famous soon enoughflotus1 will become famous soon enough
There are quite a few possible options to replace your current setup. They differ in cost and performance.
Let me list a few, ranked by a rough performance estimate for CFX:
1) Four nodes with dual-CPUs, e.g. Xeon E5-2637v4, 4 cores
2) A quad-socket node e.g. with Xeon E5-4655v4, 8 cores
3) Two nodes with dual-socket CPUs, e.g. Xeon E5-2667v4, 8 Cores
3) Four nodes with single CPUs, e.g. Xeon E5-1660v4 or I7-6900k (8 cores each)
5) A single workstation with dual-CPUs, e.g. Xeon E5-2697Av4 or E5-2683v4 (16 cores each)

You can connect the nodes with Gigabit or 10G-Ethernet, but I would recommend Infiniband if performance is an important factor for the larger simulations -at least if you connect 4 nodes. Of course you don't need to buy several workstations, you can also get the multinode-setups in a server.
The reason why number 5 might even be slower than your current setup is not so much the lower clock speed, this is compensated by the newer CPU architecture. It is the lack of memory bandwidth. 8 channels DDR4-2400 deliver less bandwidth than 16 channels DDR3-1600, provided your current setup has properly populated memory channels. I can not recommend this option.
Which of the other options is suitable for you mostly depends on your budget.

Edit: keep in mind that AMD Naples will be released within the next few months, as well as Intels Skylake-EP. If you are not in a hurry with buying the new system you might want to wait since they will offer a pretty substantial increase in CFD performance thanks to more and faster memory channels.
KalleT and divergence like this.
__________________
Please do not send me CFD-related questions via PM

Last edited by flotus1; May 17, 2017 at 10:42.
flotus1 is offline   Reply With Quote

Old   May 25, 2017, 12:43
Default
  #3
Senior Member
 
Erik
Join Date: Feb 2011
Location: Earth (Land portion)
Posts: 718
Rep Power: 14
evcelica is on a distinguished road
Your current setup is pretty good, You won't gain too much increase except with flotus1's option #1, since that will more than double your memory bandwidth. Options 2-4 are pretty much just the same thing as what you have, with a little bit faster memory.
Your suggestion and option 5 will be slower than what you have
I said won't update my cluster until there is a big change like the hex channel memory that was rumored to come with the Skylake-EP.
I second the Infiniband. You do not need a switch with only 2 nodes, but will with 4.

I see that having 1 memory channel for each core is about the limit of linear scaling (when using double precision) once you start using 2 and 4 cores per memory channel, you are really bottlenecked by the memory bandwidth and won't see much for gains.

Option 1 has 32 cores and 32 memory channels = Beautiful.
KalleT and divergence like this.
evcelica is offline   Reply With Quote

Old   May 26, 2017, 03:33
Default
  #4
New Member
 
Kalle Tiitinen
Join Date: Jan 2010
Location: Finland
Posts: 7
Rep Power: 10
KalleT is on a distinguished road
Thank you guys for your insight. I will ask quotes from our hardware partner as per your suggestions. I will come back to you when I have new setup compared against the current one.
KalleT is offline   Reply With Quote

Old   August 2, 2017, 21:10
Default
  #5
Member
 
Join Date: Jan 2015
Posts: 38
Rep Power: 5
Christophe is on a distinguished road
I would get an Infiniband switch, and hit ebay and get more of that exact same node. Stuff that's a few years old is about 1/6th the cost of new. You'll probably spend $25k on a new workstation. You can get the IB switch, used, and the IB cards and prob 6-8 more nodes for that price. That's what I did. I've benchmarked core for cor e some of the older E5s vs somewhat newer E5s and it was dead even.


Sent from my iPhone using CFD Online Forum mobile app
Christophe is offline   Reply With Quote

Old   September 4, 2017, 03:00
Default Intel new Xeon platform released
  #6
New Member
 
Kalle Tiitinen
Join Date: Jan 2010
Location: Finland
Posts: 7
Rep Power: 10
KalleT is on a distinguished road
Dear all,

Our hardware vendor has now started to offer machines with the latest Intel CPU:s. We also got new licensing scheme for CFD enabling us to use 48 cores for CFX (typically running 4 simulations on 12 cores each).

I am considering to go with 2 compute nodes each having two 12-core Xeon Gold 6146 processors. Do you think if there would be real benefit in having more nodes with less cores per processor instead? I suppose that having 6 memory channels instead of four allows more cores per processor to be efficiently utilized but I haven't found any official recommendations for CFD regarding these new processors.
KalleT is offline   Reply With Quote

Old   September 4, 2017, 09:10
Default
  #7
Senior Member
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 1,692
Rep Power: 26
flotus1 will become famous soon enoughflotus1 will become famous soon enough
I hereby officially recommend Skylake-SP processors over their Broadwell-EP predecessors

Quote:
Do you think if there would be real benefit in having more nodes with less cores per processor instead?
More nodes with less cores per node will be faster, although it is difficult to estimate the difference. I guess it comes down to the question if you can draw any benefit from shorter solution times. If this is the case, I would definitely recommend 4 smaller nodes because it will pay off very soon even if the performance benefit is small. And I don't know how much Ansys charges you for their licenses, I can only guess that over the course of 3 years it is several times more than the cost for 4 cluster nodes. By the way, do you have separate pre/postprocessing workstations or will you use the cluster nodes?
__________________
Please do not send me CFD-related questions via PM
flotus1 is offline   Reply With Quote

Old   September 5, 2017, 01:24
Default
  #8
New Member
 
Kalle Tiitinen
Join Date: Jan 2010
Location: Finland
Posts: 7
Rep Power: 10
KalleT is on a distinguished road
Than you for your reply. I waas thinking about having a separate head node for pre- and post-processing. This I think is question of whether it-department will support us to have Linux on this server. My understanding is that CFD Post for example is not supported for interactive session under Windows Server OS.
KalleT is offline   Reply With Quote

Reply

Tags
ansys, cfx, processors

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
32 CPUs Workstation V.S. Cluster for Fluent Anna Tian FLUENT 40 July 17, 2014 00:10
Cluster v.s. very large workstation Anna Tian Hardware 3 February 16, 2014 19:50
Dual cpu workstation VS 2 node cluster single cpu workstation Verdi Hardware 18 September 2, 2013 03:09
Star-CCM+ at Cluster [Windows Server 2008 R2 Enterprise x64] cwl STAR-CCM+ 2 July 29, 2013 00:10
OpenFOAM141 FoamX and paraView problem on cluster server seigi OpenFOAM Installation 1 January 28, 2008 16:44


All times are GMT -4. The time now is 16:11.