|
[Sponsors] |
December 14, 2016, 19:35 |
Hardware config for 32 ANSYS Licenses
|
#1 |
New Member
Join Date: Jan 2015
Posts: 29
Rep Power: 11 |
Hi all,
I'm currently a research student at university and I'm planning to do some heavy duty CFD with CFX within the next 12 month (~40m mesh URANS). As I understand, ANSYS allows you to use up to 16 cores on one computer, and you need an extra HPC license for every extra core you use. Actually the cost for extra 16 HPC licenses for a year's lease seems quite reasonable (a lot cheaper than a workstation). Currently, my group has two Xeon workstations, both having dual E5 2640 v3 (8 core x 2.6GHz). One of them has 32GB RAM and the other with 128GB (4x32GB). I'm thinking of the following upgrades: 1. Use 8 memory sticks (16GB each) for both workstations, to fully utilise the quad channel feature. Please correct me if I'm wrong, but does 4x32GB on one of my machines mean that I'm not using all the memory bandwidth? How much more performance can I gain from this upgrade? 2. Is it possible to connect the two workstations via Infiniband? Or 10Gb Ethernet is the best I can do? Would it make a huge difference for a 2U 'server' to have infiniband? I'm looking at Mellanox's website, but haven't got a clue what card/switches I should be buying, and roughly how much I'm expected to pay for them. 3. If it's impossible to realise the option above, I will probably get a 2U server and below is what I have in mind: * Dual Xeon E5 2667 v4 (8 core, 3.2GHz, 25MB cache, 76.8 GB/s bandwidth) * 8 x 16GB DDR4 2400 I thought the above balances the cost of both the system and the licences, as well as bandwidth and single core performance. Any advice would be much appreciated here. Thanks! |
|
December 15, 2016, 12:47 |
|
#2 | ||||
Super Moderator
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,399
Rep Power: 46 |
Quote:
Quote:
Quote:
Edit: found a tutorial NEW TUTORIAL: setting 2-node cluster with infiniband (WIN7) Quote:
|
|||||
December 15, 2016, 14:03 |
|
#3 |
New Member
Join Date: Jan 2015
Posts: 29
Rep Power: 11 |
Thank you flotus1.
I'm currently in contact with a company called Flowsim based in the U.K. And they recommended going for 6 or 8 nodes of i7-6800k with infiniband connection. For a 36 core cluster it will be around £10k incl. their service to help set everything up. From your experience, how will it perform compared to my existing 16core E5v3 workstation? Given it's a 6 core CPU should I expect close to 36/8=4.5 speed up (since I haven't upgraded the memories)? Alternatively I can spend some time researching and get my existing two workstations upgraded (memory wise) and connected via infiniband. That amounts to 32 cores but each core being a bit slower in terms of clock speed and memory bandwidth. Factorising this in, is it fair to say I either end up paying £10k for a 4.5x speed up with less hassle of setups, or around £1.2k (memory) + £1k (infiniband) for 3-4x speedup? If so the latter choice is more cost effective. |
|
December 17, 2016, 10:25 |
|
#4 |
Super Moderator
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,399
Rep Power: 46 |
Are you the only person using the computers? Otherwise you should factor in that with your first option, get additional resources and still can use the ones you already have. Fix the memory and you get two pretty decent CFD workstations, even without Infiniband interconnect.
|
|
January 22, 2017, 14:51 |
|
#5 | |
New Member
Join Date: Jan 2015
Posts: 29
Rep Power: 11 |
Quote:
I will be the only one using the computer; for the two existing workstations I have sole access to at least one of them (which is mainly used for short jobs or post-processing now). I've been doing quite a bit of research and here's what I thought is a balanced option for 36 CPU cores (physical): 6-node cluster with QDR infiniband connection, each node with: * single socket E5 1650 v4 CPU (6 core) * 32GB DDR4 2400 * A cheap GPU that works with Linux * SSD+HDD for head node only I was initially recommend i7-6800K but I thought Xeon would be more reliable in the long run. I'd like to hear your comments on the configuration above (it'll come to around £10,000 without IB switches and cards), and if it's worth going for dual socket of E5 2643 v4 instead (which I believe will be more pricy). Thank you! |
||
January 23, 2017, 07:33 |
|
#6 |
Senior Member
Robert
Join Date: Jun 2010
Posts: 117
Rep Power: 16 |
It would make most sense to build up here.
Put in the memory to utilize all the available bandwidth. Make sure each CPU has enough memory locally to run all its processes. Set up the run to bind the processes to a CPU. This, at least on CCM+, yields a significant gain. See how the a representative run scales first on one node then on both using the existing networking (1G ethernet). Look at the scaling if it scales badly across two nodes a better network interface would help, if not it won't. It is probable by the time you get to 8 cores on a CPU that you are flattening the memory bandwidth. A higher frequency might help (and the v4s have a higher memory bandwidth) but probably less than 1:1. Buy beer with saved money, or a faster system if that looks like it will give you significant gains. |
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Computer hardware for Ansys fluent | zombiaska | FLUENT | 0 | September 26, 2013 06:49 |
ANSYS Fluent hardware requirements | zombiaska | Hardware | 0 | September 16, 2013 14:10 |
How to configure a workstation hardware for best performance in ansys | acasas | Hardware | 1 | May 9, 2013 11:45 |
Hardware for running ANSYS Fluent 12.1 | ngj_22 | Hardware | 2 | September 29, 2010 07:26 |
Ansys installation on VISTA 32/64 bit sol'n! | Andrzej F | CFX | 0 | September 22, 2008 16:51 |