[evan247]

Hi all,

I'm currently a research student at university and I'm planning to do some heavy duty CFD with CFX within the next 12 month (~40m mesh URANS). As I understand, ANSYS allows you to use up to 16 cores on one computer, and you need an extra HPC license for every extra core you use. Actually the cost for extra 16 HPC licenses for a year's lease seems quite reasonable (a lot cheaper than a workstation).

Currently, my group has two Xeon workstations, both having dual E5 2640 v3 (8 core x 2.6GHz). One of them has 32GB RAM and the other with 128GB (4x32GB). I'm thinking of the following upgrades:

1. Use 8 memory sticks (16GB each) for both workstations, to fully utilise the quad channel feature. Please correct me if I'm wrong, but does 4x32GB on one of my machines mean that I'm not using all the memory bandwidth? How much more performance can I gain from this upgrade?

2. Is it possible to connect the two workstations via Infiniband? Or 10Gb Ethernet is the best I can do? Would it make a huge difference for a 2U 'server' to have infiniband? I'm looking at Mellanox's website, but haven't got a clue what card/switches I should be buying, and roughly how much I'm expected to pay for them.

3. If it's impossible to realise the option above, I will probably get a 2U server and below is what I have in mind:

* Dual Xeon E5 2667 v4 (8 core, 3.2GHz, 25MB cache, 76.8 GB/s bandwidth)
* 8 x 16GB DDR4 2400

I thought the above balances the cost of both the system and the licences, as well as bandwidth and single core performance. Any advice would be much appreciated here. Thanks!

[flotus1]

As I understand, ANSYS allows you to use up to 16 cores on one computer, and you need an extra HPC license for every extra core you use

I don't think that there is a limit on the number of cores you can use on a single node.

1. Use 8 memory sticks (16GB each) for both workstations, to fully utilise the quad channel feature. Please correct me if I'm wrong, but does 4x32GB on one of my machines mean that I'm not using all the memory bandwidth? How much more performance can I gain from this upgrade?

You are right. You would need at least 8 DIMMs to use the full potential of the machine. For Ansys CFX with a cell size of 40mio and 16 cores used the benefit wil be somewhere between 50% to almost 100%.

2. Is it possible to connect the two workstations via Infiniband? Or 10Gb Ethernet is the best I can do? Would it make a huge difference for a 2U 'server' to have infiniband? I'm looking at Mellanox's website, but haven't got a clue what card/switches I should be buying, and roughly how much I'm expected to pay for them.

Of course you can use Infiniband, you won't even need a switch for 2 nodes. Setting it up properly will require some research or the help of an expert. You are in for ~600$ per card and ~100$ for a cable. If this is too much, ebay is always overflowing with cheap used Infiniband hardware.
Edit: found a tutorial NEW TUTORIAL: setting 2-node cluster with infiniband (WIN7)

3. If it's impossible to realise the option above, I will probably get a 2U server and below is what I have in mind:
* Dual Xeon E5 2667 v4 (8 core, 3.2GHz, 25MB cache, 76.8 GB/s bandwidth)
* 8 x 16GB DDR4 2400
I thought the above balances the cost of both the system and the licences, as well as bandwidth and single core performance. Any advice would be much appreciated here. Thanks!

A good setup for a CFD workstation. However, it will not be dramatically faster than the workstations you already have, provided you fix the memory configuration.

[evan247]

Thank you flotus1.

I'm currently in contact with a company called Flowsim based in the U.K. And they recommended going for 6 or 8 nodes of i7-6800k with infiniband connection. For a 36 core cluster it will be around £10k incl. their service to help set everything up. From your experience, how will it perform compared to my existing 16core E5v3 workstation? Given it's a 6 core CPU should I expect close to 36/8=4.5 speed up (since I haven't upgraded the memories)?

Alternatively I can spend some time researching and get my existing two workstations upgraded (memory wise) and connected via infiniband. That amounts to 32 cores but each core being a bit slower in terms of clock speed and memory bandwidth. Factorising this in, is it fair to say I either end up paying £10k for a 4.5x speed up with less hassle of setups, or around £1.2k (memory) + £1k (infiniband) for 3-4x speedup? If so the latter choice is more cost effective.

[flotus1]

Are you the only person using the computers? Otherwise you should factor in that with your first option, get additional resources and still can use the ones you already have. Fix the memory and you get two pretty decent CFD workstations, even without Infiniband interconnect.

[evan247]

Originally Posted by flotus1

Are you the only person using the computers? Otherwise you should factor in that with your first option, get additional resources and still can use the ones you already have. Fix the memory and you get two pretty decent CFD workstations, even without Infiniband interconnect.

Hi flotus1,

I will be the only one using the computer; for the two existing workstations I have sole access to at least one of them (which is mainly used for short jobs or post-processing now).

I've been doing quite a bit of research and here's what I thought is a balanced option for 36 CPU cores (physical):

6-node cluster with QDR infiniband connection, each node with:

* single socket E5 1650 v4 CPU (6 core)
* 32GB DDR4 2400
* A cheap GPU that works with Linux
* SSD+HDD for head node only

I was initially recommend i7-6800K but I thought Xeon would be more reliable in the long run. I'd like to hear your comments on the configuration above (it'll come to around £10,000 without IB switches and cards), and if it's worth going for dual socket of E5 2643 v4 instead (which I believe will be more pricy). Thank you!

[RobertB]

It would make most sense to build up here.

Put in the memory to utilize all the available bandwidth. Make sure each CPU has enough memory locally to run all its processes.

Set up the run to bind the processes to a CPU. This, at least on CCM+, yields a significant gain.

See how the a representative run scales first on one node then on both using the existing networking (1G ethernet).

Look at the scaling if it scales badly across two nodes a better network interface would help, if not it won't.

It is probable by the time you get to 8 cores on a CPU that you are flattening the memory bandwidth. A higher frequency might help (and the v4s have a higher memory bandwidth) but probably less than 1:1.

Buy beer with saved money, or a faster system if that looks like it will give you significant gains.