[mdgowhar]

Hi everyone!

I have ordered High Performance Scientific Computing Servers.

Usage:
STAR-CCM+ [HPC license]
OpenFOAM
Salome for FEA work

Configuration (edited) [ Thanks Alex! I have made mistake while writing.]
Master Node:
2 x AMD Opteron 6308 @3.5GHz 4-core [8 cores]
6 x 16GB RAM 1600MHz ECC Registered [In Total 96 GB]

Compute Node:
4 x AMD Opteron 6308 @3.5GHz 4-core [16 cores]
8 x 16GB RAM 1600MHz ECC Registered [In Total 128 GB]

(1 x 8 cores in Master Node) +(3 x 16 cores in Compute Node) = 56 cores in total

Since buying Infiniband card and switch are out of my budget, I would like to interconnect using Manageable Ethernet switch. Please anyone suggest me which product is best to buy. And Suitable Operating System.

Thanks in advance.

Regards,
Mohammed Gowhar

[flotus1]

If I am not mistaken the memory on your compute nodes is poorly configured.
The AMD Opteron 6308 has 4 memory channels. 4 of these CPUs on one node need 16 DIMMs to achive maximum memory bandwidth.

[mdgowhar]

Originally Posted by flotus1

If I am not mistaken the memory on your compute nodes is poorly configured.
The AMD Opteron 6308 has 4 memory channels. 4 of these CPUs on one node need 16 DIMMs to achive maximum memory bandwidth.

Im sorry.. My mistake

In compute node:
8 x 16GB RAM [In Total 128GB} Equally populated.
This board is capable of 1TB ECC registered RAM

In master node:
6 x 16GB RAM [In Total 96GB} Equally populated.
This board is capable of 512GB ECC registered RAM

[flotus1]

I still think that with only 8 DIMMs per compute node, you are wasting half of the theoretical maximum memory bandwidth of your CPUs. Equally populating the 8 DIMMs among the 4 CPUs will result only in dual-channel mode for each CPU.
See for example the recommendations from Supermicro: http://www.supermicro.com/aplus/memo...%28%2B%29-LN4F
I know this is not the topic of this thread, but I thought it might be worth pointing this out.

[mdgowhar]

I strongly agree with you point. But I dont have budget to buy more RAMs.

Moreover If I buy the 8GB RAM with same total memory size, The total modulus occupied by 8GB and it will be difficult to upgrade in the future and become costlier.
Thats why I opt for dual Rank for now. May be after one or two years I can upgrade it with adding additional 16GB RAM to fill the remaining slots.

I really thank you for pointing out this. Definitely I look forward for alternative solution to have better performance.

Since I already placed order, Is it worth to buy additional 8GB module to fill remaining slots?

[flotus1]

I would avoid this for two reasons.
The first one is stability. Using different types of DIMMs is to some extent a game of chance, especially in a HPC system with a rather complicated memory setup.
The second reason is performance. Even In the ideal case that your system supports this asymmetric quad-socket quad-channel setup, only 2/3 of the total RAM will run in quad-channel mode. The rest can only be accessed much slower which will be a bottleneck for cases that need much RAM. But in reality, this exotic setup could even decrease overall performance.

As far as your original question with the network switches is concerned: maybe buying used Infiniband cards, cables and a switch is an option for you to save some money. It might be worth it.
http://www.hpcadvisorycouncil.com/pd...igE_in_HPC.pdf

Edit: Are you sure that the motherboards you ordered only have 16 DIMM slots? All 4-socket Opteron boards I could find have 32 slots.

[mdgowhar]

Thanks for your valuable suggestion.
Yes, 4-Socket Opteron Mother Board has 32 slots.

If I installed RAM in dual channel mode and run the simulation within 16 cores, how far it will affect the performance (in terms of %).

Because, Already I have other expenses such as Rack Cabinet [42U], KVM Switches and PDU [Schneider Electrics].

[flotus1]

Unfortunately I cant find the original thread where this was discussed. But here are some results that show how memory frequency (and thus bandwidth) can affect performance in CFD.
scaling.jpg

Based on these charts alone, quad-channel memory could increase performance by up to 70% compared to dual-channel. But since the results do not include latency which was probably slower for the faster memory frequencies, the benefit of quad-channel memory could even be higher.

[mdgowhar]

Originally Posted by flotus1

Unfortunately I cant find the original thread where this was discussed. But here are some results that show how memory frequency (and thus bandwidth) can affect performance in CFD.
Attachment 42513

Based on these charts alone, quad-channel memory could increase performance by up to 70% compared to dual-channel. But since the results do not include latency which was probably slower for the faster memory frequencies, the benefit of quad-channel memory could even be higher.

I would like to clarify regarding this.

This chart is based on CPU Frequency Speed, not on Memory Frequency.
I am thinking of alternative solutions and found a post on Unbalanced Memory Performance for High Performance Computing by Dell.

Please review this link.
http://en.community.dell.com/techcen...ry-performance

[flotus1]

Please have a closer look at the chart. Yes, on the horizontal axis we have the CPU frequency. But the data series labeled "1600", "1866" and "2133" indicate memory frequencies.
It is nothing new that performance in CFD applications can correlate with memory bandwidth. I would even say that the only justification to buy Opteron Processors for CFD in 2015 is that they offer the cheapest way to get 16 memory channels on one node. If this is enough to compensate the outdated architecture and the lack of FPUs is a different topic.

[evcelica]

Like Flotus1 says, You really need to balance your memory for good performance. I ran some benchmarks on a 4 CPU XEON with and without unbalanced memory, and it was 2.18x faster with balanced memory. The performance was horrible with unbalanced memory.

The machine had 6 DIMMS per CPU, so it was running in Quad channel configuration, but 2 channels had 2 DIMMS, and 2 channels only had 1 DIMM. I had them purchase 2 more DIMMS per CPU to populate the channels evenly, and performance skyrocketed.

If you can fill the channels evenly, and use all 4 memory channels on each machine, that would be best.
Just use less machines, but properly configure them if you have to, it doesn't help to have more poorly configured machines.

[mdgowhar]

Yea. I got that. Thanks Alex.

Thanks Erik! I would consider your point.

I have a sheet of Memory Configuration for this HPC
Please find the same.

Existing designed is there. Moreover, I have added 4 suggestions.
Kindly review it. If you have any suggestions, kindly modify and send me back.

Existing.png

Suggestion_1.png

Suggestion_2.png

Suggestion_3.png

Suggestion_4.png

[mdgowhar]

Here is the excel sheet to modify easily.

Memory_Config_HPC.xlsx

[flotus1]

So you are searching for the "best" way to misconfigure the memory on your system. I strongly discourage buying smaller DIMMs and mixing them with the existing ones, especially if the budget is already exhausted. For now your best option seems to be to use dual-channel mode for all CPUs. If you really cant change the configuration you ordered this appears to be the only reasonable option.