[Noco]

Hello!

And I want to buy for good price 5 old one computers (I have 3 of them now for testing) from 2010:

- 2 CPU Xeon X5660
- Aquarius S5520SC
- Nvidia Quadro FX 1800
- DDR 12x8192 + SSD (1 of 5) DDR 12x4096 (4 of 5)
- Windows 7 Pro
- Ansys CFX R17

I also want to buy 1 new 2 CPU AMD Epyc 7301 in 2018 (it will be start of the new cluster, I plan to buy some extra further)

Questions:
1. What will be the fastest way to link this 5 computers? Infiniband is not possible, so which ethernet/other type?

2. Do I need to change something (like DD3, SSD) to receive significant improvment? Maybe switch to some Linux?

3. I can not find normal instruction how to organise parallel computing (using ansys parallel solver) in CFX R17 (I use old instruction for CFX 10 + ansys manual - it works somehow, but as for now 3% of time is calculation, 97% of time - writing to sata disk). Never did this before. Where can I find good instruction for CFX R17 with parallel solver for 6-7 and more computers?

[flotus1]

it works somehow, but as for now 3% of time is calculation, 97% of time - writing to sata disk

Not quite sure how you measured this or for which setup, but this could indicate one of two problems: 1) Your workload is heavily I/O bound, e.g. by writing lots of results to disk during calculation 2) Your models do not fit in memory
The solution to 1 is not adding more PCs, but making sure that you do not write more results than absolutely necessary and getting fast SSD storage.

Why exactly is infiniband not possible? Used IB cards are even cheaper than 10 gigabit ethernet cards.

[Noco]

By clock on the wall

Well we always open CPU load graphs during calculations. They were active 3% of time (about 5-7 minutes), then 7 hours the computer was thinking, and after this 7 hours he provided result.

Normally we have:
1. start solver, all adjustments, starting 1 iteration,
- all CPU cores are 80-100% load
- RAM is loaded 10-12%

2. The iteration calculation finished
- all CPU cores - 1-2%
- RAM is loaded 10-12%
- SATA is writing/checking the writing/add to the folder (we found some ghost-writing protocol, which made some ghost-copies (from Windows 10) who make mistakes sometimes and delete this process)

3. Next iteration
- the grid of the new model is formed by criteria
- all CPU cores - 80-100%
.......

The normal cycle time we have is about 11-13 minutes per iteration. With 1 computer. With 3 computers using parallel - about 7 minutes, but another 7 hours to provide result.

[Rec]

Hello!

Tell me what model of the IB-switch is used for you, what IB-cards are installed in the computers, what is their bandwidth? Using a trunk connection(2 or more connection to 1 PC)?

Thank you.

Originally Posted by flotus1

Not quite sure how you measured this or for which setup, but this could indicate one of two problems: 1) Your workload is heavily I/O bound, e.g. by writing lots of results to disk during calculation 2) Your models do not fit in memory
The solution to 1 is not adding more PCs, but making sure that you do not write more results than absolutely necessary and getting fast SSD storage.

Why exactly is infiniband not possible? Used IB cards are even cheaper than 10 gigabit ethernet cards.

[Noco]

I am also interested in IB-switch model, IB-cards, bandwidth.

What is best-practice?

[Noco]

Regarding my 'old' cluster, based on 5 computers 2 CPU Xeon 5660 (2*6 cores, 2,8 GHz, 3 memory channels).

Please advise is it make sense to add in this cluster using Ansys parallel solver:

1. One computer from 2017
- 1 CPU i9 7980XE (1*18 cores, 2,6 GHz, 4 memory channels)
- ASUS S2066 PRIME X299-A RTL
- GPU GeForce PCI-E 11264Mb 1080 Ti InnoVision (not helps actually)
- 64Gb DDR4 2133 MHz (4x16)
- 512 SSD for system
- 2x8 Tb SATA RAID0 for storage
- Windows 10
- ANSYS CFX R17

P.S. We tested i9, using the same task, but with different amount of cores and receive quite not understandable results (attached). 10 cores are faster, then 18.

2. Two computers from 2018
- 2 CPU AMD EPYC 7301 (2*16 cores, 2,2 GHz, 8 memory channels)
- Supermicro H11DSi
- GPU - no or 1060 Ti
- 512 SSD for system
- 2x8 Tb SATA RAID0 for storage
- Windows 10
- ANSYS CFX R17

What will be the bottlenecks of this system and does it make sense to make it all together? Or better to run separately?