CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > ANSYS > CFX

Parallel Computing for ANSYS CFX R17

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree3Likes
  • 1 Post By ghorrocks
  • 1 Post By urosgrivc
  • 1 Post By ghorrocks

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   January 14, 2018, 05:12
Default Parallel Computing for ANSYS CFX R17
  #1
Member
 
Ivan
Join Date: Oct 2017
Location: 3rd planet
Posts: 34
Rep Power: 8
Noco is on a distinguished road
Hello!

I have 1 new computer:

1 CPU i9 7980XE with Kraken X62
- ASUS S2066 PRIME X299-A RTL
- GeForce PCI-E 11264Mb 1080 Ti InnoVision (not helps actually)
- 64Gb DDR4 2133 MHz (4x16)
- 512 SSD for system
- 2x8 SATA RAID0 for storage
- Windows 10
- ANSYS CFX R17

And I can buy for good price 5 old one from 2010:

2 CPU Xeon X5660
- Aquarius S5520SC
- Nvidia Quadro FX 1800
- DDR 12x8192 + SSD (1 of 5) DDR 12x4096 (4 of 5)
- Windows 7 Pro

I also want to buy 1 new 2 CPU Xeon Gold / AMD Epyc in 2018 (it will be 7 computer)

Questions:
1. I can not find normal instruction how to organise parallel computing (using ansys parallel solver) in CFX R17 (I use old instruction for CFX 10 + ansys manual - it works somehow, but as for now 3% of time is calculation, 97% of time - writing to sata disk). Never did this before. Where can I find good instruction for CFX R17 with parallel solver for 6-7 and more computers?

2. Is it make sense to buy this old computers, install ethernet cards, buy extra licenses, and use them as slaves with new computer? I will have 5*12+18= 78 cores system, but what will be bottlenecks for this system?

Thank you in advance!

Last edited by Noco; January 14, 2018 at 09:12.
Noco is offline   Reply With Quote

Old   January 14, 2018, 18:14
Default
  #2
Super Moderator
 
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,639
Rep Power: 142
ghorrocks is just really niceghorrocks is just really niceghorrocks is just really niceghorrocks is just really nice
For distributed parallel computing, all nodes need to be the same computer. Mixing computer speeds will not give you an effective speedup.

The parallel setup has not changed in CFX for a long time, across many versions. Your comment that your cluster does 3% calculation and 97% disk access probably means one of the following:
1) At least one node has run out of memory
2) You are writing too many results files
ghorrocks is online now   Reply With Quote

Old   January 15, 2018, 04:11
Default
  #3
Member
 
Ivan
Join Date: Oct 2017
Location: 3rd planet
Posts: 34
Rep Power: 8
Noco is on a distinguished road
How can I check that one or more nodes run out of the memory?

I run 3 old computers as a test:

1 master:

2 CPU Xeon X5660
- Aquarius S5520SC
- Nvidia Quadro FX 1800
- DDR 12x8192
- SSD
- Windows 7 Pro

2 slaves:

2 CPU Xeon X5660
- Aquarius S5520SC
- Nvidia Quadro FX 1800
- SATA
- DDR 12x4096
- Windows 7 Pro

Regarding the 'writing too many results files' - maybe, but it was first time like this, the master was writing during about 7 hours. Can it be related with SSD/HD difference writing speed?
Noco is offline   Reply With Quote

Old   January 15, 2018, 17:35
Default
  #4
Super Moderator
 
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,639
Rep Power: 142
ghorrocks is just really niceghorrocks is just really niceghorrocks is just really niceghorrocks is just really nice
To check memory - log onto the slaves as they are running and look at taskmanager (windows) or top (linux) and check the memory allocated.

If your results files are huge it could take 7 hours to write. But that would require seriously huge files. It could also be your network is very slow, or your virus checker is killing it (or some other background process).
Noco likes this.
ghorrocks is online now   Reply With Quote

Old   January 16, 2018, 01:53
Default
  #5
Member
 
Ivan
Join Date: Oct 2017
Location: 3rd planet
Posts: 34
Rep Power: 8
Noco is on a distinguished road
Ok, I will try to check this.

Other issue:

When I run:
- 2 CPU Xeon X5660
- Aquarius S5520SC
- Nvidia Quadro FX 1800
- DDR 12x4096
- Windows 7 Pro
- ANSYS CFX R17

During iteration I have up to DDR 30Gb load from 48 Gb. All 12 cores are 80-100% load.

But with the same task with new one:
- 1 CPU i9 7980XE with Kraken X62
- ASUS S2066 PRIME X299-A RTL
- GeForce PCI-E 11264Mb 1080 Ti InnoVision (not helps actually)
- 64Gb DDR4 2133 MHz (4x16)
- 512 SSD for system
- 2x8 SATA RAID0 for storage
- Windows 10
- ANSYS CFX R17

During iteration I have only up to DDR4 8 Gb load from 64 Gb. All 18 cores are 90-100% load. Why 18 cores do not use all 64 Gb? There is official and not official recommendation that for CFD we need 8 Gb per core, so 18*8= 144 (128 maximum support for this motherboard), I have 64, but it is nor fully load.
Noco is offline   Reply With Quote

Old   January 16, 2018, 02:06
Default
  #6
Senior Member
 
urosgrivc
Join Date: Dec 2015
Location: Slovenija
Posts: 365
Rep Power: 11
urosgrivc is on a distinguished road
It will only use as much memory as your simulation needs.
How much memory is used is mostly dependant on the mesh size.
More elements you have more ram you need.
And this cannot be because of the diferences betveen the computers
Are you sure that the simulation is exactly the same in both tests?

and The i9 has 4 memory chanels so you dont need more than 4 sticks of ram.
and the dual xeon has 6 total memory chanels, 3 per procesor, so 6 or 12 sticks of ram are prefered for that case.
so you use up as much memory bandwidth as posible.
Noco likes this.

Last edited by urosgrivc; January 17, 2018 at 08:47.
urosgrivc is offline   Reply With Quote

Old   January 17, 2018, 07:26
Default Memory sharing
  #7
Rec
New Member
 
Sergey
Join Date: Jan 2018
Posts: 18
Rep Power: 8
Rec is on a distinguished road
When distributed calculations, the amount of memory will be equally used in 1,2,3 computers?
Or memory is distributed as well as calculations?

Thank you.

Quote:
Originally Posted by urosgrivc View Post
It will only use as much memory as your simulation needs.
How much memory is used is mostly dependant on the mesh size.
More elements you have more ram you need.
And this cannot be because of the diferences betveen the computers
Are you sure that the simulation is exactly the same in both tests.

and The i9 has 4 memory chanels so you dont need more than 4 sticks of ram
Rec is offline   Reply With Quote

Old   January 17, 2018, 17:14
Default
  #8
Super Moderator
 
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,639
Rep Power: 142
ghorrocks is just really niceghorrocks is just really niceghorrocks is just really niceghorrocks is just really nice
In distributed simulations the mesh is divided into segments and each process does one segment. So each process sees a smaller mesh.
Rec likes this.
ghorrocks is online now   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Remote cluster parallel solve without master, Ansys CFX 14.5 aaguirre CFX 4 January 27, 2015 10:15
CFX Parallel ihsb CFX 1 October 15, 2014 18:12
correction number and parallel computing psosnows OpenFOAM Bugs 5 June 17, 2013 21:56
RSH problem for parallel running in CFX Nicola CFX 5 June 18, 2012 19:31
CFX, NT parallel, Linux, best platform Heiko Gerhauser CFX 1 August 21, 2001 10:46


All times are GMT -4. The time now is 17:58.