CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > ANSYS > CFX

Proper cores number to be used in CFX simulation

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree3Likes
  • 1 Post By monkey1
  • 1 Post By ghorrocks
  • 1 Post By Anna Tian

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   March 6, 2015, 23:27
Question Proper cores number to be used in CFX simulation
  #1
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
I'm currently using CFX 15.0 to run my structured grids single phase CFD simulations. My workstation has 4 CPUs and 32 cores in total. I just found that if I use 32 cores for a 0.4 million grids simulation, it will be calculated even slower than the case I put 16 cores to calculate it.


I'm wondering what are the rules to select proper cores number. Is there a function to corelate the most proper cores number and the simulate grids number?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   March 8, 2015, 05:01
Default
  #2
Super Moderator
 
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,665
Rep Power: 143
ghorrocks is just really niceghorrocks is just really niceghorrocks is just really niceghorrocks is just really nice
There is no general rule. The effectiveness of parallelisation is different for every simulation. But as the partitions get smaller the effectiveness reduces and at some point more partitions will run slower.

Also keep in mind:

1) In distributed parallel, that as you get up to around 32 partitions or more that ethernet is unlikely to be adequate. You will probably need a high speed, low latency interconnect like infiniband.

2) In local parallel, many architectures have internal data bottlenecks. For instance a little while ago the front side bus which connected the CPU to the memory in a intel CPU system was a major bottleneck and prevented large numbers of cores being used effectively. So there is a maximum number of cores a single system can use effectively as well.
ghorrocks is offline   Reply With Quote

Old   March 9, 2015, 22:47
Question
  #3
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by ghorrocks View Post
There is no generalrule. The effectiveness of parallelisation is different for every simulation.But as the partitions get smaller the effectiveness reduces and at some pointmore partitions will run slower.


Thanks for your answer, ghorrocks. Even though there is no general rule to relate the cores number I shall use, people shall still need a rough approximation rule as a basic indication. Could you let me know your experience? For example, how many cores would you use if the simulation has 0.4 million or 1 million and the CFX license is not a constraint?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   March 10, 2015, 03:33
Default
  #4
Senior Member
 
Join Date: Jul 2011
Location: Berlin, Germany
Posts: 173
Rep Power: 14
monkey1 is on a distinguished road
According to ANSYS support there IS an "optimum" number of cores. They say 250.000 Elements / core gives the best efficiency (newest Info I got was 500.000 Elements / core is also ok). Meaning a mesh with 400.000 Elements would require 2 cores.
Using more Elements per core slows down the computational speed because every core has too much to do and using less cells slows down the speed because the inter core communication takes too much time compared to the solving speed.
Veronique Pe likes this.
monkey1 is offline   Reply With Quote

Old   March 10, 2015, 04:22
Default
  #5
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by monkey1 View Post
According to ANSYS support there IS an "optimum" number of cores. They say 250.000 Elements / core gives the best efficiency (newest Info I got was 500.000 Elements / core is also ok). Meaning a mesh with 400.000 Elements would require 2 cores.
Using more Elements per core slows down the computational speed because every core has too much to do and using less cells slows down the speed because the inter core communication takes too much time compared to the solving speed.
I see, from your post, that the cores number doesn't increase with elements number linearly. Then if I have simulation of 5 million elements, how many cores shall I use to run the simulation?

I think there must be someone, institute or company did some tests to give quite detailed cores number usage indications for single phase flow. Could anyone share the test results? It is quite strange that Ansys didn't have any words on this issue in its tutorial. I think Ansys should do these tests and publish the results to give users a basic indication.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   March 10, 2015, 04:35
Default
  #6
Senior Member
 
Join Date: Jul 2011
Location: Berlin, Germany
Posts: 173
Rep Power: 14
monkey1 is on a distinguished road
Ansys did these Tests, and when having an Ansys course they tell you about this.
And as I said it is Number of elemets / 250.000 = Number of cores
So for example a 5 Mio Elements simulation wouldirdeally require 5.000.000/250.000=20 Cores!
If you get a decimal value like for your 0.4 Mio Elements / 250.000= 1.6 Cores then just do a round off to the next full number, meaning 2 cores.

For me it worked quit well until now.
monkey1 is offline   Reply With Quote

Old   March 10, 2015, 04:57
Default
  #7
New Member
 
Véronique Penin
Join Date: Mar 2014
Location: Lyon, France
Posts: 21
Rep Power: 12
Veronique Pe is on a distinguished road
Hello,

Thanks for the answer, I had the same question.
But, the number of elements / core is independant of the simulation? single phase or multi phase? transient or steady ? turbulence model of zero equation or two equation or STT? it's just the convergence time will be impact by the simulation choice?

Thanks,

Véronique
Veronique Pe is offline   Reply With Quote

Old   March 10, 2015, 05:05
Default
  #8
Super Moderator
 
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,665
Rep Power: 143
ghorrocks is just really niceghorrocks is just really niceghorrocks is just really niceghorrocks is just really nice
The thing which slow parallel simulations down is the inter-partition communications and coupling. So all the options you describe are just additional equations to solve, so the inter-partition communications will just increase at approximately the same rate as the simulation time - the result being that the speed up factor will be pretty much unchanged.

Things which really affect parallel efficiency are models which interact poorly with multiple partitions. I can only really think of three models which definitely require care on parallel:
* Monte Carlo radiation models (they result in a lot of extra inter-partition communication and can seriously degrade parallel performance.)
* Free surface models (if the free surface is nearly colinear with a partition boundary you will have convergence difficulties)
* Compressible flow with shocks (if the shock is nearly colinear with a partition boundary you will have convergence difficulties)
Veronique Pe likes this.
ghorrocks is offline   Reply With Quote

Old   March 10, 2015, 05:07
Default
  #9
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Veronique Pe View Post
Hello,

Thanks for the answer, I had the same question.
But, the number of elements / core is independant of the simulation? single phase or multi phase? transient or steady ? turbulence model of zero equation or two equation or STT? it's just the convergence time will be impact by the simulation choice?

Thanks,

Véronique
Yes. It depends. It even much more significantly depends on your hardware. This is a very basic rough rule.
Veronique Pe likes this.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Foam::error::PrintStack almir OpenFOAM Running, Solving & CFD 91 December 21, 2022 05:50
DPMFoam - Serious Error --particle-laden flow in simple geometric config benz25 OpenFOAM Running, Solving & CFD 27 December 19, 2017 21:47
Specify number of cores that CFX should use. Lance CFX 16 July 20, 2016 10:04
simpleFoam parallel AndrewMortimer OpenFOAM Running, Solving & CFD 12 August 7, 2015 19:45
ICEM and CFX report different number of elements Chris Basciano CFX 0 July 20, 2007 18:24


All times are GMT -4. The time now is 05:03.