[Lance]

Hi, is there a command to specify the number of cores that CFX should use, similar to Fluent's -tX, where X is an integer?

Right now CFX uses all the eight cores on the compute node because I specify -par-dist $(hostlist -e -s, -a'*8' $SLURM_NODELIST) in my start script, which resolves to node_name*8. What I want is that CFX only uses four cores and Ansys the remaining four for my FSI simulation. But if I change -a'*8' to -a'*4' Ansys doesn't understand that it should use the remaining four cores available on the compute node. Instead both CFX and Ansys tries to share four cores, which makes the simulation run very slow.

So, is there a way to say how many number of the available cores CFX should use?

Cheers
Lance

[ghorrocks]

I don't understand your question. If you have 8 cores and tell CFX to use 4, then CFX will use 4 and there are another 4 available for ANSYS. So what is the problem?

Have you tried to use process affinity? That way you can assign specific processes to specific cores and can reduce conflicts. But 99% of the time this is not required as both Windows and Linux do a good job by default of sharing the load around.

[Lance]

Hi Glenn,
I'll try to be more clear:
when running top on the compute node, I see four CFX processes at 100% CPU but only one Ansys which runs at ~400% CPU. I was expecting to see four Ansys at 100% each, which lead me to believe that Ansys only runs on one node? Or is this behavior normal?

top - 07:57:12 up 11 days, 15:37,  3 users,  load average: 5.22, 3.19, 1.41
Tasks: 195 total,   5 running, 189 sleeping,   0 stopped,   1 zombie
Cpu(s): 50.7%us,  0.4%sy,  0.0%ni, 48.8%id,  0.0%wa,  0.1%hi,  0.1%si,  0.0%st
Mem:  16439436k total,  3295936k used, 13143500k free,   270088k buffers
Swap: 33999992k total,        0k used, 33999992k free,   684480k cac
 
 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                               
17112 user_xx   25   0 9553m 474m  25m R 336.7  3.0   2:29.71 ansys.e120                                                            
17161 user_xx   25   0  426m 355m  23m R 100.1  2.2   4:46.80 solver-hpmpi.ex                                                       
17160 user_xx   25   0  442m 371m  24m R 99.8  2.3   4:46.48 solver-hpmpi.ex                                                        
17162 user_xx   25   0  435m 364m  24m R 99.5  2.3   4:45.90 solver-hpmpi.ex                                                        
16893 user_xx   16   0 8510m  33m  20m S  0.7  0.2   0:04.61 SolverManager.e

[ghorrocks]

No, this behaviour is normal. The tech-heads may wish to correct me here, but this is my understanding of what is happening:

CFX runs as 4 separate processes. This appears on the task manager as four processes. Each process has its own memory allocation and the processes communicate with each other through MPICH or whatever multi-processor framework you are working within.

ANSYS runs as a single process, but is multi-threaded so it can run four process threads simultaneously. This appears on the task manager as one process with 400% CPU load. The process accesses one big chunk of memory, and all threads can access the full memory. The communications between threads is taken care of internally in the ANSYS executable.

[Lance]

wow, fast response from the other side of the globe.
Many thanks Glenn, what would we do without you

[Jade M]

Hi Lance,

I was wondering about this myself. Do you have a HPC license? If not, then my understanding is that CFX will only utilize, in total, 100% of one core. There are two levels of the HPC license, one where two cores can be utilized, and one where all cores can be utilized. My CFX technical support says

There are a couple of ways of speeding up simulations, but unfortunately in general nothing comes for free...: 1) Run in parallel. With CFX, you can take a single problem and partition it into multiple pieces to be run on several cores or even several networked computers. This requires those cores/computers as well as ANSYS parallel licensing keys. 2) Reduce mesh size. Of course this can compromise your solution. 3) Mesh with more efficient methods. It is possible that instead of using 3M elements to mesh a e.g. pipe, using a hex mesh or swept mesh method may result in a more efficient filling of the volume with elements yielding the same solution with less elements, faster run times, less memory usage... This is most prevalent in areas of tight gaps or small, high aspect ratio features, where the isotropic fill behaviour of tet elements can drive very high mesh counts.

[ghorrocks]

my understanding is that CFX will only utilize, in total, 100% of one core.

If running in serial, yes, it will use 100% of one core. (Providing nothing is stuffing things up like excessive page file useage or virus checking!)

There are two levels of the HPC license, one where two cores can be utilized, and one where all cores can be utilized.

This is for the ANSYS FEA package. CFX is different. Any parallel in CFX requires a parallel license.

There are a couple of ways of speeding up simulations

Yes, there are many. Here's my list. Firstly things which do not compromise simulation accuracy:
1) Run parallel
2) Get a faster CPU
3) Spend the time to optimise your simulation. This can take months of work for a complex simulation.

Here's things which gain speed at the price of accuracy:
1) Coarser mesh (do a mesh sensitivity study and work out which mesh gives you the accuracy you require - ie optimise)
2) Looser convergence (likewise do a sensitivity study to determine what is required)
3) Bigger time steps if transient (likewise do a sensitivity study to determine what is required)
4) Run single precision (but some simulations need double precision to run)
5) Simpler physics (eg degassing boundaries in multiphase, using incompressible buoyancy rather then fully compressible flow with buoyancy)
6) Don't do CFD at all and develop an analytical model or dimensional model or empirical model or similar.

[Jade M]

The statement "There are two levels of the HPC license, one where two cores can be utilized, and one where all cores can be utilized" is correct. Yes, this applies to parallel licenses.