[Maffei]

Hi,

I am trying to run the CFX solver using my university cluster, but when I execute the following command:

cfx5solve -numa auto -def Fluid\ Flow\ CFX.def -start-method "Intel MPI Distributed Parallel" -par-dist n02*40,n03*40 -batch -monitor log

I observed that both nodes have 40 processes (through htop command), but only node n03 uses 100% of the 40 CPUs. The processes in node n02 use only 3% of CPUs.

Did you know how to make the CFX solver uses all the nodes of both nodes?

[ghorrocks]

The most likely cause for this is that you are trying to feed the multipartition data for 80 partitions through one interconnect. Unless you have a very high-end network between these two machines the interconnect will be flooded and will bottleneck the simulation.

It would be better to have 8 machines with 10 partitions each rather than 2 machines with 40 partitions each as the network load will get spread over more interconnects.

[ghorrocks]

Oh - and the problem might not necessarily be the network connection. It could be the connection of the network adapter to the CPU, so that means the FSB, memory interconnect and other motherboard stuff. So the motherboard quality is critical as well.

[Maffei]

Thanks for the answer,

Ok, I will check these points and the possibility to have more nodes with fewer CPUs in each one. Is there another possibility? Because I made one test where I stated the same case, but I don't allocate the node which the thinks works well on the first try and I observe one node with 40 CPUs working at 3% of its capacity. I was wondering if is something related to the host machine configuration (either Ansys or cluster).

[ghorrocks]

It is a really good idea to test the capabilities of your cluster before you start using it. I recommend getting a benchmark simulation, and then running it 1,2,4,8,16,32 etc partitions on one machine and check the scaling, and then do the same across 2 machines, then 4, 8 etc.

This will tell you how many partitions you can put on a single node (as it is likely performance will drop off before you use all cores), and how it scales across multiple nodes. It is very instructive - and it will also tell you the optimum configuration you should define to get best performance from your cluster. As you are seeing, optimum performance is almost certainly not using the maximum number of partitions.

[Maffei]

Ok, thanks for the explanation. I will try it. One more thing, can the cluster performance be software dependent? This is because I have a lab friend who use to run OpenFOAM using 4 nodes with all CPUs and he doesn't report problems like this.

[ghorrocks]

Yes, cluster performance is software dependant. Different software has different loads on main memory, L1 and L2 caches, hard drive, inter-partition communications and so on. Also the optimisation options when the software is compiled makes a big difference.

But most respectable Navier Stokes based CFD codes should be very similar. You should only note major differences if going to extreme numbers of partitions. But if you compare a CFD software with a ray tracing software for instance - I would expect them to scale very differently on a large cluster.