Worse parallel efficiency with openfoam running on HPC cluster
 

Hi all,

I am now testing benchmark case (cavity) on our own HPC cluster. The total number of cells is 15,000,000. As it is newly installed, I found that it cannot perfrom good parallel effiency with increasing nodes. We use intel-2020 for compiler and IntelMPI.
For HPC cluster, each node has 64 CPUs and we totally have 5 nodes.
This is the jobfile for intel-2020 and also the speed-up plot. I dont't know whether it is due to any wrong settings for MPI. Do you have any suggestions?

Thanks!
Jin

This is normal behaviour. Parallel efficiency levels-off at some point. This is why you see a speed-up from using 1, 2 or 3 nodes. However, the reason why 5 nodes performs poorer than using 4 nodes, is the ever increasing communications-workload in contrast to the ever decreasing per-node computational workload.

When using 3 nodes (64 CPUs each), your per-node computational effort is around 78.000 cells for each parallel process.

When using 5 nodes, every parallel process only deals with roughly 46.000 cells.

So, while using more and more parallel process seems a good idea, we need to bear in mind, that the effort of communication scales super-linearly with the number of parallel processes, while the reduction of computational effort only scales linearly.

If you double the number of CPUs devoted to a parallel simulation, the per-node computational size is cut in half, while the effort for communication between all these parallel processes increases by a factor of between 2 and 4 (i.e. the squre of 2). The factor for the incease oin communication depends on your decomposition.

If you decompose badly, each parallel process needs to communicate with every other process. If you decompose well, each parallel process only needs to communicate with a few processes.