How to improve Parallel Performance of OpenFOAM on AWS?
 

Hi,

I'm trying to use the AWS as a computing resource for OpenFOAM simulations.

Recently, I tested various instance types for testing the parallel performances and would like to share the results with you. And I also want to discuss how to improve the parallel performance of OpenFOAM on AWS.

1. Simple Test of motorBike case

• Case : motorBike
• Solver: simpleFoam
• Instance Type: c7g.2xlarge
• vCPU: 8

Results:
Procs E.Time Speed Up
------ -------- ----------
1 400s 1
2 216s 1.86
4 115s 3.47
6 82s 4.89
8 70s 5.73

So far, I thought that the result was not bad.


2. Test with Larger Instance (vCPU=64)

• Case : propeller (MRF) with AMI
• Solver: simpleFoam
• Instance Type: c7g.metal
• vCPU: 64
• EFA: no

Results:
Procs Time(S) S/U
------ ------ ------
1 3902.0 1
8 451.35 8.6
16 283.74 13.8
24 231.99 16.8
32 198.19 19.7
48 170.59 22.9
64 157.32 24.8

https://www.cfd-online.com/Forums/at...1&d=1710505627

As you can see, the scalability saturated around 24 cores.

I read a following articles and tried again with EFA enabled.
https://docs.aws.amazon.com/ko_kr/AW...efa-start.html
https://aws.amazon.com/ko/blogs/hpc/...rmance-on-aws/

Procs Time S/U
1 3902 1
32 198.74 19.6
64 160.95 24.2

But the results were the same.


3. AMD Instance

• Case : propeller (MRF) with AMI
• Solver: simpleFoam
• Instance Type: c6a.24xlarge
• vCPU: 96
• EFA: yes
• SMT: on

Results:
Procs Time S/U
------ ----- ------
1 3900.0 1
8 401.25 9.72
16 259.84 15.01
24 213.92 18.23
32 199.92 19.51
48 154.23 25.29
64 148.67 26.23
96 142.81 27.31

https://www.cfd-online.com/Forums/at...1&d=1710505983

Not really. Of course, I forgot to turn off the SMT option, but still the result is not satisfactory at all.

Does anyone who has an experience on the OpenFOAM-ing on Cloud?

As a reference, I used OpenMPI with EFA for the tests.

I hope to find the way to improve the parallel performance of the OpenFOAM on AWS.

Thanks!