OpenFOAM on AMD GPUs. Container from Infinity Hub: user experiences and performance
AMD recently provided an OpenFOAM container capable of running on AMD GPUs.
It is in their Infinity Hub: https://www.amd.com/en/technologies/...y-hub/openfoam And my questions are: -How have been the experiences of the community using this OpenFOAM container on AMD GPUs? -Are you reaching cool performance improvements vs just CPU solvers? Thanks a lot, Alexis (PS. I will start using it and post my experiences too) |
Hi,
Were you able to launch any simulation using the GPU version? Is it 100% GPU or only the pressure solver is solved in the GPU? Do you know if it could be compatible que Nvidia GPU to test it? Best Regards Marcelino |
OpenFOAM on AMD GPUs. Container from Infinity Hub: Experiences with Radeon VII
Thought I'd share my experiences with this! :D
My findings, unfortunately with my setup, have been that it remains much faster to solve on CPU than GPU. :( I used the HPC_Motorbike example and code provided by AMD in the docker container (not available on the link any longer btw) as-is on my Radeon VII. For the CPU examples, I modified the run to suit a typical CPU-based set of solvers using the standard tutorial fvSolution files. Results as follows. Times shown are SimpleFoam total ClockTime to 20 iterations; and time per iteration, excluding the first time step:
I get that GPUs are made for large models but I am already nearly reaching the 16GB of vram even in this model (5,223,573 cells). I can't run the Medium sized model (~9M cells I think) because I run out of vram :( I'm running this on my desktop PC for funzies because I don't even want to know how much faster this will be on my usual solving machine (48 core xeon). So, in summary, based on my experiences with a Radeon VII and the Small HPC_motorbike case:
Cheers, Tom |
Quote:
The initial run script appears to be flexible to support CUDA devices too. I've not dug any deeper and don't have a suitable GPU to test with further, sorry. Code:
Available Options: HIP or CUDA |
Thanks for your input. Much appreciated.
1/ Can you confirm that the bulk of the CPU time goes into the pressure-solve (independent of CPU vs. GPU)? 2/ How do you precondition PETSc-CG for the pressure solve? 3/ Are you willing to walk an extra mile and compare two flavours of PETSc-CG. Flavour-1: using AMG to precondition PETSc-CG allowing AMG to do a set-up at each linear system solve. Flavour-2: using AMG to precondition PETSc-CG (so far identical to Flavour-1), this time freezing the hierarchy that AMG construct. |
Quote:
1) I don't have a specific clocktime breakdown, but it would appear so, yes. 2) PETSC-CG is preconditioned using GAMG: Code:
p |
Thanks again.
It appears that by setting Code:
periodicCoeffs To obtain statistics on OpenFoam native GAMG coarsening, insert in system/controlDict I have two follow-up questions if you allow. 1/ How does runtime of PETSc-GAMG compare with OpenFoam-native-GAMG (the latter used as a preconditioner to be fair)? 2/ Do you see statistics of PETSc-GAMG coarsening printed somewhere? It would be interesting to compare these statistics (in particular the geometric and algebraic complexities) with the statistics of OpenFoam-native-GAMG. The latter can be easily obtained by inserting debug switches in system/controlDict; Quote:
|
All times are GMT -4. The time now is 02:21. |