Adding an external library which uses openmp to a solver

dolfus · November 26, 2020, 13:40

Hello all,

I've added a dynamic external library '.so' to the interFoam solver. The library is parallelized using openmp and compiled using the flag -fopenmp in gcc compiler.

When I use this library on a C program written out of OpenFOAM, things go right. However, using it within OpenFOAM, the cpu-time increases by a factor of 2 running the same simulation (compared to the C program). It looks like OpenFOAM is not taking on account this parallelization and only uses a single core (checked with top command).

Regarding the OpenFOAM solver, I've compiled it adding to the make file the flag -fopenmp, and also writing the #include omp.h statement at interFoam.C file. No matter what value of OMP_NUM_THREADS I use for the OpenFOAM simualtion, the cpu-time is almost the same.

I don't know if anyone has experienced a similar problem.

Thank you

olesen · November 27, 2020, 11:26

You might need to have COMP_OPENMP and LINK_OPENMP in your wrapping code in order to have proper linkage. Not really sure.

https://develop.openfoam.com/Develop...ral/Gcc/openmp

EDIT: I mean to have them actually used in the corresponding Make/options file.

dolfus · November 27, 2020, 12:41

Thank you for your reply.

I've added those flags to the compiler, but it is still not working. In fact, if I run the simulation with OMP_NUM_THREADS=1, CPU-time is a 10% lesser that in the case with 2 threads, which makes no sense.
I'm not sure what's going on.

olesen · November 27, 2020, 14:09

Probably need to test something really basic, like threads with an "printf" to see if anything is actually running with threads when linked with openfoam.

dolfus · November 30, 2020, 06:48

I've checked it out with a printf on my library and, surprisingly, it actually runs using threads when linked with openFOAM.
So it should work fine but it's not. Probably I'm still missing something, or maybe this is the best way openFOAM can handle openmp on an external library.
I also noticed that when running with 2 threads execution time is almost twice the clock time, but when running with 1 thread they are quite similar.

dolfus · December 1, 2020, 12:55

After several tests I think your solution fixed my problem of communication with openMP. Thank you very much.

Despite that, I would like to add something I realized, in case someone find it usefull. Turns out that runTime.elapsedCpuTime() sums up the times consumed by all threads. That's why I was obtaining such different times between my code and OF. Using omp_get-wtime() instead of OF's solved this issue. However, using OF with MPI on a single PC with 2 processors there is no such difference between CPU-time and clock-time.

November 26, 2020, 13:40	Adding an external library which uses openmp to a solver	#1
dolfus New Member Join Date: Feb 2019 Posts: 23 Rep Power: 7	Hello all, I've added a dynamic external library '.so' to the interFoam solver. The library is parallelized using openmp and compiled using the flag -fopenmp in gcc compiler. When I use this library on a C program written out of OpenFOAM, things go right. However, using it within OpenFOAM, the cpu-time increases by a factor of 2 running the same simulation (compared to the C program). It looks like OpenFOAM is not taking on account this parallelization and only uses a single core (checked with top command). Regarding the OpenFOAM solver, I've compiled it adding to the make file the flag -fopenmp, and also writing the #include omp.h statement at interFoam.C file. No matter what value of OMP_NUM_THREADS I use for the OpenFOAM simualtion, the cpu-time is almost the same. I don't know if anyone has experienced a similar problem. Thank you Last edited by dolfus; November 26, 2020 at 13:55. Reason: typo

November 30, 2020, 06:48		#5
dolfus New Member Join Date: Feb 2019 Posts: 23 Rep Power: 7	I've checked it out with a printf on my library and, surprisingly, it actually runs using threads when linked with openFOAM. So it should work fine but it's not. Probably I'm still missing something, or maybe this is the best way openFOAM can handle openmp on an external library. I also noticed that when running with 2 threads execution time is almost twice the clock time, but when running with 1 thread they are quite similar. Last edited by dolfus; November 30, 2020 at 10:11.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Duplicate library entries when running a solver with custom library	francescomarra	OpenFOAM Programming & Development	3	May 17, 2022 08:37
Coupling to external code using MPI calls from within a solver	Regis_	OpenFOAM Programming & Development	0	October 31, 2017 10:50
Running external application in solver	dariodario32	OpenFOAM Programming & Development	2	July 10, 2017 08:52
External aerodynamics of an aircraft: pressure based solver or density based solver?	shereez234	FLUENT	2	April 30, 2015 15:30
adding Porous media to lagrangian Solver	kalyan	OpenFOAM Programming & Development	0	September 19, 2014 07:41

November 27, 2020, 11:26		#2
olesen Senior Member Mark Olesen Join Date: Mar 2009 Location: https://olesenm.github.io/ Posts: 1,685 Rep Power: 40	You might need to have COMP_OPENMP and LINK_OPENMP in your wrapping code in order to have proper linkage. Not really sure. https://develop.openfoam.com/Develop...ral/Gcc/openmp EDIT: I mean to have them actually used in the corresponding Make/options file.

November 27, 2020, 12:41		#3
dolfus New Member Join Date: Feb 2019 Posts: 23 Rep Power: 7	Thank you for your reply. I've added those flags to the compiler, but it is still not working. In fact, if I run the simulation with OMP_NUM_THREADS=1, CPU-time is a 10% lesser that in the case with 2 threads, which makes no sense. I'm not sure what's going on.

November 27, 2020, 14:09		#4
olesen Senior Member Mark Olesen Join Date: Mar 2009 Location: https://olesenm.github.io/ Posts: 1,685 Rep Power: 40	Probably need to test something really basic, like threads with an "printf" to see if anything is actually running with threads when linked with openfoam.

December 1, 2020, 12:55		#6
dolfus New Member Join Date: Feb 2019 Posts: 23 Rep Power: 7	After several tests I think your solution fixed my problem of communication with openMP. Thank you very much. Despite that, I would like to add something I realized, in case someone find it usefull. Turns out that runTime.elapsedCpuTime() sums up the times consumed by all threads. That's why I was obtaining such different times between my code and OF. Using omp_get-wtime() instead of OF's solved this issue. However, using OF with MPI on a single PC with 2 processors there is no such difference between CPU-time and clock-time.