CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Programming & Development

Has any one tried AmgX on OpenFOAM

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree4Likes

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   January 23, 2023, 03:44
Default
  #21
Senior Member
 
Klaus
Join Date: Mar 2009
Posts: 256
Rep Power: 22
klausb will become famous soon enough
I found out, that the AmgX-OpenFOAM integration has been done in another way using a new interface called FOAM2CSR instead of the wrapper.

So it's:

OpenFOAM > PETSc4FOAM > FOAM2CSR > AmgX

See youtube: SU2 Conference 21: An Implementation of AmgX for GPU Acceleration of OpenFOAM

https://www.youtube.com/watch?v=1d1DVURNz4E


FOAM2CSR

https://gitlab.hpc.cineca.it/openfoam/foam2csr
klausb is offline   Reply With Quote

Old   January 24, 2023, 02:33
Default
  #22
New Member
 
Join Date: Dec 2022
Posts: 23
Rep Power: 3
Nitish25 is on a distinguished road
Quote:
Originally Posted by dlahaye View Post
Let us talk very briefly now and find a slot to talk more extensively next week.

Please send me link for Google meet either here or on private cfdonline address.

Dhanyavaad.
Respected Sir,
As per the last conversation you told that to schedule meeting at 24 jan for detailed information
PF Link
https://meet.google.com/ujq-qheo-sdz

Thank you for your valuable time and efforts

Thanks & regards
dlahaye likes this.

Last edited by Nitish25; January 24, 2023 at 04:07.
Nitish25 is offline   Reply With Quote

Old   January 24, 2023, 03:17
Default
  #23
New Member
 
Join Date: Dec 2022
Posts: 23
Rep Power: 3
Nitish25 is on a distinguished road
Quote:
Originally Posted by dlahaye View Post
Let us talk very briefly now and find a slot to talk more extensively next week.

Please send me link for Google meet either here or on private cfdonline address.

Dhanyavaad.
Respected Sir,
As per the last conversation you told that to schedule meeting at 24 jan for detailed information
PF Link
https://meet.google.com/ujq-qheo-sdz

Thank you for your valuable time and efforts

Thanks & regards
Nitish25 is offline   Reply With Quote

Old   January 24, 2023, 04:13
Default
  #24
New Member
 
Join Date: Dec 2022
Posts: 23
Rep Power: 3
Nitish25 is on a distinguished road
Quote:
Originally Posted by Nitish25 View Post
Respected Sir,
As per the last conversation you told that to schedule meeting at 24 jan for detailed information
PF Link
https://meet.google.com/ujq-qheo-sdz

Thank you for your valuable time and efforts

Thanks & regards
sir if u are free, plz join the meeting now, I'm eagerly waiting for u.
Nitish25 is offline   Reply With Quote

Old   January 24, 2023, 04:32
Default
  #25
Senior Member
 
Domenico Lahaye
Join Date: Dec 2013
Posts: 736
Blog Entries: 1
Rep Power: 17
dlahaye is on a distinguished road
We scheduled our meeting for 16:00 this afternoon, i.e., 5.5 hours from now.

What is progress on compiling and linking AMGX tutorial, independently from OpenFoam?
dlahaye is offline   Reply With Quote

Old   January 24, 2023, 04:51
Default
  #26
Senior Member
 
Domenico Lahaye
Join Date: Dec 2013
Posts: 736
Blog Entries: 1
Rep Power: 17
dlahaye is on a distinguished road
Quote:
Originally Posted by klausb View Post
I found out, that the AmgX-OpenFOAM integration has been done in another way using a new interface called FOAM2CSR instead of the wrapper.

So it's:

OpenFOAM > PETSc4FOAM > FOAM2CSR > AmgX

See youtube: SU2 Conference 21: An Implementation of AmgX for GPU Acceleration of OpenFOAM

https://www.youtube.com/watch?v=1d1DVURNz4E


FOAM2CSR

https://gitlab.hpc.cineca.it/openfoam/foam2csr
Dear Klaus,

1/ I imagine that FOAM2CSR applies to the OpenFoam LDU matrix format. FOAM2CSR does *not* apply to the PETSc matrix format. Can you confirm?

2/ Does AMGX apply to an decomposed case, or does AMGX enforce its own parallel decomposition?

3/ When using AMGX to an incompressible case (like motorBike), AMGX is to be applied to the pressure field only?

Thanks. Domenico.
dlahaye is offline   Reply With Quote

Old   January 24, 2023, 05:07
Default
  #27
New Member
 
Join Date: Dec 2022
Posts: 23
Rep Power: 3
Nitish25 is on a distinguished road
Quote:
Originally Posted by dlahaye View Post
We scheduled our meeting for 16:00 this afternoon, i.e., 5.5 hours from now.

What is progress on compiling and linking AMGX tutorial, independently from OpenFoam?
we have successfully build the amgx by following the steps as mention in their website https://github.com/NVIDIA/AMGX i.e. as shown below

mkdir build
cd build
cmake ../
make -j16 all

while, build it automatically compiles all the amgx example and generates their executable in the build directory.

we are able to run the amgx_capi (it is an executable file of AMGX/examples/amgx_capi.c )example on gpu inside the build directory by following command
examples/amgx_capi -m ../examples/matrix.mtx -c ../src/configs/FGMRES_AGGREGATION.json

so manually compilation of amgx/examples is not required.

Thank you for ur valuable time.
Nitish25 is offline   Reply With Quote

Old   January 24, 2023, 08:16
Default
  #28
Senior Member
 
Domenico Lahaye
Join Date: Dec 2013
Posts: 736
Blog Entries: 1
Rep Power: 17
dlahaye is on a distinguished road
Yes and no.

Yes, I understand that the manually building the examples of AMGX/examples is *not* required.

No, however, I am not interested in building the examples of AMGX/examples as a final goal. Building the examples of AMGX/examples is intended as a seperate step to *understand* the compile and build process of AMGX/examples.

There are now two ways to proceed

1/ First alternative

Repeat the entire build process, and replace in the last step "make -j16 all" by "make -j16 all >& log.makeAMGX" in such a way that the output of the make process is written to the log-file log.makeAMGX that we can inspect. A search on amgx_capi.c should provide the information we are after.

2/ Second alternative

Assume that all the header files are included in AMGX/include and its subdirectory and that all the library files (files with extension *.a or *.so) are
in some lib directory.

We can proceed with a copy-and-paste of the source of amgx_capi.c into myLaplacianFoam and try to compile, link and run it.

Let me know how it goes.
dlahaye is offline   Reply With Quote

Old   January 25, 2023, 03:34
Default
  #29
New Member
 
Join Date: Dec 2022
Posts: 23
Rep Power: 3
Nitish25 is on a distinguished road
Quote:
Originally Posted by dlahaye View Post
Yes and no.

Yes, I understand that the manually building the examples of AMGX/examples is *not* required.

No, however, I am not interested in building the examples of AMGX/examples as a final goal. Building the examples of AMGX/examples is intended as a seperate step to *understand* the compile and build process of AMGX/examples.

There are now two ways to proceed

1/ First alternative

Repeat the entire build process, and replace in the last step "make -j16 all" by "make -j16 all >& log.makeAMGX" in such a way that the output of the make process is written to the log-file log.makeAMGX that we can inspect. A search on amgx_capi.c should provide the information we are after.

2/ Second alternative

Assume that all the header files are included in AMGX/include and its subdirectory and that all the library files (files with extension *.a or *.so) are
in some lib directory.

We can proceed with a copy-and-paste of the source of amgx_capi.c into myLaplacianFoam and try to compile, link and run it.

Let me know how it goes.
hello sir
I have repeated the entire built process of AMGX and save in log file .
I hv attached the log file.
I am able to find build of amgx_capi.c at
[ 95%] Building C object examples/CMakeFiles/amgx_capi.dir/amgx_capi.c.o
in log file.

kindly reply next step.
Attached Files
File Type: txt log of MAKE AMGX.txt (21.3 KB, 5 views)
Nitish25 is offline   Reply With Quote

Old   January 25, 2023, 04:00
Default
  #30
Senior Member
 
Domenico Lahaye
Join Date: Dec 2013
Posts: 736
Blog Entries: 1
Rep Power: 17
dlahaye is on a distinguished road
We are making a small step in the good direction.

The log file, however, fails to give sufficient information. We need more information.

1/ Compile stage

Here is what we now know

Line 242 in log file
[ 95%] Building C object examples/CMakeFiles/amgx_capi.dir/amgx_capi.c.o

To obtain more information, the compiler (i.e. gcc -c) needs to output more information.

2/ Link stage

Here is what we now know

Line 247 in log file:
[ 97%] Linking C executable amgx_capi

To obtain more information, the linker (i.e. gcc -L ) needs to output more information.

3/ Next Step

To obtain the required additional information, can you please type

cd examples/CMakeFiles/amgx_capi.dir

and subsequently type

make

?

Dhanyavaad.
dlahaye is offline   Reply With Quote

Old   January 25, 2023, 04:16
Default
  #31
New Member
 
Join Date: Dec 2022
Posts: 23
Rep Power: 3
Nitish25 is on a distinguished road
Quote:
Originally Posted by dlahaye View Post
We are making a small step in the good direction.

The log file, however, fails to give sufficient information. We need more information.

1/ Compile stage

Here is what we now know

Line 242 in log file
[ 95%] Building C object examples/CMakeFiles/amgx_capi.dir/amgx_capi.c.o

To obtain more information, the compiler (i.e. gcc -c) needs to output more information.

2/ Link stage

Here is what we now know

Line 247 in log file:
[ 97%] Linking C executable amgx_capi

To obtain more information, the linker (i.e. gcc -L ) needs to output more information.

3/ Next Step

To obtain the required additional information, can you please type

cd examples/CMakeFiles/amgx_capi.dir

and subsequently type

make

?

Dhanyavaad.
sir
we have entered the examples/CMakeFiles/amgx_capi.dir
and have typed make
but it says
make: *** No targets specified and no makefile found. Stop.

kindly reply
Attached Images
File Type: jpg after typing make.jpg (48.1 KB, 15 views)
Nitish25 is offline   Reply With Quote

Old   January 25, 2023, 04:42
Default
  #32
Senior Member
 
Domenico Lahaye
Join Date: Dec 2013
Posts: 736
Blog Entries: 1
Rep Power: 17
dlahaye is on a distinguished road
I am confused.

The file "makefile" (or variant) seems to be missing.

What is the output of

cd <AMGX-main-directory>

find . -name *makefile* -print

?
dlahaye is offline   Reply With Quote

Old   January 25, 2023, 04:51
Default
  #33
New Member
 
Join Date: Dec 2022
Posts: 23
Rep Power: 3
Nitish25 is on a distinguished road
Quote:
Originally Posted by dlahaye View Post
I am confused.

The file "makefile" (or variant) seems to be missing.

What is the output of

cd <AMGX-main-directory>

find . -name *makefile* -print

?
after type
find . -name *makefile* -print

in main amgx directory,
it print:
./examples/install_makefiles_mpi
./examples/install_makefiles_nompi


when I type
find . -name *Makefile* -print
it prints:
./build/CMakeFiles/Makefile.cmake
./build/CMakeFiles/Makefile2
./build/examples/Makefile
./build/thrust/Makefile
./build/Makefile
./build/src/Makefile
./examples/Makefile.cray
./examples/amgx_spmv_example/Makefile
./examples/install_makefiles_mpi/Makefile
./examples/install_makefiles_nompi/Makefile
./thrust/Makefile
Attached Images
File Type: jpg Screenshot 2023-01-25 152003.jpg (18.0 KB, 4 views)
Nitish25 is offline   Reply With Quote

Old   January 25, 2023, 06:04
Default
  #34
New Member
 
Join Date: Dec 2022
Posts: 23
Rep Power: 3
Nitish25 is on a distinguished road
Quote:
Originally Posted by Nitish25 View Post
after type
find . -name *makefile* -print

in main amgx directory,
it print:
./examples/install_makefiles_mpi
./examples/install_makefiles_nompi


when I type
find . -name *Makefile* -print
it prints:
./build/CMakeFiles/Makefile.cmake
./build/CMakeFiles/Makefile2
./build/examples/Makefile
./build/thrust/Makefile
./build/Makefile
./build/src/Makefile
./examples/Makefile.cray
./examples/amgx_spmv_example/Makefile
./examples/install_makefiles_mpi/Makefile
./examples/install_makefiles_nompi/Makefile
./thrust/Makefile
plz find the attached image
Attached Images
File Type: jpg Makefile.jpg (31.5 KB, 6 views)
Nitish25 is offline   Reply With Quote

Old   January 25, 2023, 07:38
Default
  #35
Senior Member
 
Domenico Lahaye
Join Date: Dec 2013
Posts: 736
Blog Entries: 1
Rep Power: 17
dlahaye is on a distinguished road
1/ What is the output of the following:

cd examples

ls

make -f Makefile.cray amgx_capi

2/ I send you a link to join a slack group

Talk soon. D.
dlahaye is offline   Reply With Quote

Old   February 2, 2023, 16:40
Default
  #36
Senior Member
 
Klaus
Join Date: Mar 2009
Posts: 256
Rep Power: 22
klausb will become famous soon enough
Dear Domenico,

regarding your questions:

1/ I imagine that FOAM2CSR applies to the OpenFoam LDU matrix format. FOAM2CSR does *not* apply to the PETSc matrix format. Can you confirm?

See timestamp 2:46 in the youtube video. It explains the role of FOAM2CSR, it handles "the GPU co-ordination, matrix consolidation, data transfers, transformations and dispatching linear systems to AmgX." I haven't looked into the source code of FOAM2CSR to see what's its relationship with PETSc4FOAM. In the video, they talk about converting the LDU format to the CSR format used by AmgX, I assume because it's computationally cheaper to do the conversion.


2/ Does AMGX apply to an decomposed case, or does AMGX enforce its own parallel decomposition?

That's a misunderstanding I'd say. AmgX is the solver library, FOAM2CSR handles the "consolidation" of decomposed cases according to the number of available GPUs if the number of CPU cores and available GPUs differ.


3/ When using AMGX to an incompressible case (like motorBike), AMGX is to be applied to the pressure field only?

No, it's not limited to the pressure field.


Klaus
klausb is offline   Reply With Quote

Old   February 9, 2023, 02:45
Default
  #37
Dcn
New Member
 
Join Date: Aug 2022
Posts: 16
Rep Power: 3
Dcn is on a distinguished road
Quote:
Originally Posted by klausb View Post
I found out, that the AmgX-OpenFOAM integration has been done in another way using a new interface called FOAM2CSR instead of the wrapper.

So it's:

OpenFOAM > PETSc4FOAM > FOAM2CSR > AmgX

See youtube: SU2 Conference 21: An Implementation of AmgX for GPU Acceleration of OpenFOAM

https://www.youtube.com/watch?v=1d1DVURNz4E


FOAM2CSR

https://gitlab.hpc.cineca.it/openfoam/foam2csr
Sir i have compile and build the foam2CSR and have got libfoam2csr.so and have also build PETSc4FOAM and have got libpetscfoam.so , similarly i have successfully build the Amgx library , now can you kindly tell me how to use FOAM2CSR with Amgx
Thank you
Dcn is offline   Reply With Quote

Old   February 10, 2023, 15:12
Default
  #38
Senior Member
 
Domenico Lahaye
Join Date: Dec 2013
Posts: 736
Blog Entries: 1
Rep Power: 17
dlahaye is on a distinguished road
Dear Klaus,

Thank you for your reply. Much appreciated!

Re-1: Thank you for pointing out the link to the video.

Re-2: I fail to understand what the purpose of FOAM2CSR is. Once in PETSc, the matrix is in a format that PETSc understands. Clearly, I am missing something.

Re-3: I agreed that GAMG can solve velocity components. I wonder, however, whether the use of GAMG for the velocity components offers any gains.

Thanks again. Talk soon, Domenico.
dlahaye is offline   Reply With Quote

Old   February 10, 2023, 15:26
Default
  #39
Senior Member
 
Domenico Lahaye
Join Date: Dec 2013
Posts: 736
Blog Entries: 1
Rep Power: 17
dlahaye is on a distinguished road
Dear Dcn,

I fail to understand what the purpose of FOAM2CSR is. Clearly and most obviously, I am missing the point.

I therefore developed myLaplacianFoam at placed it at https://github.com/ziolai/software/t...yLaplacianFoam. This code convert the LDU matrix format to CSR format using code available from https://gist.github.com/BinWang0213/...b8e25ac014cc48.

In a next step I wish to extend myLaplacianFoam with something like

Quote:
// Call Initialize
AMGX_initialize();

AMGX_resources_create(&rsrc, cfg, &amgx_mpi_comm, 1, &lrank);
AMGX_matrix_create(&A, rsrc, mode);
AMGX_vector_create(&x, rsrc, mode);
AMGX_vector_create(&b, rsrc, mode);
AMGX_solver_create(&solver, rsrc, mode, cfg);

// Call AMGX set-up stage
AMGX_solver_setup(solver, A);

// Call AMGX solve stage
AMGX_solver_solve(solver, b, x);

// Call Finalize
AMGX_finalize();
Does this make sense?
dlahaye is offline   Reply With Quote

Old   May 16, 2024, 06:06
Default
  #40
New Member
 
Join Date: Apr 2023
Posts: 2
Rep Power: 0
Walid Badawy is on a distinguished road
Hi Nitish, did you managed to do it with Klaus workflow!
Walid Badawy is offline   Reply With Quote

Reply

Tags
amgx, gpu, library, nvidia, openfoam 2.2.x

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Frequently Asked Questions about Installing OpenFOAM wyldckat OpenFOAM Installation 3 November 14, 2023 11:58
OpenFOAM Foundation releases OpenFOAMŪ 3.0.0 CFDFoundation OpenFOAM Announcements from OpenFOAM Foundation 1 November 7, 2015 15:16
OpenFOAM Foundation Releases OpenFOAM v2.3.0 opencfd OpenFOAM Announcements from OpenFOAM Foundation 3 December 23, 2014 03:43
Suggestion for a new sub-forum at OpenFOAM's Forum wyldckat Site Help, Feedback & Discussions 20 October 28, 2014 09:04
64bitrhel5 OF installation instructions mirko OpenFOAM Installation 2 August 12, 2008 18:07


All times are GMT -4. The time now is 23:19.