GPU Linear Solvers for OpenFOAM
ofgpu is a free GPL library from Symscape that provides GPU linear solvers for OpenFOAM®. The experimental library targets NVIDIA CUDA devices on Windows, Linux, and (untested) Mac OS X. It uses the Cusp library's Krylov solvers to produce equivalent GPU (CUDA-based) versions of the standard OpenFOAM linear solvers:
ofgpu also has support for the OpenFOAM preconditioners:
For more details see "GPU Linear Solver Library for OpenFOAM". OpenFOAM is a registered trademark of OpenCFD and is unaffiliated with Symscape. |
Hi Richard,
Thanks for the nice work. I was able to compile OpenFOAM-1.7.x and use your plugin on the GPU. Your testcase also worked nicely. Some comments that may help other people that are installing this library. - I set gcc 4.3 as a compiler in ofgpu for C, and g++ (this is standard) for C++. - Other people in my department are already using CUDA, but not Cusp and Thrust, I downloaded Cusp and Thrust code-packages to /ofgpu/src - To patch, I used patch -p0 < file.patch To compile OpenFOAM 1.7.x with the gpu library, I did the following - chmod 770 src/gpu/Allwmake - touch src/gpu/gpuless/Make/options (this file was missing) - I think there is a mistake in src/gpu/cuda/Make/options, I used LIB = $(FOAM_LIBBIN)/cuda/libgpu there - In OF/lib I made a symbolic link named libofgpu.so to the library in the ofgpu directory You can check the activity of your graphical card with nvidia-smi -a -l For the standard GPU in my machine with nice CPUs, this test case (pitzDaily) there was only slow down of course. Richard, thanks for the nice work, I read on the website you're working on implementing additional preconditioners. I'm very of course very interested in those. Keep up the good work! Best regards, Bernhard |
Is It necessary to compile all openFoam, or just a library??
Pablo |
In principle you only have to recompile those parts of the code that are affected by the patch I suppose. Since I have been working with 1.7.1, and this library was designed for 1.7.x, I decided to take the save route and download and compile 1.7.x separate from my other installation. It would be worth trying it on your existing compiled OpenFOAM version.
|
Thanks for the feedback
Thanks for the feedback Bernard.
You raised some good points: Quote:
Quote:
I've edited the build description to make this clearer at: http://www.symscape.com/gpu-openfoam |
Casting problem
Hi,
I've successfully compiled ofgpu.During compilation of OpenFOAM there is casting problem between ofgpu::index and Foam::label. I can solve the problem by manually cast-forcing. But, is there any quick solution? Thanks. The related error is shown here: PBiCGgpu.C:103: error: no matching function for call to ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, Foam::label, Foam::label, ofgpu::index*, ofgpu::index*, ofgpu::index*, ofgpu::real*, ofgpu::real*, ofgpu::index*, ofgpu::index*, ofgpu::real*, ofgpu::real*, ofgpu::real*, int, ofgpu::real, ofgpu::real, ofgpu::real, ofgpu::real, Foam::label, bool&) /home/tesla/OpenFOAM/ofgpu/src/ofgpu/sparsematrixargs.h:46: note: candidates are: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, ofgpu::index, ofgpu::index, const ofgpu::index*, const ofgpu::index*, const ofgpu::index*, const ofgpu::real*, const ofgpu::real*, const ofgpu::index*, const ofgpu::index*, const ofgpu::real*, ofgpu::real*, const ofgpu::real*, ofgpu::index, ofgpu::real, ofgpu::real, ofgpu::real&, ofgpu::real&, ofgpu::index&, bool&) /home/tesla/OpenFOAM/ofgpu/src/ofgpu/sparsematrixargs.h:24: note: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const ofgpu::SparseMatrixArgs&) make: *** [Make/linux64GccDPOpt/PBiCGgpu.o] Error 1 |
Foam::label as long, why?
So I guess on your system (Linux?) you have Foam::label defined as a long?
That would result from defining 'FOAM_LABEL_64' for label.H Is this the case? Do you really need access to integers larger than 2,000,000,000? Do you want to solve for more than 2 Billion cells/faces/nodes? So my first suggestion is that if you don't need to define FOAM_LABEL_64 then don't (you save memory too) and recompile OpenFOAM. Second option if you do want to keep FOAM_LABEL_64 then try changing the definition of ofgpu::index to long in index.h. This will cost you scarce GPU memory if it works. |
The patch file issues
Dear gocarts,
Thanks for the quick reply. You are right, FOAM_LABEL_64 is defined automatically because the operating system is "Ubuntu-Desktop-10.10 64-bit". I've solved the problem with casting. The compilation proceeds and fails upon Allmake in "/home/tesla/OpenFOAM/OpenFOAM-1.7.1/applications/solvers/multiphase/compressibleInterFoam" directory. Main errors are: compressibleInterFoam.C:43:22: error: RASModel.H: No such file or directory compressibleInterFoam.C:44:22: error: LESModel.H: No such file or directory followed by: compressibleInterFoam.C:51: error: "incompressible" has not been declared compressibleInterFoam.C:52: error: "incompressible" has not been declared This is caused by the patch file which makes some changes and adds "#include RASModel.H" and "#include LESModel.H" lines to compressibleInterFoam.C. However, these files are not exists in compressibleInterFoam directory. These files are in the following path: ./src/turbulenceModels/incompressible/RAS/RASModel/RASModel. ./src/turbulenceModels/compressible/RAS/RASModel/RASModel.H Should I make a soft link? or I have missed something in earlier stages Thanks. System: Ubuntu-10.10, OpenFOAM-1.7.1, CUDA 3.1 and GCC 4.4 |
What about ofgpu 0.2?
Wouldn't you prefer to build the latest version of ofgpu 0.2 with OpenFOAM 2.0.x? http://www.symscape.com/gpu-0-2-openfoam
By the looks of your errors it seems that your build may have failed to create the turbulence libraries. The missing includes should be in: ./src/turbulenceModels/incompressible/lnInclude ./src/turbulenceModels/compressible/lnInclude |
Ofgpu_0.2
Happy to see OFGPU 0.2 is compatible with CUDA 4.0.
I have downgraded thrust, cusp and cuda to make the required compatibility. I will test OFGPU 0.2 soon, Thank you |
Don't define FOAM_LABEL_64
Just to follow up on:
Quote:
./wmake/rules/<platform>/c++ Edit: Should be FOAM_LABEL64 not FOAM_LABEL_64 |
label.H
Actually It is not defined! Following commands show that FOAM_LABEL_64 does not exist but FOAM_LABEL64 is used in label.H and uLabel.H
root@gpu30:~/OpenFOAM# grep -r ./ -e FOAM_LABEL_64 root@gpu30:~/OpenFOAM# grep -r ./ -e FOAM_LABEL64 ./OpenFOAM-1.7.1/src/OpenFOAM/lnInclude/label.H:#if FOAM_LABEL64 ./OpenFOAM-1.7.1/src/OpenFOAM/lnInclude/uLabel.H:#if FOAM_LABEL64 ./OpenFOAM-1.7.1/src/OpenFOAM/primitives/ints/label/label.H:#if FOAM_LABEL64 ./OpenFOAM-1.7.1/src/OpenFOAM/primitives/ints/uLabel/uLabel.H:#if FOAM_LABEL64 So the label type is int not long. |
Double precision vs single precision
I think then your problems are two fold:
|
How to run the sample?
OK. I think the OpenFOAM is now compiled successfully.
I'm new in OpenFOAM. How can run "pitzDailyGpu" sample? I've extracted the archive, now there exist 3 folders: 0 constant system Thanks. |
OF compilation problems
Hej,
I still have a problem with the compiling I use GCC 4.4, thrust 1.4, cusp 0.2, OF 2.0.x (Oct 21st) and applied the patch. Everything else before of-gpu is compiled without errors. After applying the patch, I can't compile OF 2.0.x anymore. I get problems. There must be a problem with the include or something because it is missing basically all the files for include like the following Code:
Making dependency list for source file pimpleFoam.C I tried compiling it without the patch and that works flawlessly. |
Need to match exact OpenFOAM 2.0.x for patch
I think your compilation failed earlier. You need to find the first error to diagnose the problem.
The problem you are seeing is a symptom (not the cause) related to missing lnInclude directories. My guess is that you patched the wrong version of OpenFOAM 2.0.x. Did you see errors during the patch process? You have to exactly match the git version of OpenFOAM 2.0.x with the patch: git commit: 2a7a68737dba87d82500df67029a15aeda662c42, Date: Sept. 14 11:28:07 2011 i.e., git clone http://github.com/OpenFOAM/OpenFOAM-2.0.x cd OpenFOAM-2.0.x git checkout 2a7a68737dba87d82500df67029a15aeda662c42 |
still occuring error
Hej Richard,
I did as you suggested git clone http://github.com/OpenFOAM/OpenFOAM-2.0.x cd OpenFOAM-2.0.x git checkout 2a7a68737dba87d82500df67029a15aeda662c42 and then installed the patch, which goes in without errors now then i do chmod ug+x <OpenFOAM-root>/src/gpu/Allwmake WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2 where I am not sure about the last one, ofgpu-0-2 contains Code:
CMakeCache.txt CMakeCache.txt~ CMakeFiles cmake_install.cmake CMakeLists.txt COPYING.txt install install_manifest.txt Makefile ofgpu README.txt I then start the compilation with ./Allwmake and get the following first error Code:
lex.yy.c:1391: warning: ‘yyunput’ defined but not used Code:
+ [ TRUE = ] |
Paths wrong
Errors I see:
So there's something wrong with your paths. Don't bother compiling MPI (doesn't work with GPU), so set "WM_MPLIB=NONE" Check that "/home/roman/OpenFOAM/ofgpu-0-2/ofgpu/*.h" exists The fact that you don't have "<ofgpu-root>/install/release/lib" sounds like you didn't set "Set the CMAKE_INSTALL_PREFIX variable to <ofgpu-root>/install" or you didn't set the ofgpu build type to "release" Check that the ofgpu build and install was successful. |
bashrc args
Another thought.
You have to supply your settings "WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2" as args to etc/bashrc e.g.,: export FOAM_INST_DIR="$(HOME)/projects/of/of2.0"; source $(FOAM_INST_DIR)/OpenFOAM-2.0.x/etc/bashrc WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2 etc... |
Performance of OFGPU
Hi all,
First I should thank gocarts for supporting OFGPU :) I've succeeded to compile and run OpenFOAM-1.7.1 and OFGPU-0.1 and evaluate it. Here I have reported the performance comparison between CPU and GPU. The benchmark is pitzDailyGPU. One can run the benchmark using following commands: blockMesh simpleFoam System CPU: AMD Opteron 6134 System GPU: NVIDIA Tesla C2050 For 12225 Cells: CPU Time: 16 s GPU Time: 45 s For 122250 Cells: CPU Time: 303 s GPU Time: 105 s For 299100 Cells: CPU Time: 808 s GPU Time: 186 s |
All times are GMT -4. The time now is 09:06. |