|
[Sponsors] |
April 23, 2011, 19:16 |
GPU Linear Solvers for OpenFOAM
|
#1 |
Senior Member
Richard Smith
Join Date: Mar 2009
Location: Enfield, NH, USA
Posts: 138
Blog Entries: 4
Rep Power: 17 |
ofgpu is a free GPL library from Symscape that provides GPU linear solvers for OpenFOAM®. The experimental library targets NVIDIA CUDA devices on Windows, Linux, and (untested) Mac OS X. It uses the Cusp library's Krylov solvers to produce equivalent GPU (CUDA-based) versions of the standard OpenFOAM linear solvers:
ofgpu also has support for the OpenFOAM preconditioners:
For more details see "GPU Linear Solver Library for OpenFOAM". OpenFOAM is a registered trademark of OpenCFD and is unaffiliated with Symscape.
__________________
Symscape, Computational Fluid Dynamics for all |
|
May 2, 2011, 04:04 |
|
#2 |
Senior Member
Bernhard
Join Date: Sep 2009
Location: Delft
Posts: 790
Rep Power: 22 |
Hi Richard,
Thanks for the nice work. I was able to compile OpenFOAM-1.7.x and use your plugin on the GPU. Your testcase also worked nicely. Some comments that may help other people that are installing this library. - I set gcc 4.3 as a compiler in ofgpu for C, and g++ (this is standard) for C++. - Other people in my department are already using CUDA, but not Cusp and Thrust, I downloaded Cusp and Thrust code-packages to /ofgpu/src - To patch, I used patch -p0 < file.patch To compile OpenFOAM 1.7.x with the gpu library, I did the following - chmod 770 src/gpu/Allwmake - touch src/gpu/gpuless/Make/options (this file was missing) - I think there is a mistake in src/gpu/cuda/Make/options, I used LIB = $(FOAM_LIBBIN)/cuda/libgpu there - In OF/lib I made a symbolic link named libofgpu.so to the library in the ofgpu directory You can check the activity of your graphical card with nvidia-smi -a -l For the standard GPU in my machine with nice CPUs, this test case (pitzDaily) there was only slow down of course. Richard, thanks for the nice work, I read on the website you're working on implementing additional preconditioners. I'm very of course very interested in those. Keep up the good work! Best regards, Bernhard |
|
May 2, 2011, 04:46 |
|
#3 |
Senior Member
Pablo
Join Date: Mar 2009
Posts: 102
Rep Power: 17 |
Is It necessary to compile all openFoam, or just a library??
Pablo |
|
May 2, 2011, 05:51 |
|
#4 |
Senior Member
Bernhard
Join Date: Sep 2009
Location: Delft
Posts: 790
Rep Power: 22 |
In principle you only have to recompile those parts of the code that are affected by the patch I suppose. Since I have been working with 1.7.1, and this library was designed for 1.7.x, I decided to take the save route and download and compile 1.7.x separate from my other installation. It would be worth trying it on your existing compiled OpenFOAM version.
|
|
May 2, 2011, 09:13 |
Thanks for the feedback
|
#5 | ||
Senior Member
Richard Smith
Join Date: Mar 2009
Location: Enfield, NH, USA
Posts: 138
Blog Entries: 4
Rep Power: 17 |
Thanks for the feedback Bernard.
You raised some good points: Quote:
Quote:
I've edited the build description to make this clearer at: http://www.symscape.com/gpu-openfoam
__________________
Symscape, Computational Fluid Dynamics for all |
|||
October 17, 2011, 08:52 |
Casting problem
|
#6 |
New Member
Ahmad Lashgar
Join Date: Aug 2010
Posts: 10
Rep Power: 16 |
Hi,
I've successfully compiled ofgpu.During compilation of OpenFOAM there is casting problem between ofgpu::index and Foam::label. I can solve the problem by manually cast-forcing. But, is there any quick solution? Thanks. The related error is shown here: PBiCGgpu.C:103: error: no matching function for call to ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, Foam::label, Foam::label, ofgpu::index*, ofgpu::index*, ofgpu::index*, ofgpu::real*, ofgpu::real*, ofgpu::index*, ofgpu::index*, ofgpu::real*, ofgpu::real*, ofgpu::real*, int, ofgpu::real, ofgpu::real, ofgpu::real, ofgpu::real, Foam::label, bool&) /home/tesla/OpenFOAM/ofgpu/src/ofgpu/sparsematrixargs.h:46: note: candidates are: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, ofgpu::index, ofgpu::index, const ofgpu::index*, const ofgpu::index*, const ofgpu::index*, const ofgpu::real*, const ofgpu::real*, const ofgpu::index*, const ofgpu::index*, const ofgpu::real*, ofgpu::real*, const ofgpu::real*, ofgpu::index, ofgpu::real, ofgpu::real, ofgpu::real&, ofgpu::real&, ofgpu::index&, bool&) /home/tesla/OpenFOAM/ofgpu/src/ofgpu/sparsematrixargs.h:24: note: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const ofgpu::SparseMatrixArgs&) make: *** [Make/linux64GccDPOpt/PBiCGgpu.o] Error 1 |
|
October 17, 2011, 09:36 |
Foam::label as long, why?
|
#7 |
Senior Member
Richard Smith
Join Date: Mar 2009
Location: Enfield, NH, USA
Posts: 138
Blog Entries: 4
Rep Power: 17 |
So I guess on your system (Linux?) you have Foam::label defined as a long?
That would result from defining 'FOAM_LABEL_64' for label.H Is this the case? Do you really need access to integers larger than 2,000,000,000? Do you want to solve for more than 2 Billion cells/faces/nodes? So my first suggestion is that if you don't need to define FOAM_LABEL_64 then don't (you save memory too) and recompile OpenFOAM. Second option if you do want to keep FOAM_LABEL_64 then try changing the definition of ofgpu::index to long in index.h. This will cost you scarce GPU memory if it works.
__________________
Symscape, Computational Fluid Dynamics for all |
|
October 17, 2011, 10:02 |
The patch file issues
|
#8 |
New Member
Ahmad Lashgar
Join Date: Aug 2010
Posts: 10
Rep Power: 16 |
Dear gocarts,
Thanks for the quick reply. You are right, FOAM_LABEL_64 is defined automatically because the operating system is "Ubuntu-Desktop-10.10 64-bit". I've solved the problem with casting. The compilation proceeds and fails upon Allmake in "/home/tesla/OpenFOAM/OpenFOAM-1.7.1/applications/solvers/multiphase/compressibleInterFoam" directory. Main errors are: compressibleInterFoam.C:43:22: error: RASModel.H: No such file or directory compressibleInterFoam.C:44:22: error: LESModel.H: No such file or directory followed by: compressibleInterFoam.C:51: error: "incompressible" has not been declared compressibleInterFoam.C:52: error: "incompressible" has not been declared This is caused by the patch file which makes some changes and adds "#include RASModel.H" and "#include LESModel.H" lines to compressibleInterFoam.C. However, these files are not exists in compressibleInterFoam directory. These files are in the following path: ./src/turbulenceModels/incompressible/RAS/RASModel/RASModel. ./src/turbulenceModels/compressible/RAS/RASModel/RASModel.H Should I make a soft link? or I have missed something in earlier stages Thanks. System: Ubuntu-10.10, OpenFOAM-1.7.1, CUDA 3.1 and GCC 4.4 |
|
October 17, 2011, 10:14 |
What about ofgpu 0.2?
|
#9 |
Senior Member
Richard Smith
Join Date: Mar 2009
Location: Enfield, NH, USA
Posts: 138
Blog Entries: 4
Rep Power: 17 |
Wouldn't you prefer to build the latest version of ofgpu 0.2 with OpenFOAM 2.0.x? http://www.symscape.com/gpu-0-2-openfoam
By the looks of your errors it seems that your build may have failed to create the turbulence libraries. The missing includes should be in: ./src/turbulenceModels/incompressible/lnInclude ./src/turbulenceModels/compressible/lnInclude
__________________
Symscape, Computational Fluid Dynamics for all |
|
October 17, 2011, 10:25 |
Ofgpu_0.2
|
#10 |
New Member
Ahmad Lashgar
Join Date: Aug 2010
Posts: 10
Rep Power: 16 |
Happy to see OFGPU 0.2 is compatible with CUDA 4.0.
I have downgraded thrust, cusp and cuda to make the required compatibility. I will test OFGPU 0.2 soon, Thank you |
|
October 17, 2011, 10:32 |
Don't define FOAM_LABEL_64
|
#11 |
Senior Member
Richard Smith
Join Date: Mar 2009
Location: Enfield, NH, USA
Posts: 138
Blog Entries: 4
Rep Power: 17 |
Just to follow up on:
I can't see where this is defined by default. It's not about whether you are compiling 64-bit, it relates to the size of integers. If you can find where it's defined I suggest removing it. Most likely it's somewhere in: ./wmake/rules/<platform>/c++ Edit: Should be FOAM_LABEL64 not FOAM_LABEL_64
__________________
Symscape, Computational Fluid Dynamics for all Last edited by gocarts; October 17, 2011 at 10:35. Reason: Correct error |
|
October 17, 2011, 10:38 |
label.H
|
#12 |
New Member
Ahmad Lashgar
Join Date: Aug 2010
Posts: 10
Rep Power: 16 |
Actually It is not defined! Following commands show that FOAM_LABEL_64 does not exist but FOAM_LABEL64 is used in label.H and uLabel.H
root@gpu30:~/OpenFOAM# grep -r ./ -e FOAM_LABEL_64 root@gpu30:~/OpenFOAM# grep -r ./ -e FOAM_LABEL64 ./OpenFOAM-1.7.1/src/OpenFOAM/lnInclude/label.H:#if FOAM_LABEL64 ./OpenFOAM-1.7.1/src/OpenFOAM/lnInclude/uLabel.H:#if FOAM_LABEL64 ./OpenFOAM-1.7.1/src/OpenFOAM/primitives/ints/label/label.H:#if FOAM_LABEL64 ./OpenFOAM-1.7.1/src/OpenFOAM/primitives/ints/uLabel/uLabel.H:#if FOAM_LABEL64 So the label type is int not long. |
|
October 17, 2011, 10:51 |
Double precision vs single precision
|
#13 |
Senior Member
Richard Smith
Join Date: Mar 2009
Location: Enfield, NH, USA
Posts: 138
Blog Entries: 4
Rep Power: 17 |
I think then your problems are two fold:
__________________
Symscape, Computational Fluid Dynamics for all |
|
October 17, 2011, 11:03 |
How to run the sample?
|
#14 |
New Member
Ahmad Lashgar
Join Date: Aug 2010
Posts: 10
Rep Power: 16 |
OK. I think the OpenFOAM is now compiled successfully.
I'm new in OpenFOAM. How can run "pitzDailyGpu" sample? I've extracted the archive, now there exist 3 folders: 0 constant system Thanks. |
|
October 21, 2011, 10:51 |
OF compilation problems
|
#15 |
Senior Member
Roman Thiele
Join Date: Aug 2009
Location: Eindhoven, NL
Posts: 374
Rep Power: 21 |
Hej,
I still have a problem with the compiling I use GCC 4.4, thrust 1.4, cusp 0.2, OF 2.0.x (Oct 21st) and applied the patch. Everything else before of-gpu is compiled without errors. After applying the patch, I can't compile OF 2.0.x anymore. I get problems. There must be a problem with the include or something because it is missing basically all the files for include like the following Code:
Making dependency list for source file pimpleFoam.C could not open file fvCFD.H for source file pimpleFoam.C Making dependency list for source file icoUncoupledKinematicParcelDyMFoam.C could not open file fvCFD.H for source file icoUncoupledKinematicParcelDyMFoam.C could not open file dynamicFvMesh.H for source file icoUncoupledKinematicParcelDyMFoam.C could not open file volFieldsFwd.H for source file pimpleFoam.C could not open file surfaceFieldsFwd.H for source file pimpleFoam.C could not open file fvMatricesFwd.H for source file pimpleFoam.C could not open file pimpleControl.H for source file pimpleFoam.C could not open file createPhi.H for source file pimpleFoam.C could not open file initContinuityErrs.H for source file pimpleFoam.C could not open file readTimeControls.H for source file pimpleFoam.C could not open file CourantNo.H for source file pimpleFoam.C could not open file setDeltaT.H for source file pimpleFoam.C could not open file continuityErrs.H for source file pimpleFoam.C could not open file volFieldsFwd.H for source file icoUncoupledKinematicParcelDyMFoam.C could not open file surfaceFieldsFwd.H for source file icoUncoupledKinematicParcelDyMFoam.C could not open file fvMatricesFwd.H for source file icoUncoupledKinematicParcelDyMFoam.C could not open file basicKinematicCollidingCloud.H for source file icoUncoupledKinematicParcelDyMFoam.C could not open file createDynamicFvMesh.H for source file icoUncoupledKinematicParcelDyMFoam.C could not open file readGravitationalAcceleration.H for source file icoUncoupledKinematicParcelDyMFoam.C could not open file createPhi.H for source file icoUncoupledKinematicParcelDyMFoam.C I tried compiling it without the patch and that works flawlessly.
__________________
~roman |
|
October 21, 2011, 11:03 |
Need to match exact OpenFOAM 2.0.x for patch
|
#16 |
Senior Member
Richard Smith
Join Date: Mar 2009
Location: Enfield, NH, USA
Posts: 138
Blog Entries: 4
Rep Power: 17 |
I think your compilation failed earlier. You need to find the first error to diagnose the problem.
The problem you are seeing is a symptom (not the cause) related to missing lnInclude directories. My guess is that you patched the wrong version of OpenFOAM 2.0.x. Did you see errors during the patch process? You have to exactly match the git version of OpenFOAM 2.0.x with the patch: git commit: 2a7a68737dba87d82500df67029a15aeda662c42, Date: Sept. 14 11:28:07 2011 i.e., git clone http://github.com/OpenFOAM/OpenFOAM-2.0.x cd OpenFOAM-2.0.x git checkout 2a7a68737dba87d82500df67029a15aeda662c42
__________________
Symscape, Computational Fluid Dynamics for all |
|
October 21, 2011, 13:04 |
still occuring error
|
#17 |
Senior Member
Roman Thiele
Join Date: Aug 2009
Location: Eindhoven, NL
Posts: 374
Rep Power: 21 |
Hej Richard,
I did as you suggested git clone http://github.com/OpenFOAM/OpenFOAM-2.0.x cd OpenFOAM-2.0.x git checkout 2a7a68737dba87d82500df67029a15aeda662c42 and then installed the patch, which goes in without errors now then i do chmod ug+x <OpenFOAM-root>/src/gpu/Allwmake WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2 where I am not sure about the last one, ofgpu-0-2 contains Code:
CMakeCache.txt CMakeCache.txt~ CMakeFiles cmake_install.cmake CMakeLists.txt COPYING.txt install install_manifest.txt Makefile ofgpu README.txt I then start the compilation with ./Allwmake and get the following first error Code:
lex.yy.c:1391: warning: ‘yyunput’ defined but not used lex.yy.c:1432: warning: ‘input’ defined but not used + set -e + wmakePrintBuild -check version changed from previous build + /bin/rm -f OpenFOAM/Make/*/global.? + wmakeLnInclude OpenFOAM wmakeLnInclude: linking include files to OpenFOAM/lnInclude + wmakeLnInclude OSspecific/POSIX wmakeLnInclude: linking include files to OSspecific/POSIX/lnInclude + OSspecific/POSIX/Allwmake wmakeLnInclude: linking include files to ./lnInclude Making dependency list for source file signals/sigSegv.C Making dependency list for source file signals/sigFpe.C Making dependency list for source file signals/sigInt.C Making dependency list for source file signals/sigQuit.C Making dependency list for source file regExp.C Making dependency list for source file fileStat.C Making dependency list for source file timer.C Making dependency list for source file POSIX.C Making dependency list for source file clockTime/clockTime.C Making dependency list for source file cpuTime/cpuTime.C Making dependency list for source file memInfo/memInfo.C Making dependency list for source file printStack.C Making dependency list for source file fileMonitor.C POSIX.C: In function ‘bool Foam::ping(const Foam::word&, Foam::label, Foam::label)’: POSIX.C:1074:25: warning: use of old-style cast [-Wold-style-cast] POSIX.C:1074:25: warning: use of old-style cast [-Wold-style-cast] + Pstream/Allwmake + wmake libso dummy wmakeLnInclude: linking include files to ./lnInclude Making dependency list for source file UIPread.C Making dependency list for source file UOPwrite.C Making dependency list for source file UPstream.C + set +x wmakeLnInclude: linking include files to ./lnInclude Making dependency list for source file UIPread.C Making dependency list for source file UOPwrite.C could not open file omp.h for source file UIPread.C could not open file omp.h for source file UOPwrite.C Making dependency list for source file UPstream.C could not open file omp.h for source file UPstream.C could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file UIPread.C Making dependency list for source file PstreamGlobals.C could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file UOPwrite.C could not open file omp.h for source file PstreamGlobals.C could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file UPstream.C could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file PstreamGlobals.C Code:
+ [ TRUE = ] + gpu/Allwmake + wmake libso gpuless wmakeLnInclude: linking include files to ./lnInclude Making dependency list for source file PBiCGgpu.C Making dependency list for source file PCGgpu.C + wmake libso cuda wmakeLnInclude: linking include files to ./lnInclude Making dependency list for source file cudaGpu.C could not open file ofgpu/ofgpuconfig.h for source file cudaGpu.C Making dependency list for source file PBiCGgpu.C could not open file ofgpu/pbicg.h for source file PBiCGgpu.C could not open file ofgpu/sparsematrixargs.h for source file PBiCGgpu.C Making dependency list for source file PCGgpu.C could not open file ofgpu/pcg.h for source file PCGgpu.C could not open file ofgpu/sparsematrixargs.h for source file PCGgpu.C PBiCGgpu.C:23:25: fatal error: ofgpu/pbicg.h: No such file or directorycudaGpu.C:23:31: fatal error: ofgpu/ofgpuconfig.h: No such file or directory compilation terminated.
__________________
~roman |
|
October 21, 2011, 13:31 |
Paths wrong
|
#18 |
Senior Member
Richard Smith
Join Date: Mar 2009
Location: Enfield, NH, USA
Posts: 138
Blog Entries: 4
Rep Power: 17 |
Errors I see:
So there's something wrong with your paths. Don't bother compiling MPI (doesn't work with GPU), so set "WM_MPLIB=NONE" Check that "/home/roman/OpenFOAM/ofgpu-0-2/ofgpu/*.h" exists The fact that you don't have "<ofgpu-root>/install/release/lib" sounds like you didn't set "Set the CMAKE_INSTALL_PREFIX variable to <ofgpu-root>/install" or you didn't set the ofgpu build type to "release" Check that the ofgpu build and install was successful.
__________________
Symscape, Computational Fluid Dynamics for all Last edited by gocarts; October 21, 2011 at 13:42. Reason: typo |
|
October 21, 2011, 15:31 |
bashrc args
|
#19 |
Senior Member
Richard Smith
Join Date: Mar 2009
Location: Enfield, NH, USA
Posts: 138
Blog Entries: 4
Rep Power: 17 |
Another thought.
You have to supply your settings "WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2" as args to etc/bashrc e.g.,: export FOAM_INST_DIR="$(HOME)/projects/of/of2.0"; source $(FOAM_INST_DIR)/OpenFOAM-2.0.x/etc/bashrc WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2 etc...
__________________
Symscape, Computational Fluid Dynamics for all |
|
October 22, 2011, 09:34 |
Performance of OFGPU
|
#20 |
New Member
Ahmad Lashgar
Join Date: Aug 2010
Posts: 10
Rep Power: 16 |
Hi all,
First I should thank gocarts for supporting OFGPU I've succeeded to compile and run OpenFOAM-1.7.1 and OFGPU-0.1 and evaluate it. Here I have reported the performance comparison between CPU and GPU. The benchmark is pitzDailyGPU. One can run the benchmark using following commands: blockMesh simpleFoam System CPU: AMD Opteron 6134 System GPU: NVIDIA Tesla C2050 For 12225 Cells: CPU Time: 16 s GPU Time: 45 s For 122250 Cells: CPU Time: 303 s GPU Time: 105 s For 299100 Cells: CPU Time: 808 s GPU Time: 186 s |
|
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Direct vs Iterative Linear Solvers for non-linears | bill | Main CFD Forum | 16 | November 5, 2014 07:18 |
solution diverges when linear upwind interpolation scheme is used | subash | OpenFOAM | 0 | May 29, 2010 01:23 |
New linear system solvers | chegdan | OpenFOAM Running, Solving & CFD | 11 | April 30, 2010 10:22 |
Water pump OpenFOAM 15 ANSYS CFX 110 comparation | waynezw0618 | OpenFOAM Running, Solving & CFD | 39 | March 5, 2009 12:57 |
Linear Solvers | Sachin | Main CFD Forum | 0 | May 6, 2006 12:07 |