GPU Linear Solvers for OpenFOAM
 

ofgpu is a free GPL library from Symscape that provides GPU linear solvers for OpenFOAM®. The experimental library targets NVIDIA CUDA devices on Windows, Linux, and (untested) Mac OS X. It uses the Cusp library's Krylov solvers to produce equivalent GPU (CUDA-based) versions of the standard OpenFOAM linear solvers:

• PCG - Preconditioned conjugate gradient solver for symmetric matrices (e.g., p)
• PBiCG - Preconditioned biconjugate gradient solver for asymmetric matrices (e.g., Ux, k)

ofgpu also has support for the OpenFOAM preconditioners:

• no
• diagonal

For more details see "GPU Linear Solver Library for OpenFOAM".

OpenFOAM is a registered trademark of OpenCFD and is unaffiliated with Symscape.

Hi Richard,

Thanks for the nice work. I was able to compile OpenFOAM-1.7.x and use your plugin on the GPU. Your testcase also worked nicely.

Some comments that may help other people that are installing this library.
- I set gcc 4.3 as a compiler in ofgpu for C, and g++ (this is standard) for C++.
- Other people in my department are already using CUDA, but not Cusp and Thrust, I downloaded Cusp and Thrust code-packages to /ofgpu/src
- To patch, I used patch -p0 < file.patch

To compile OpenFOAM 1.7.x with the gpu library, I did the following
- chmod 770 src/gpu/Allwmake
- touch src/gpu/gpuless/Make/options (this file was missing)
- I think there is a mistake in src/gpu/cuda/Make/options, I used LIB = $(FOAM_LIBBIN)/cuda/libgpu there
- In OF/lib I made a symbolic link named libofgpu.so to the library in the ofgpu directory

You can check the activity of your graphical card with nvidia-smi -a -l

For the standard GPU in my machine with nice CPUs, this test case (pitzDaily) there was only slow down of course.

Richard, thanks for the nice work, I read on the website you're working on implementing additional preconditioners. I'm very of course very interested in those. Keep up the good work!

Best regards,
Bernhard

Is It necessary to compile all openFoam, or just a library??

Pablo

In principle you only have to recompile those parts of the code that are affected by the patch I suppose. Since I have been working with 1.7.1, and this library was designed for 1.7.x, I decided to take the save route and download and compile 1.7.x separate from my other installation. It would be worth trying it on your existing compiled OpenFOAM version.

Thanks for the feedback
 

Thanks for the feedback Bernard.

You raised some good points:

I'll update the source code patch shortly to fix these 2 problems.

I think these 2 issues are caused because you are missing one or more of the following:

• CMAKE_INSTALL_PREFIX variable as <ofgpu-root>/install - while configuring ofgpu with ccmake
• ln -s <ofgpu-root>/install/release/lib <ofgpu-root>/install/lib - after 'make install'
• export CUDA_ARCH_PATH=<ofgpu-root> - prior to building and running OpenFOAM

I've edited the build description to make this clearer at: http://www.symscape.com/gpu-openfoam

Casting problem
 

Hi,

I've successfully compiled ofgpu.During compilation of OpenFOAM there is casting problem between ofgpu::index and Foam::label.
I can solve the problem by manually cast-forcing. But, is there any quick solution?

Thanks.

The related error is shown here:
PBiCGgpu.C:103: error: no matching function for call to ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, Foam::label, Foam::label, ofgpu::index*, ofgpu::index*, ofgpu::index*, ofgpu::real*, ofgpu::real*, ofgpu::index*, ofgpu::index*, ofgpu::real*, ofgpu::real*, ofgpu::real*, int, ofgpu::real, ofgpu::real, ofgpu::real, ofgpu::real, Foam::label, bool&)
/home/tesla/OpenFOAM/ofgpu/src/ofgpu/sparsematrixargs.h:46: note: candidates are: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, ofgpu::index, ofgpu::index, const ofgpu::index*, const ofgpu::index*, const ofgpu::index*, const ofgpu::real*, const ofgpu::real*, const ofgpu::index*, const ofgpu::index*, const ofgpu::real*, ofgpu::real*, const ofgpu::real*, ofgpu::index, ofgpu::real, ofgpu::real, ofgpu::real&, ofgpu::real&, ofgpu::index&, bool&)
/home/tesla/OpenFOAM/ofgpu/src/ofgpu/sparsematrixargs.h:24: note: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const ofgpu::SparseMatrixArgs&)
make: *** [Make/linux64GccDPOpt/PBiCGgpu.o] Error 1

Foam::label as long, why?
 

So I guess on your system (Linux?) you have Foam::label defined as a long?

That would result from defining 'FOAM_LABEL_64' for label.H

Is this the case?

Do you really need access to integers larger than 2,000,000,000? Do you want to solve for more than 2 Billion cells/faces/nodes?

So my first suggestion is that if you don't need to define FOAM_LABEL_64 then don't (you save memory too) and recompile OpenFOAM.

Second option if you do want to keep FOAM_LABEL_64 then try changing the definition of ofgpu::index to long in index.h. This will cost you scarce GPU memory if it works.

The patch file issues
 

Dear gocarts,
Thanks for the quick reply.

You are right, FOAM_LABEL_64 is defined automatically because the operating system is "Ubuntu-Desktop-10.10 64-bit". I've solved the problem with casting.

The compilation proceeds and fails upon Allmake in "/home/tesla/OpenFOAM/OpenFOAM-1.7.1/applications/solvers/multiphase/compressibleInterFoam" directory.

Main errors are:
compressibleInterFoam.C:43:22: error: RASModel.H: No such file or directory
compressibleInterFoam.C:44:22: error: LESModel.H: No such file or directory
followed by:
compressibleInterFoam.C:51: error: "incompressible" has not been declared
compressibleInterFoam.C:52: error: "incompressible" has not been declared

This is caused by the patch file which makes some changes and adds "#include RASModel.H" and "#include LESModel.H" lines to compressibleInterFoam.C. However, these files are not exists in compressibleInterFoam directory.

These files are in the following path:
./src/turbulenceModels/incompressible/RAS/RASModel/RASModel.
./src/turbulenceModels/compressible/RAS/RASModel/RASModel.H

Should I make a soft link? or I have missed something in earlier stages

Thanks.

System: Ubuntu-10.10, OpenFOAM-1.7.1, CUDA 3.1 and GCC 4.4

What about ofgpu 0.2?
 

Wouldn't you prefer to build the latest version of ofgpu 0.2 with OpenFOAM 2.0.x? http://www.symscape.com/gpu-0-2-openfoam

By the looks of your errors it seems that your build may have failed to create the turbulence libraries. The missing includes should be in:

./src/turbulenceModels/incompressible/lnInclude
./src/turbulenceModels/compressible/lnInclude

Ofgpu_0.2
 

Happy to see OFGPU 0.2 is compatible with CUDA 4.0.
I have downgraded thrust, cusp and cuda to make the required compatibility.

I will test OFGPU 0.2 soon,
Thank you

Don't define FOAM_LABEL_64
 

Just to follow up on:

I can't see where this is defined by default. It's not about whether you are compiling 64-bit, it relates to the size of integers. If you can find where it's defined I suggest removing it. Most likely it's somewhere in:

./wmake/rules/<platform>/c++

Edit: Should be FOAM_LABEL64 not FOAM_LABEL_64

label.H
 

Actually It is not defined! Following commands show that FOAM_LABEL_64 does not exist but FOAM_LABEL64 is used in label.H and uLabel.H

root@gpu30:~/OpenFOAM# grep -r ./ -e FOAM_LABEL_64
root@gpu30:~/OpenFOAM# grep -r ./ -e FOAM_LABEL64
./OpenFOAM-1.7.1/src/OpenFOAM/lnInclude/label.H:#if FOAM_LABEL64
./OpenFOAM-1.7.1/src/OpenFOAM/lnInclude/uLabel.H:#if FOAM_LABEL64
./OpenFOAM-1.7.1/src/OpenFOAM/primitives/ints/label/label.H:#if FOAM_LABEL64
./OpenFOAM-1.7.1/src/OpenFOAM/primitives/ints/uLabel/uLabel.H:#if FOAM_LABEL64

So the label type is int not long.

Double precision vs single precision
 

I think then your problems are two fold:

• You need to patch the exact version of OpenFOAM 1.7.x (not 1.7.1) specified in the instructions at: http://www.symscape.com/gpu-openfoam
• I guess you are trying to compile in double precision, whereas ofgpu assumes single precision

I haven't tried double precision, so I can't help you with that, though someone in the comments for the instructions mentioned it.

How to run the sample?
 

OK. I think the OpenFOAM is now compiled successfully.

I'm new in OpenFOAM. How can run "pitzDailyGpu" sample?
I've extracted the archive, now there exist 3 folders:
0 constant system

Thanks.

OF compilation problems
 

Hej,

I still have a problem with the compiling

I use GCC 4.4, thrust 1.4, cusp 0.2, OF 2.0.x (Oct 21st) and applied the patch. Everything else before of-gpu is compiled without errors.

After applying the patch, I can't compile OF 2.0.x anymore. I get problems.

There must be a problem with the include or something because it is missing basically all the files for include like the following

Unfortunately, this happens at all stages during the compile process.

I tried compiling it without the patch and that works flawlessly.

Need to match exact OpenFOAM 2.0.x for patch
 

I think your compilation failed earlier. You need to find the first error to diagnose the problem.

The problem you are seeing is a symptom (not the cause) related to missing lnInclude directories.

My guess is that you patched the wrong version of OpenFOAM 2.0.x. Did you see errors during the patch process?

You have to exactly match the git version of OpenFOAM 2.0.x with the patch:

git commit: 2a7a68737dba87d82500df67029a15aeda662c42, Date: Sept. 14 11:28:07 2011

i.e.,

git clone http://github.com/OpenFOAM/OpenFOAM-2.0.x
cd OpenFOAM-2.0.x
git checkout 2a7a68737dba87d82500df67029a15aeda662c42

still occuring error
 

Hej Richard,

I did as you suggested

git clone http://github.com/OpenFOAM/OpenFOAM-2.0.x
cd OpenFOAM-2.0.x
git checkout 2a7a68737dba87d82500df67029a15aeda662c42

and then

installed the patch, which goes in without errors now

then i do

chmod ug+x <OpenFOAM-root>/src/gpu/Allwmake
WM_GPU='CUDA'
CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2

where I am not sure about the last one, ofgpu-0-2 contains
so i think i have it right, the library when I compiled it, never was under a install/release/lib folder but right away under install/lib so i didn't need to create the soft-link

I then start the compilation with

./Allwmake

and get the following first error

it then has exactly the same errors as before, that it can't open the header files for the C files, because it can't find them, like beginning with the gpu part

Paths wrong
 

Errors I see:

• "could not open file omp.h for source file UIPread.C"
• "could not open file ofgpu/ofgpuconfig.h for source file cudaGpu.C"

So there's something wrong with your paths.

Don't bother compiling MPI (doesn't work with GPU), so set "WM_MPLIB=NONE"
Check that "/home/roman/OpenFOAM/ofgpu-0-2/ofgpu/*.h" exists

The fact that you don't have "<ofgpu-root>/install/release/lib" sounds like you didn't set "Set the CMAKE_INSTALL_PREFIX variable to <ofgpu-root>/install" or you didn't set the ofgpu build type to "release"

Check that the ofgpu build and install was successful.

bashrc args
 

Another thought.

You have to supply your settings "WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2"

as args to etc/bashrc e.g.,:

export FOAM_INST_DIR="$(HOME)/projects/of/of2.0"; source $(FOAM_INST_DIR)/OpenFOAM-2.0.x/etc/bashrc WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2 etc...

Performance of OFGPU
 

Hi all,

First I should thank gocarts for supporting OFGPU :)

I've succeeded to compile and run OpenFOAM-1.7.1 and OFGPU-0.1 and evaluate it. Here I have reported the performance comparison between CPU and GPU. The benchmark is pitzDailyGPU. One can run the benchmark using following commands:
blockMesh
simpleFoam

System CPU: AMD Opteron 6134
System GPU: NVIDIA Tesla C2050

For 12225 Cells:
CPU Time: 16 s
GPU Time: 45 s

For 122250 Cells:
CPU Time: 303 s
GPU Time: 105 s

For 299100 Cells:
CPU Time: 808 s
GPU Time: 186 s

Hej Richard

I made sure that the files exist and I sourced everything again.

I then start the code, but I am still getting errors. The path should be right. Compiling of of-gpu went find and I didn't get any errors. I soft linked the install/release/lib to install/lib.

I marked something in read. I know that the path does not exist. I checked that the green-marked file and path exist. and they do. Further down the compilation it is missing many more files and always results in the same errors. like
in this one all the green lnInclude directories are missing.

To solve "fatal error: ofgpu/ofgpuconfig.h: No such file or directory"
 

Hi romant,

ofgpu-0.2 does not create ./src directory. you should make it your self.
To solve "fatal error: ofgpu/ofgpuconfig.h: No such file or directory" copy "ofgpu-0-2/ofgpu" directory to "ofgpu-0-2/src/":

mkdir ofgpu-0-2/src -p
cp -r ofgpu-0-2/ofgpu ofgpu-0-2/src/

Ahmad

thank you, but this still leaves some errors to work on. like all the linking errors that I mentioned with the normal openfoam :( and now it at least tries to compile part of the of-gpu, but still runs into problem, namely, when building PBiCGgpu.C

and of course the missing lnInclude directories.

$WM_PRECISION or a similar environment variable
 

Hi Roman,

I had the same problem. You should export $WM_PRECISION or a similar environment variable in <open-foam-root>/etc/bashrc

It should have "SP" value since ofgpu supports only single-precision floating point. For you, its value is "DP" probably.

Recompiling all modules is needed :)

Missing src directory
 

I see the problem - I missed the src directory in my package hierarchy (ofgpu-0-2.tgz). I've fixed it now, should be good to go. Thanks for finding/reporting this problem.

I've also added a note about single precision in the instructions: http://www.symscape.com/gpu-0-2-openfoam

Unknown function type cudaGpu
 

I finished compiling and obtained libgpu.so in
platforms/linux64GccSPOpt/lib/cuda/
but when I tried pitzDailyGpu, I've got the following error
Starting time loop

--------------------------
--> FOAM Warning :
From function dlOpen(const fileName&, const bool)
in file POSIX.C at line 1141
dlopen error : gpu: cannot open shared object file: No such file or directory
--> FOAM Warning :
From function dlLibraryTable::open(const fileName&, const bool)
in file db/dynamicLibrary/dlLibraryTable/dlLibraryTable.C at line 96
could not load "gpu"
--> FOAM Warning :
From function dlLibraryTable::open(const dictionary&, const word&, const TablePtr&)
in file lnInclude/dlLibraryTableTemplates.C at line 67
Could not open library "gpu"

--> FOAM FATAL ERROR:
Unknown function type cudaGpu
-------------------------------------------

am I missing something here ?

Check libofgpu.so
 

Check that libofgpu.so is also in your LD_LIBRARY_PATH

Greetings to all!
That name in bold is unrolled to "libgpu.so".

@arphy: Along with Richard's suggestion, I'm also guessing that there is a typo somewhere in you case's "controlDict" file? "gpu" instead of "ofgpu".

Best regards,
Bruno

gpu in controlDict is correct
 

It's correct to load libgpu.so ("gpu" in your controlDict) and I was thinking that given libgpu.so is in the right location, it is more likely that libofgpu.so (which libgpu.so references) was not being found and therefore loading libgpu.so would fail.

For reference libgpu.so is the bridge library between OpenFOAM and libofgpu.so. libofgpu.so is independent of OpenFOAM and only references CUDA libraries.

No, as discussed above controlDict should refer to gpu not ofgpu.

Bruno, partly this confusion is likely because I named the libs so similar and it's easy to miss refer to the wrong one.

Try replacing "gpu" with "libgpu.so"
 

Another thought, try replacing "gpu" in controlDict with "libgpu.so" - though this shouldn't be necessary. I suggest this because the dlopen error message shouldn't be reported in it's raw form, at least not in OpenFOAM 2.0.x, yet that is what you are seeing.

Just to be clear which version of OpenFOAM and ofgpu are you using?

Thanks you very much for your help. It is working now.
This part in my controlDict looks like this
functions
{
cudaGpu
{
type cudaGpu;
functionObjectLibs ("libgpu.so");
cudaDevice 0;
}
}

What do you think why the normal setting does not work ?
BTW. My OpenFoam is the one I clone from the repository yesterday (have no idea how to check the version via git)
I will redo another make with your designated version of OpenFoam.

Match git commit
 

I think the version you checked out of git has a modified Foam::dlopen routine.

For the patch to work correctly you need to match the exact version of OpenFOAM from git.

git commit: 2a7a68737dba87d82500df67029a15aeda662c42, Date: Sept. 14 11:28:07 2011.

Use:

1. git clone http://github.com/OpenFOAM/OpenFOAM-2.0.x
2. cd OpenFOAM-2.0.x
3. git checkout 2a7a68737dba87d82500df67029a15aeda662c42
4. apply patch

error with nviroment variable for OpenFOAM-2.1.x
 

As I download OpenFOAam-2.1.x from https://github.com/OpenFOAM/OpenFOAM-2.1.x and apply the patch.

I run the foamSystemCheck to check if I can compile OpenFOAM
then I run foamInstallationTest to check if I source my ~/.bashrc correctly
and I got the errors shown below. No error during applying patch. Plz help !!

ERROR MESSAGE IN TERMINAL:

Need Help debugging this message
 

I'm trying to compile cuda with wmake within ~/OpenFOAM/OpenFOAM-2.1.0/src/gpu/cuda/ and have the errors shown below. What does it mean ?

Probably that you should have used 'wmake libso' to build a library instead of an executable.

***EDIT***
I wasn't configuring cmake correctly. I followed directions in the comments of the symscape website and now have everything working.
************

Applying
$ cmake -DCMAKE_INSTALL_PREFIX=/opt/ofgpu-0-2/install

gives me the following errors

How do I resolve these errors? Should I be using -DCMAKE_INSTALL_PREFIX=/opt/ofgpu-0-2/src/ofgpu/install?

I can apply the OF patch with no errors.
I'm on 64bit Ubuntu 10.04. For reference my $HOME/.bashrc file contains

pitzDailyGpu example is not working
 

Hello,

I am trying to use ofgpu-1-1-0 library to accelerate the performance of linear solver in OpenFoam 2.2.x.
I have followed the steps given on http://www.symscape.com/gpu-1-1-openfoam link.
Try to execute the given example pitzDailyGpu.
But, the error is like...

-------------------------------------------------------------------------------------------------------------------------------
--> FOAM Warning :
From function dlOpen(const fileName&, const bool)
in file POSIX.C at line 1179
dlopen error : libgpu.so: cannot open shared object file: No such file or directory
--> FOAM Warning :
From function dlLibraryTable::open(const fileName&, const bool)
in file db/dynamicLibrary/dlLibraryTable/dlLibraryTable.C at line 99
could not load "libgpu.so"
--> FOAM Warning :
From function dlLibraryTable::open(const dictionary&, const word&, const TablePtr&)
in file lnInclude/dlLibraryTableTemplates.C at line 67
Could not open library "libgpu.so"



--> FOAM FATAL ERROR:
Unknown function type cudaGpu

Valid functions are :

4
(
patchProbes
probes
sets
surfaces
)



From function functionObject::New(const word& name, const Time&, const dictionary&)
in file db/functionObjects/functionObject/functionObject.C at line 92.

FOAM exiting
-------------------------------------------------------------------------------------------------------------------------------

As told in the earlier post that change the functionObjectLibs ("gpu"); to functionObjectLibs ("libgpu.so");
still, the same error is occurring.

How can I resolve this error?

Hello,
I'm having trouble compiling ofgpu. I didn't really understand how to make this compilation. Can you help me?
Best Regards,
Diego