CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

Discussion thread on how to install and use RapidCFD

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   April 9, 2015, 07:27
Default Discussion thread on how to install and use RapidCFD
  #1
New Member
 
Oskar Grochowalski
Join Date: Jan 2015
Posts: 9
Rep Power: 3
newoscar is on a distinguished road
Quote:
Originally Posted by newoscar View Post
After weeks of hard work simFlow team made impossible become possible - OpenFOAM solvers are now able to run on CUDA! And thats just for starters.

All the computation is done entirely on GPU. No need to copy data during calculations between CPU and GPU. All of this for getting the job done faster.

Features:
  • most incompressible and compressible solvers on static mesh are available
  • all the calculations are done on the GPU
  • no overhead for GPU-CPU memory copy
  • can run in parallel on multiple GPUs

RapidCFD project page
______________

Moderator note: This thread has been created for discussing how to install and use RapidCFD. For announcements, subscribe to the announcement thread: http://www.cfd-online.com/Forums/ope...nning-gpu.html

Last edited by wyldckat; May 1, 2016 at 15:49. Reason: see "Moderator note:"
newoscar is offline   Reply With Quote

Old   August 4, 2015, 08:49
Default Discussiong thread on how to install and use RapidCFD
  #2
Senior Member
 
Robert
Join Date: Sep 2010
Posts: 158
Rep Power: 8
lordvon is on a distinguished road
Are there any installation instructions?

I am currently attempting installation by:
-Logging in with 'sudo bash'
-Placing 'RapidCFD-dev' in the '/opt/' folder
-'source etc/bashrc' in the RapidCFD-dev folder
-'./Allwmake' in the RapidCFD-dev folder

I get a lot of the following errors (separated by line breaks):

UOPwrite.C:29:17: fatal error: mpi.h: No such file or directory (mpirun works on my computer)

warning: call through incomplete class "Foam::tmp<Foam::Field<Foam::scalar>>" will always produce an error when instantiated

/usr/local/cuda-7.0/include/math_functions.hpp:3191:22: error: call of overloaded ‘sqrt(double&)’ is ambiguous

/usr/bin/ld: cannot find -ltriSurface
/usr/bin/ld: cannot find -lsurfMesh
collect2: error: ld returned 1 exit status
make: *** [/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libmeshTools.so] Error 1

/usr/bin/ld: cannot find -ltriSurface
/usr/bin/ld: cannot find -lmeshTools
/usr/bin/ld: cannot find -lsurfMesh
collect2: error: ld returned 1 exit status
make: *** [/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libedgeMesh.so] Error 1

Last edited by lordvon; August 4, 2015 at 16:26.
lordvon is offline   Reply With Quote

Old   August 5, 2015, 12:50
Default
  #3
New Member
 
Oskar Grochowalski
Join Date: Jan 2015
Posts: 9
Rep Power: 3
newoscar is on a distinguished road
There are no installation instructions yet, but with growing community we are planning to publish it.
The good news is that RapidCFD builds just like a regular OpenFOAM. If you are not familiar with this process follow this steps:
- set value of 'foamInstall' in 'etc/bashrc' file. This variable should point to the location of 'RapidCFD-dev' directory. By default this variable is set to $HOME/RapidCFD .
- source 'etc/bashrc' file
- run the 'Allwmake' file

To build in parallel set WM_NCOMPPROCS environment variable to the number of processes you want to use. This is recomended since compilation with nvcc is very slow.

Post a reply with your results
newoscar is offline   Reply With Quote

Old   November 16, 2015, 00:54
Default
  #4
aee
Member
 
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 30
Rep Power: 6
aee is on a distinguished road
I'm not completely clear: does this software use the same environment variables as OF? If so, I risk messing up my OF 3 installation, right?
aee is offline   Reply With Quote

Old   November 16, 2015, 12:50
Default
  #5
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,748
Blog Entries: 39
Rep Power: 103
wyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of lightwyldckat is a glorious beacon of light
Quote:
Originally Posted by aee View Post
I'm not completely clear: does this software use the same environment variables as OF? If so, I risk messing up my OF 3 installation, right?
Quick answer: You can have more than one OpenFOAM version installed. Check the instructions given here: http://openfoamwiki.net/index.php/In...with_the_Shell
wyldckat is offline   Reply With Quote

Old   December 29, 2015, 09:59
Default Errors installing RapidCFD
  #6
New Member
 
Tom
Join Date: Dec 2015
Location: Melbourne, Australia
Posts: 5
Rep Power: 2
mesh-monkey is on a distinguished road
Hi all,

I, like lordvon, am having troubles installing/compiling RapidCFD on my machine and am receiving similar errors.
I'm hoping someone here would be able to help me out as I've scoured as much of the web as I could to find answers but to no avail.

After following newoscar's instructions, as well as those on the github wiki and openfoam-wiki (thanks wyldckat!) as best possible, I seem to be receiving two types of issues, and many of each:
  1. cuda math_functions, e.g.:
    Code:
    /usr/local/cuda-6.5/targets/x86_64-linux/include/math_functions.h:14276:22: error: call of overloaded sqrt(double&) is ambiguous
       return 1.0 / sqrt(a);
                          ^
  2. cannot find -l* , e.g.:
    Code:
    + wmake libso triSurface
    /usr/bin/ld: cannot find -lsurfMesh
    collect2: error: ld returned 1 exit status
    /home/tom/OpenFOAM/RapidCFD-dev/wmake/Makefile:172: recipe for target '/home/tom/OpenFOAM/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libtriSurface.so' failed
    make: *** [/home/tom/OpenFOAM/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libtriSurface.so] Error 1
The log file of ./Allwmake is attached.

Some additional information about my setup that might be helpful:
  • Ubuntu 15.04 Desktop x64;
  • gcc (Ubuntu 4.8.4-1ubuntu15) 4.8.4;
  • CUDA version 6.5 & Quadro 6000 - confirmed working (CUDA samples run fine);
  • OpenMPI 1.8.4;
  • I've also tried on Ubuntu 14.04 Server and received identical errors.
Any help to figure out what's wrong would be greatly appreciated!

Cheers, Tom
Attached Files
File Type: gz make1.log.tar.gz (7.3 KB, 4 views)
mesh-monkey is offline   Reply With Quote

Old   January 7, 2016, 13:28
Default RapidCFD compile issues
  #7
aee
Member
 
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 30
Rep Power: 6
aee is on a distinguished road
I have exactly the same problem compiling RapidCFD - both the Cuda math issue and the compiler looking for the -l* entries in usr/bin/ld (which does not exist). I haven't figured out where (or even if) that path is set in the bashrc or .sh files. Why would it look in usr/bin if the WM and FOAM env variables have been set?

I tried installing in opt/ and in $HOME. I tried setting the MPI to SYSTEMOPENMPI from OPENMPI (in line with my OpenFoam installations). I tried setting all the env variables directly. Now I am going to try giving up until someone else figures this out.





aee is offline   Reply With Quote

Old   January 9, 2016, 20:28
Default RapidCFD compilation Fix
  #8
New Member
 
Tom
Join Date: Dec 2015
Location: Melbourne, Australia
Posts: 5
Rep Power: 2
mesh-monkey is on a distinguished road
Hello again,

I think I rectified the issue, courtesy of aokomoriuta (via google translate) that led me to some old install instructions on the github wiki.

Quote:
"However in CUDA headers there is a name clash between Foam and std math functions. Temporary remedy is to modify CUDA math_functions.h header to explicitly call std math functions. Modifications for CUDA 6.5 are available here.
After applying their patch from the link above (attached), the program compiled without a hitch. I will come back with solution results hopefully soon!

I definitely downloaded the latest code as .zip, so not sure if/why any code changes didn't come through or work, but I will log a bug on the github.

For those looking at this in the future (and previous me), the math_functions.h issues were what was causing the cannot find -l* errors. Essentially the error happened during the build of one of the first libso libraries (fileFormats) and once that failed, all of the others after that also failed - hence why at the end of the file there is a long list.

Cheers, Tom
Attached Files
File Type: gz CUDA-math-functions.patch.tar.gz (1.4 KB, 14 views)
mesh-monkey is offline   Reply With Quote

Old   January 11, 2016, 14:24
Default Cuda math functions patch - CUDA 7.5?
  #9
aee
Member
 
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 30
Rep Power: 6
aee is on a distinguished road
I have Cuda 7.5 installed. I suspect the patch might work with Cuda 7.5, but I am not going to be the first to try. Would it make sense to install Cuda 6.5 into the Thirdparty-dev folder, patch and compile it there, and then set the Cuda env variables to that?
aee is offline   Reply With Quote

Old   January 11, 2016, 19:35
Default
  #10
New Member
 
Tom
Join Date: Dec 2015
Location: Melbourne, Australia
Posts: 5
Rep Power: 2
mesh-monkey is on a distinguished road
Hi aee,

I don't see any issue doing as you described to install cuda 6.5. You could also use the default install directory (/usr/local) and change the sybomolic link (/usr/local/cuda) to point to cuda-6.5 rather than cuda-7.5.
As a note, I did find 6.5 to be a bit fussy with what gcc/g++ compiler version I used - had to be 4.7.

If you do feel like trying cuda 7.5, you could always just make a backup of the math_function file and try that way. FYI I think the patch would be applied to the math_functions.hpp file.

Another thing to note is what cuda architecture you build rapidcfd for - you will have to set this according to what is compatible with your gpu. By default it is set to sm_30 but I had to change this to sm_20 to make it work with the Q6000.

Tom
mesh-monkey is offline   Reply With Quote

Old   January 13, 2016, 13:23
Default Cuda 7.5 math_functions.hpp patch
  #11
aee
Member
 
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 30
Rep Power: 6
aee is on a distinguished road
I decided to try to use Cuda 7.5. The old math_functions.patch file for cuda 6.5 won't work, because the math functions targeted were moved to math_functions.hpp from Cuda 7 on, and the locations (line numbers) in the file are different. I edited the cuda 7.5 math_functions.hpp by hand based on a line-by-line read of the old patch file and then used #diff -c to build the attached patch file.
**** update Jan 14, 2016 ***
updated patch file with corrected version (was missing a "::erfc" that is in the old patch)
***** end update ***
The equivalent cuda 6.5 patch was applied to math_function.h starting at line 14273. In Cuda 7.5, the applicable section is found in math_functions.hpp starting at line 3377 (i.e., in the "all platforms" section).

Usage:
math_functions.hpp is found in /usr/local/cuda-7.5/targets/x86_64-linux/include/ (default install) or wherever you installed Cuda 7.5.

If in the default location, you will need root or sudo access for the following steps:
1) Place the patch file in the same directory.
2) Copy the original math_functions.hpp to math_functions.hpp.org and/or move it someplace safe.
2) execute
patch < Cuda7.5_math_functions.hpp.patch

Remaining issue:
Even though my Cuda 7.5 installation is native from the Ubuntu software center, and my environment variables are set according to the instructions, deviceQuery doesn't detect my Titan Black card.

The card appears properly set up...

lspci | grep -i nvidia

gives

02:00.0 VGA compatible controller: NVIDIA Corporation GK110B [GeForce GTX Titan Black] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1)

Perhaps there was a conflict between existing drivers and Cuda that I didn't see.
Attached Files
File Type: patch Cuda7.5_math_functions.hpp.patch (6.0 KB, 22 views)

Last edited by aee; January 14, 2016 at 13:07. Reason: updated patch file
aee is offline   Reply With Quote

Old   January 13, 2016, 18:20
Default
  #12
aee
Member
 
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 30
Rep Power: 6
aee is on a distinguished road
The issue with deviceQuery not seeing the card cleared up after a reboot. Of all things...

The compiler is still looking in usr/bin. I have to fix the openmpi-1.10.0 configuration issue (I am using Thirdparty-dev from the current openFoam). Maybe that will fix it.

Last edited by aee; January 13, 2016 at 20:39. Reason: compiler finished with errors
aee is offline   Reply With Quote

Old   January 14, 2016, 14:58
Default compiling and installing experiences
  #13
aee
Member
 
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 30
Rep Power: 6
aee is on a distinguished road
I was finally able to compile rapidCFD with the newly patched cuda7.5 math_function.hpp -- after cleaning out everyting in RapidCFD-dev and cloning the git, sourcing etc/bashrc, exporting WM_NCOMPPROCS=12 and running ./Allwmake. A detailed readme.txt is attached.

I can run small and large interFoam cases, after adjusting the case files for unsupported models, solvers and preconditoners.
e.g., for on the OF2.3.1 tutorial LES/nozzleFlow2D
Unknown LESdelta type smooth

Valid LESdelta types are :
1(vanDriest) and
4
(
IDDESDelta
Prandtl
cubeRootVol
maxDeltaxyz
)


From function LESdelta::New(const fvMesh&, const dictionary&)
in file LESdelta/LESdelta.C at line 121.

FOAM exiting

Even for this small case, one CPU core is maxed out, while nvidia-smi shows interfoam is using very little of the GPU. I may have to play with the case files to see what actually runs on the gpu. I imagine it doesn't help that I am using the GPU for display as well.

~/RapidCFD/andrew-dev$ nvidia-smi
Thu Jan 14 10:48:48 2016
+------------------------------------------------------+
| NVIDIA-SMI 352.68 Driver Version: 352.68 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+================= =====+======================|
| 0 GeForce GTX TIT... Off | 0000:02:00.0 On | N/A |
| 41% 67C P2 97W / 250W | 615MiB / 6143MiB | 47% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|================================================= ============================|
| 0 647 C interFoam 226MiB |
| 0 1400 G /usr/bin/X 236MiB |
| 0 2392 G compiz 131MiB |
| 0 23280 G /usr/lib/firefox/firefox 2MiB |
+-----------------------------------------------------------------------------+


On a larger mesh (1.4M cells), it uses more gpu RAM, but the cpu core remains maxed out.
~/RapidCFD/andrew-dev$ nvidia-smi
.
.
.

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|================================================= ============================|
| 0 969 C interFoam 1668MiB |
| 0 1400 G /usr/bin/X 225MiB |
| 0 2392 G compiz 131MiB |
| 0 23280 G /usr/lib/firefox/firefox 2MiB |
+-----------------------------------------------------------------------------+


Oddly, a laminar simulation seems to run at the same speed as a LES simulation. I assume that the limiting factor is something being done on the cpu.
Attached Files
File Type: txt RapidCFD-Cuda75-math_functions_patch.readme.txt (2.9 KB, 19 views)

Last edited by aee; January 16, 2016 at 13:40. Reason: clarify
aee is offline   Reply With Quote

Old   April 24, 2016, 21:24
Default what should I do if I have not root privilege to patch cuda math_function
  #14
New Member
 
Pin Lyu
Join Date: Mar 2016
Location: Harbin
Posts: 6
Rep Power: 2
tripodtwostep is on a distinguished road
Hello,
If I don't have root privilege of the cluster, is there any quick method to patch cuda math_function?
If not, should I compile cuda by myself?

Pin

Quote:
Originally Posted by aee View Post
Usage:
math_functions.hpp is found in /usr/local/cuda-7.5/targets/x86_64-linux/include/ (default install) or wherever you installed Cuda 7.5.

If in the default location, you will need root or sudo access for the following steps:
1) Place the patch file in the same directory.
2) Copy the original math_functions.hpp to math_functions.hpp.org and/or move it someplace safe.
2) execute
patch < Cuda7.5_math_functions.hpp.patch
tripodtwostep is offline   Reply With Quote

Old   April 25, 2016, 01:48
Default
  #15
aee
Member
 
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 30
Rep Power: 6
aee is on a distinguished road
That might depend on your cluster setup. If the clients boot off the workstation on which you have a user account, and if you can set the environment for them, you should be able to do it. You'd have to install Cuda for your account, patch it and propogate it (set the env to point to your Cuda installation). I haven't tried RapidCFD on a cluster though, so you might ask around for more detailed help.
aee is offline   Reply With Quote

Old   April 25, 2016, 14:21
Default
  #16
New Member
 
Pin Lyu
Join Date: Mar 2016
Location: Harbin
Posts: 6
Rep Power: 2
tripodtwostep is on a distinguished road
Firstly, I think patch cuda in its default directory /usr/local/cuda is not suitable for me because I have no root privilege in the remote cluster.
Secondly, after some investigations, I think it might be possible to install cuda toolkit in my own account. (see part 4.5 and installation option "--toolkitpath=<path>", http://docs.nvidia.com/cuda/cuda-get...#axzz46raIptgZ )
Thirdly, I read your RapidCFD installation note. It is very useful. I was also wondering which version of thirdparty package should I use before reading your note.
Thank you.

Quote:
Originally Posted by aee View Post
That might depend on your cluster setup. If the clients boot off the workstation on which you have a user account, and if you can set the environment for them, you should be able to do it. You'd have to install Cuda for your account, patch it and propogate it (set the env to point to your Cuda installation).
tripodtwostep is offline   Reply With Quote

Old   April 25, 2016, 17:02
Default
  #17
aee
Member
 
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 30
Rep Power: 6
aee is on a distinguished road
I think I used the latest 3.0, but since it is only Paraview, you may not need to do anything other than point your env to it.
aee is offline   Reply With Quote

Old   April 25, 2016, 17:06
Default
  #18
aee
Member
 
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 30
Rep Power: 6
aee is on a distinguished road
I have a bunch of Aliases which set the bash env for of231, of30, of30x, mfix or rapidcfd. Once again, I don't know how that would work in your cluster, since it may be expecting each node to have the same software environment and to have the software installed local to the node (in which case, you are probably out of luck).
aee is offline   Reply With Quote

Old   April 26, 2016, 08:36
Default
  #19
New Member
 
Pin Lyu
Join Date: Mar 2016
Location: Harbin
Posts: 6
Rep Power: 2
tripodtwostep is on a distinguished road
I installed the cuda toolkit somewhere in my home folder (use --silent --toolkit --toolkitpath=$MYCUDAHOME options), patched cuda 7.5 math_function.hpp, compiled RapidCFD-dev. Now I have solvers such as pisoFoam in ~/RapidCFD/RapidCFD-dev/platforms/linux64NvccDPOpt/bin.
However, pisoFoam crashed after "create mesh for time 0" with informations below:
---------start--------
terminate called after throwing an instance of 'thrust::system::system_error'
what(): function_attributes(): after cudaFuncGetAttributes: invalid device function
---------end---------
I looked up this error info in internet, somebody solved it after install cuda driver. For my case, I could compile cuda samples. After executing cuda deviceQuery, I get a positive answer:
CUDA Driver=CUDART, CUDA Driver Version=7.5, CUDA Runtime Version=7.5, Device0=Quadro 4000.

Could anybody give me a hint what is wrong? (I ran the cavity case in pisofoam-ras tutorials folder, used of3.0 blockMesh. I didn't do other modifications on the dictionaries. Should I change fvSolotion PCG to cudaCG like foam-extend?)

Quote:
Originally Posted by aee View Post
I have a bunch of Aliases which set the bash env for of231, of30, of30x, mfix or rapidcfd. Once again, I don't know how that would work in your cluster, since it may be expecting each node to have the same software environment and to have the software installed local to the node (in which case, you are probably out of luck).
tripodtwostep is offline   Reply With Quote

Old   May 21, 2016, 22:45
Default texture object vs texture references compute capability 2.0 vs 3.0
  #20
New Member
 
Tom
Join Date: Dec 2015
Location: Melbourne, Australia
Posts: 5
Rep Power: 2
mesh-monkey is on a distinguished road
I get a similar issue - I believe it is related to the cuda compute capability (CC, CUDA_ARCH or sm_xx). RapidCFD by default is designed to run using CC 3.0 - mostly cards with Kepler chips (see listing here). Your Quadro 4000, like my Quadro 6000, has a Fermi chip and is CC 2.0.

There was a previous pull request to adjust the code to make it work, but appears to have been abandoned.

The issue comes down to the difference in how CC 3.0 handles textures objects. The file in question is src/OpenFOAM/containers/Lists/gpuList/textures.H

I have tried unsuccessfully to modify the code to make it work, however I'm well over my head programming-wise.
Any cuda-programming geniuses around that could help?

Thanks, Tom

Quote:
Originally Posted by tripodtwostep View Post
I installed the cuda toolkit somewhere in my home folder (use --silent --toolkit --toolkitpath=$MYCUDAHOME options), patched cuda 7.5 math_function.hpp, compiled RapidCFD-dev. Now I have solvers such as pisoFoam in ~/RapidCFD/RapidCFD-dev/platforms/linux64NvccDPOpt/bin.
However, pisoFoam crashed after "create mesh for time 0" with informations below:
---------start--------
terminate called after throwing an instance of 'thrust::system::system_error'
what(): function_attributes(): after cudaFuncGetAttributes: invalid device function
---------end---------
I looked up this error info in internet, somebody solved it after install cuda driver. For my case, I could compile cuda samples. After executing cuda deviceQuery, I get a positive answer:
CUDA Driver=CUDART, CUDA Driver Version=7.5, CUDA Runtime Version=7.5, Device0=Quadro 4000.

Could anybody give me a hint what is wrong? (I ran the cavity case in pisofoam-ras tutorials folder, used of3.0 blockMesh. I didn't do other modifications on the dictionaries. Should I change fvSolotion PCG to cudaCG like foam-extend?)
mesh-monkey is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 01:38.