extend-3.0 cuda installation
 

Hi

I tried to install cuda solvers for ext3.0. What i did ->

- installed 3.0 without problems
- installed nvidia driver and cuda 5.5
- then i changed in the prefs.sh ->
# System installed CUDA
export CUDA_SYSTEM=1
export CUDA_DIR=/usr/bin/cuda-5.5
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

- then i tried to compile the cudaSolvers

but i get a message like this

Found nvcc but \$CUDA_IGNORE set -- not enabling CUDA support.

Or where is this flag? $CUDA_IGNORE

OK found it in the /Allwmake.firstInstall

thx for help me out

Br

Christian

Hi

Something is wrong with my setup.
I cannot compile Cuda.
Did someone the installation successfully?


Br

Christian

Hi Christian,
the installation works fine on my machine.
Can you please post more information?

* Is the Cuda installation itself working, can you compile and run the Cuda tutorial cases?
* Where is your Cuda installation located? /usr/bin/cuda seems a little strange to me. If you installed from .deb package, Cuda is located in /usr/local/cuda/ . If you type "which nvcc" and "locate bin/nvcc" what output do you get?
* How do you try to compile the Cuda solvers in foam-extend? Which commands are you running?
* What is the exact error message that you get?

Best regards,
Dominik

Hi

Thx for answer. I am at work in the moment. Will get the infos asap. But good to hear that cuda is possible. :)

Br

Christian

* Is the Cuda installation itself working, can you compile and run the Cuda tutorial cases?
yes make is working in samples

* Where is your Cuda installation located? /usr/bin/cuda seems a little strange to me. If you installed from .deb package, Cuda is located in /usr/local/cuda/ . If you type "which nvcc" and "locate bin/nvcc" what output do you get? you are right, i posted wrong path

which nvcc ->
/usr/local/cuda/bin/nvcc

prefs.sh:
export CUDA_SYSTEM=1
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

settings.sh
# CUDA if available
# ~~~~~~~~~~~~~~~~~
[ -z "$CUDA_SYSTEM" ] && [ -e /usr/local/cuda/bin/nvcc ] && {
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
}



* How do you try to compile the Cuda solvers in foam-extend? Which commands are you running?

./Allwmake in the cudaSolvers

* What is the exact error message that you get?

Found nvcc but $CUDA_IGNORE set -- not enabling CUDA support

but yesterday the flag was ok and then there were errors, too.

what can i do?

[/QUOTE]

many thanks for help http://www.cfd-online.com/Forums/images/icons/icon7.gif

christian

Hi Christian,

When I search all subdirectories for occurrence of CUDA_IGNORE ("grep -r CUDA_IGNORE"), I find only two files: Allwmake.firstInstall and src/cudaSolvers/Allwmake.

Only Allwmake.firstInstall sets this environment variable, so it appears to me that you are running ./Allwmake.firstInstall before going into src/cudaSolvers and running ./Allwmake there.

When setting CUDA_IGNORE, the Allwmake.firstInstall script posts a message:

---
$CUDA_ARCH is required by nvcc compiler but not set.
Check section '-gpu-architecture' in 'man nvcc' for details.

Proceed without compiling cudaSolvers? [Y/n]
---

So the remedy would be to set CUDA_ARCH as appropriate for your hardware. I have K20 Tesla cards here and "export CUDA_ARCH=sm20" is the setting that I use.

To automate setting of CUDA_ARCH, you can use etc/presh.sh-EXAMPLE as a template to make a customized etc/prefs.sh with your local settings.

Hope this helps,
Dominik

Hi Dominik,

Thanks for answer again.
i did all from scratch again. No my cuda settings are 100% OK. Cuda tutorils are compiled. When i run ./Allwmake.firstinstall the scipt sais something like "found nvcc, cuda is compiled by default" I adjusted the settings.sh and prefs.sh before that. While comping OpenFoam the massage with the flag is in the log file.
Cuda will be not compiled. When i do "wmake libso" in the cudaSolver folder then "make has no target" (i use the german ubuntu so i don't know the massage in english).

By the way what are your experience with cuda.

Br

Christian

Hi Christian,

When you post, please provide exactly the steps you are taking with verbatim commands, post log files (as attachments if possible) and error messages as they are because it is very difficult to identify anything from a written summary (and I cannot tell if you e.g. forgot to source the etc/bashrc file at some point, or something like that)
I can live with German error messages. ;)

Regarding my experience with cuda and foam-extend:
1) It works :) ...

2) ... except for coupled boundary conditions, like cyclic, processor and ggi. These need to be updated in every iteration that a solver (e.g. BiCG) is doing for a transport equation. This would require:
a) Uploading the intermediate data back to CPU, update the cyclic BCs and load the changed values back to the GPU (=very time-consuming)
or
b) Implementing all coupled boundary conditions with GPU code.

So this is tricky by principle. I believe that implementing general coupled boundary condition support for GPU is the major open task for the future.

3) I have not yet made a valid comparison, that is between a good CPU setup and a good GPU setup. When time allows, there will be something to show at the workshop.

Best regards,
Dominik

Hi Dominik

OK. I have to reinstall a computer next week. Then i will write out all logs etc. I would be very nice if you could have a look at them if i failure again.

Some more questions:

Do you have to decompose a cuda case, or is this be done by the hardware?
And is it possible to run a case with lagrangian particle on cuda?

!!! Many thanks for your help !!!!

Br

Christian

Hello!

I'm trying to install cudaSolvers too, but I found the same problems as Kojote.
I managed to compile foam-ext-3.0 with cuda ( "found nvcc, cuda is compiled by default") but when I try to run ./Allwmake in cudaSolvers dir I get:

linux@linux-Precision-T7600:~/foam/linux-3.0/src/cudaSolvers$ ./Allwmake
Found nvcc but $CUDA_IGNORE set -- not enabling CUDA support.

and if I try wmake libso instead of ./Allwmake I get:

linux@linux-Precision-T7600:~/foam/linux-3.0/src/cudaSolvers$ wmake libso
wmakeLnInclude: linking include files to ./lnInclude
make: *** No rule to make target `cudaCG/cgDiag.dep', needed by `Make/linux64Gcc46DPOpt/dependencies'. Stop.

How did you manage to make it work, dominik_christ?
Thanks in advance

Hi

Now i did it all again on another computer. Ubuntu 12.04 64bit / CUDA 5.5.
Nvidia driver (319.37) from the cuda run file. My card is a GeForce GTX 650 Ti and from what i know i have to use sm_30.
And i have the same situation like Federico described.

I did -> export CUDA_IGNORE=0, with the Allwmake.firstInstall the flag is not written.

And with "./Allwmake" in the cudaSolver folder or "wmake libso" getting this:

Found nvcc -- enabling CUDA support.
make: *** Keine Regel vorhanden, um das Target »cudaCG/cgDiag.dep«,
benötigt von »Make/linux64Gcc46DPOpt/dependencies«, zu erstellen. Schluss.

Br

Christian

Hello Kojote.

Since it seems that we are the only people interested in CUDA compiling on this forum, would you like to keep in contact? Do you have any news about CUDA compiling?

Hi,

Yes to keep in touch, no to the news.

Br

Christian

Dear Christian and Federico,

I could reproduce some of the issues and found some solutions:
1) The logic in Allwmake must be inverted: It tests for [ -z $CUDA_IGNORE], which is for the absence of CUDA_IGNORE. The logic has to be: If IGNORE is absent, do compile. The "then" and "else" part must therefore be swapped to:
if [ -z $CUDA_IGNORE ]
then
echo "Found nvcc -- enabling CUDA support."
wmake libso
else
echo "Found nvcc but \$CUDA_IGNORE set -- not enabling CUDA support."
fi

2) "wmake libso" should work independently from this. The compilation error that Christian is seeing comes from a broken dependency. To re-create dependencies, run "wclean" before "wmake libso" (but please read the points below before that).

3) The cufflink headers in "*.cu" and "include/*.H" in directory "cudaSolver/" have become messed up. For a quick fix, add a closing comment "*/" at the end of the descriptive header before the actual code for these files.

4) If you get any error with "gpu-architecture", then CUDA_ARCH is not set. As mentioned in the post above, you can permanently set it in etc/prefs.sh (use etc/prefs.sh-EXAMPLE as template). My configuration is:

export CUDA_SYSTEM=1
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib64
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

With these changes, I can compile cudaSolvers with the latest version of the git repository version. I will make a bugfix from above points and submit it to the foam-extend project.

Hope this solves the issues on your machines as well.

Best regards,
Dominik

hi dominik

wow will try asap

manythx for your help and for your work

christian

That was funny when I found out that somebody really wanted us to read the code :D

Also when I compiled it was asking for the cusp definitions
So I downloaded from the git repository and added the directory to make file options.

Thanks to all of you on this thread. :)

Thank you dominik_christ,

Actually I already fixed this:

And this:

with CUDA_ARCH=sm_30 (my machine!) before to try to compile with wmake libso but I still can't compile cudaSolvers.

Have you tested it? Do they work now? (the gpu solvers I mean)

Hi

good to know that you were able to compile

could you upload your cuda solver folder here or somewhere else

thx

christian

Hi

I'm still running in trouble. I did the fix for the headers and the script. After "wclean" and "wmake libso" i still get this.

Making dependency list for source file cudaSolver/cudaSolver.C
make: *** Keine Regel vorhanden, um das Target »cudaCG/cgDiag.dep«,
benötigt von »Make/linux64Gcc46DPOpt/dependencies«, zu erstellen. Schluss.

Br

Christian

Hi,

Sorry for the late reply, some how I was not subscribed to the thread and did not get the message.

The link to my github is

https://github.com/mechysolv/openfoa...extend-3.0.git

I have downloaded the cusp solvers to

from

https://github.com/cusplibrary/cusplibrary

Hope that helps,

RKC

Fix pushed to repository
 

Hi all,

I fixed the issues mentioned in the posts above (Allwake logic, messed up header sections), tested and pushed to the repository (commit number: 55a148542d). To get the fix, run "git pull", clean and recompile.

Best regards,
Dominik

Hi

Sounds good, i will give it try. Thx again for your hard work.

Br

Christian

Hi everyone!

I am also trying to use cuda with the ext3.0.
I have a working installation of cuda (in opt/cuda), and installed 3.0.
However when I try to run the Allwmake.firstInstall script, I get the error described by Dominik :
I have in my etc/prefs.sh :
# System installed CUDA
export CUDA_SYSTEM=1
export CUDA_DIR=/opt/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

I used the CUDA_ARCH setting present in etc/presh.sh-EXAMPLE.
The device I currently use is an old Quadro FX 570.
Best regards,

Pierre

Hi Pierre,
the message pops up only if CUDA_ARCH is not set. However, you say that is specified in prefs.sh . Can you please verify that prefs.sh is actually setting the variable?

Steps:
1) Source the bashrc:
. etc/bashrc
(when you are in installation root directory, e.g. ~/foam/foam-extend-3.0/)

2) Check whether CUDA_ARCH is set:
env | grep CUDA*

Please post the output of step 2)

Best regards,
Dominik

Dear Dominik,

I'm very sorry for the late answer.
It appeared that I had an issue with my swak4foam installation. After reinstalling it I manage to run the Allwmake.firstInstall script without getting the previous error.
However I still cannot compile the cuda solver. When I run the Allmake I get the following result :
Also the result of the second step gives :
Best regards,

Pierre

Hi everyone,

I am also trying to use the cudaSolver of foam-extend-3.0.
I am quite new to openfoam, so I set WM_COMPILE_OPTION:=Debug in the foam-extend-3.0etc/bashrc because I may need to debug to understand the code better, including the cudaSolver.
I got some errors related to the nvcc options during compilation, as follows:

I an not very clear on how the whole project is compiled and where I can remove/modify those options for nvcc.

I have in my etc/prefs.sh :
export CUDA_SYSTEM=1
export CUDA_DIR=/home/zzhong/cuda_4.2
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib64
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

The GPU attached to the node is Tesla C2050.

There are no errors when compiling with WM_COMPILE_OPTION:=Opt, and the libcudaSolvers.so is generated into lib/linux64GccDPOpt.

Best regards
Zhong

Dear Zhong,
Does the compilation work for you with Opt instead of Debug?
Best regards,
Dominik

Hi Dominik,

Yes, the compilation with Opt went well without errors.

Best regards
Zhong

Dear Zhong,

now that you have the cuda libraries working, do you find any advantage running the cases with cuda?
Can you please tell us about your experience?
Thank you.

F.F.

Hi Federico,

To be honest, I haven't done any serious test yet. Sorry for that.
I will let you know once I have some results.

Actually I am currently still experimenting with cufflink library rather than the cudaSolver included in the foam-extend-3.0, because at this moment I am interested in running the GPU solver in parallel, but I am not sure if the cudaSolver in foam-extend-3.0 can run in parallel.

I am trying to figure out what domain decomposition method is used and how it is implemented in foam, obviously some information (interfaces?) are exchanged between sub-domains during each iteration of the linear solver. Any ideas?

Best Regards
Zhong

Dear Kirishna

I had installed everything successfully.
but when I try to run a case I am getting a problem.

did u have similar problem ? or do u know how to resolve it?

many thanks

best
mahdi

Hi Mahdi,

This looks like an error when you have not installed the Nvidia drivers properly. I have no experience in installing Cuda. Check if you have installed nvidia cuda drivers properly.

http://forums.udacity.com/questions/...ror-in-maincpp

Cheers

Dear Krishna

I had solved that. problem was related to gcc version when compiling cuda solvers. I was using gcc 4.6 for installing cuda driver, but i didn't switch afterwards to gcc 4.8.

however I have another question? My graphic card is gforce 8400 Gs. is there any reason that I was not able to use another cuda arch like sm_20? just sm_10 worked in my case.

and Im wondering that is it possible to select some how the number of the gpus or run in parallel with gpus? since i guess foam extend is using cufftlink-library it should be possible, but i was not able to test even the cufftlink-library sepratly to test it as I guess my ubuntu and nvidia driver are more recent than the one it was tested for.


best
mahdi

Hi,

I have not used multi GPU option, currently I am using only CPU. I have google and found this, hope it should help.

https://code.google.com/p/cufflink-l...ingStartedPage

Cheers

Hi All

After a long time i am now able to run fe31 with cuda. Now its really easy. Install cuda and put the cusp folder into the cuda/include folder.
Setup the cuda paths in of. Done.

Should we start a thread "How to use cuda?"

Thx to all

Christian

Dear colleagues,

did anyone succeed in with compilation of cudaSolvers for foam-extend-3.1?

During compilation I get that there's no rule for 'cudaCG/cgDiag.dep'. Indeed, there's no such file, there's only cgDiag.cu.

CUDA variables in prefs.sh are set properly.


Best regards,
Aleksey.

I've found out how to compile cudaSolvers. It appeared that there's no nvcc rules for gcc46. nvcc rules exist only for gcc. So I've compiled foam by system compiler.

1. I've downloaded cusplibrary and modified options file as it was mentioned before in this thread.

2. In all source files of cudaSolvers I've changed "#include <cusp/blas.h>" on "#include <cusp/blas/blas.h>"

3. I changed path for cuda libs from lib to lib64 folder in prefs.sh

After that everything compiled OK.

Best regards,
Aleksey.

Hello, Aleksey_R,
Can you teach me how can I compile the foam-extend by system compiler?
I follow yours 1,2,3 steps,but I still cannot figure out what the problem is.
when I use"wmake libso" to compile the cudaSolver, I got some mistake，it shows：
/usr/local/cuda/include/cusp/system/cuda/detail/multiply/coo_flat_spmv.h:40: error: ’threadIdx’€™ was not declared in this scope
/usr/local/cuda/include/cusp/system/cuda/detail/graph/b40c/graph/bfs/enactor_hybrid.cuh:496: error: expected primary-expression before ‘>’ token
...
desire for your reply.

Best regards,
zhihongliu.

Hello!
What version of CUDA do you use? I had successful experience only with v. 6.0.

Hi Everyone,

I successfully installed foam-ext on my Archlinux distro. I also installed the nvidia driver and Cuda (test samples work). All cuda variables in the prefs.sh file are correctly set. Cusp libraries seem to be well installed in the include cuda directory.

It seems that the only remaining thing to do to install cudaSolvers is running ./Allwmake in the cudaSolvers directory.

But here is the error message :

nvcc fatal : redefinition of argument 'machine'
nvcc fatal : redefinition of argument 'machine'
nvcc fatal : redefinition of argument 'machine'

Does anybody get this error message at compilation of cudaSolvers ?

Regards
jipai