CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Installation

extend-3.0 cuda installation

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree8Likes

Reply
 
LinkBack Thread Tools Display Modes
Old   January 22, 2014, 09:40
Default extend-3.0 cuda installation
  #1
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
Hi

I tried to install cuda solvers for ext3.0. What i did ->

- installed 3.0 without problems
- installed nvidia driver and cuda 5.5
- then i changed in the prefs.sh ->
# System installed CUDA
export CUDA_SYSTEM=1
export CUDA_DIR=/usr/bin/cuda-5.5
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

- then i tried to compile the cudaSolvers

but i get a message like this

Found nvcc but \$CUDA_IGNORE set -- not enabling CUDA support.

Or where is this flag? $CUDA_IGNORE

OK found it in the /Allwmake.firstInstall

thx for help me out

Br

Christian
Kojote is offline   Reply With Quote

Old   January 23, 2014, 03:20
Default
  #2
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
Hi

Something is wrong with my setup.
I cannot compile Cuda.
Did someone the installation successfully?


Br

Christian
Kojote is offline   Reply With Quote

Old   January 23, 2014, 06:20
Default
  #3
New Member
 
Dominik Christ
Join Date: Mar 2009
Posts: 28
Rep Power: 8
dominik_christ is on a distinguished road
Hi Christian,
the installation works fine on my machine.
Can you please post more information?

* Is the Cuda installation itself working, can you compile and run the Cuda tutorial cases?
* Where is your Cuda installation located? /usr/bin/cuda seems a little strange to me. If you installed from .deb package, Cuda is located in /usr/local/cuda/ . If you type "which nvcc" and "locate bin/nvcc" what output do you get?
* How do you try to compile the Cuda solvers in foam-extend? Which commands are you running?
* What is the exact error message that you get?

Best regards,
Dominik
dominik_christ is offline   Reply With Quote

Old   January 23, 2014, 06:54
Default
  #4
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
Hi

Thx for answer. I am at work in the moment. Will get the infos asap. But good to hear that cuda is possible.

Br

Christian
Kojote is offline   Reply With Quote

Old   January 23, 2014, 13:13
Smile
  #5
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
* Is the Cuda installation itself working, can you compile and run the Cuda tutorial cases?
yes make is working in samples

* Where is your Cuda installation located? /usr/bin/cuda seems a little strange to me. If you installed from .deb package, Cuda is located in /usr/local/cuda/ . If you type "which nvcc" and "locate bin/nvcc" what output do you get? you are right, i posted wrong path

which nvcc ->
/usr/local/cuda/bin/nvcc

prefs.sh:
export CUDA_SYSTEM=1
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

settings.sh
# CUDA if available
# ~~~~~~~~~~~~~~~~~
[ -z "$CUDA_SYSTEM" ] && [ -e /usr/local/cuda/bin/nvcc ] && {
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
}



* How do you try to compile the Cuda solvers in foam-extend? Which commands are you running?

./Allwmake in the cudaSolvers

* What is the exact error message that you get?

Found nvcc but $CUDA_IGNORE set -- not enabling CUDA support

but yesterday the flag was ok and then there were errors, too.

what can i do?

[/QUOTE]

many thanks for help

christian
Kojote is offline   Reply With Quote

Old   January 27, 2014, 07:31
Default
  #6
New Member
 
Dominik Christ
Join Date: Mar 2009
Posts: 28
Rep Power: 8
dominik_christ is on a distinguished road
Hi Christian,

When I search all subdirectories for occurrence of CUDA_IGNORE ("grep -r CUDA_IGNORE"), I find only two files: Allwmake.firstInstall and src/cudaSolvers/Allwmake.

Only Allwmake.firstInstall sets this environment variable, so it appears to me that you are running ./Allwmake.firstInstall before going into src/cudaSolvers and running ./Allwmake there.

When setting CUDA_IGNORE, the Allwmake.firstInstall script posts a message:

---
$CUDA_ARCH is required by nvcc compiler but not set.
Check section '-gpu-architecture' in 'man nvcc' for details.

Proceed without compiling cudaSolvers? [Y/n]
---

So the remedy would be to set CUDA_ARCH as appropriate for your hardware. I have K20 Tesla cards here and "export CUDA_ARCH=sm20" is the setting that I use.

To automate setting of CUDA_ARCH, you can use etc/presh.sh-EXAMPLE as a template to make a customized etc/prefs.sh with your local settings.

Hope this helps,
Dominik
dominik_christ is offline   Reply With Quote

Old   January 27, 2014, 08:10
Default
  #7
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
Quote:
Originally Posted by dominik_christ View Post
Hi Christian,

When I search all subdirectories for occurrence of CUDA_IGNORE ("grep -r CUDA_IGNORE"), I find only two files: Allwmake.firstInstall and src/cudaSolvers/Allwmake.

Only Allwmake.firstInstall sets this environment variable, so it appears to me that you are running ./Allwmake.firstInstall before going into src/cudaSolvers and running ./Allwmake there.

When setting CUDA_IGNORE, the Allwmake.firstInstall script posts a message:

---
$CUDA_ARCH is required by nvcc compiler but not set.
Check section '-gpu-architecture' in 'man nvcc' for details.

Proceed without compiling cudaSolvers? [Y/n]
---

So the remedy would be to set CUDA_ARCH as appropriate for your hardware. I have K20 Tesla cards here and "export CUDA_ARCH=sm20" is the setting that I use.

To automate setting of CUDA_ARCH, you can use etc/presh.sh-EXAMPLE as a template to make a customized etc/prefs.sh with your local settings.

Hope this helps,
Dominik

Hi Dominik,

Thanks for answer again.
i did all from scratch again. No my cuda settings are 100% OK. Cuda tutorils are compiled. When i run ./Allwmake.firstinstall the scipt sais something like "found nvcc, cuda is compiled by default" I adjusted the settings.sh and prefs.sh before that. While comping OpenFoam the massage with the flag is in the log file.
Cuda will be not compiled. When i do "wmake libso" in the cudaSolver folder then "make has no target" (i use the german ubuntu so i don't know the massage in english).

By the way what are your experience with cuda.

Br

Christian
Kojote is offline   Reply With Quote

Old   January 27, 2014, 14:32
Default
  #8
New Member
 
Dominik Christ
Join Date: Mar 2009
Posts: 28
Rep Power: 8
dominik_christ is on a distinguished road
Hi Christian,

When you post, please provide exactly the steps you are taking with verbatim commands, post log files (as attachments if possible) and error messages as they are because it is very difficult to identify anything from a written summary (and I cannot tell if you e.g. forgot to source the etc/bashrc file at some point, or something like that)
I can live with German error messages.

Regarding my experience with cuda and foam-extend:
1) It works ...

2) ... except for coupled boundary conditions, like cyclic, processor and ggi. These need to be updated in every iteration that a solver (e.g. BiCG) is doing for a transport equation. This would require:
a) Uploading the intermediate data back to CPU, update the cyclic BCs and load the changed values back to the GPU (=very time-consuming)
or
b) Implementing all coupled boundary conditions with GPU code.

So this is tricky by principle. I believe that implementing general coupled boundary condition support for GPU is the major open task for the future.

3) I have not yet made a valid comparison, that is between a good CPU setup and a good GPU setup. When time allows, there will be something to show at the workshop.

Best regards,
Dominik
dominik_christ is offline   Reply With Quote

Old   January 28, 2014, 03:44
Default
  #9
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
Hi Dominik

OK. I have to reinstall a computer next week. Then i will write out all logs etc. I would be very nice if you could have a look at them if i failure again.

Some more questions:

Do you have to decompose a cuda case, or is this be done by the hardware?
And is it possible to run a case with lagrangian particle on cuda?

!!! Many thanks for your help !!!!

Br

Christian
Kojote is offline   Reply With Quote

Old   January 28, 2014, 06:23
Default
  #10
Member
 
xxxxx
Join Date: Feb 2013
Posts: 34
Rep Power: 4
xxxx is on a distinguished road
Hello!

I'm trying to install cudaSolvers too, but I found the same problems as Kojote.
I managed to compile foam-ext-3.0 with cuda ( "found nvcc, cuda is compiled by default") but when I try to run ./Allwmake in cudaSolvers dir I get:

linux@linux-Precision-T7600:~/foam/linux-3.0/src/cudaSolvers$ ./Allwmake
Found nvcc but $CUDA_IGNORE set -- not enabling CUDA support.

and if I try wmake libso instead of ./Allwmake I get:

linux@linux-Precision-T7600:~/foam/linux-3.0/src/cudaSolvers$ wmake libso
wmakeLnInclude: linking include files to ./lnInclude
make: *** No rule to make target `cudaCG/cgDiag.dep', needed by `Make/linux64Gcc46DPOpt/dependencies'. Stop.

How did you manage to make it work, dominik_christ?
Thanks in advance
__________________
Federico
xxxx is offline   Reply With Quote

Old   January 28, 2014, 11:14
Default
  #11
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
Hi

Now i did it all again on another computer. Ubuntu 12.04 64bit / CUDA 5.5.
Nvidia driver (319.37) from the cuda run file. My card is a GeForce GTX 650 Ti and from what i know i have to use sm_30.
And i have the same situation like Federico described.

I did -> export CUDA_IGNORE=0, with the Allwmake.firstInstall the flag is not written.

And with "./Allwmake" in the cudaSolver folder or "wmake libso" getting this:

Found nvcc -- enabling CUDA support.
make: *** Keine Regel vorhanden, um das Target »cudaCG/cgDiag.dep«,
benötigt von »Make/linux64Gcc46DPOpt/dependencies«, zu erstellen. Schluss.

Br

Christian
mm.abdollahzadeh likes this.
Kojote is offline   Reply With Quote

Old   January 29, 2014, 05:53
Default
  #12
Member
 
xxxxx
Join Date: Feb 2013
Posts: 34
Rep Power: 4
xxxx is on a distinguished road
Hello Kojote.

Since it seems that we are the only people interested in CUDA compiling on this forum, would you like to keep in contact? Do you have any news about CUDA compiling?
__________________
Federico
xxxx is offline   Reply With Quote

Old   January 29, 2014, 05:57
Default
  #13
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
Hi,

Yes to keep in touch, no to the news.

Br

Christian
Kojote is offline   Reply With Quote

Old   January 31, 2014, 11:45
Default
  #14
New Member
 
Dominik Christ
Join Date: Mar 2009
Posts: 28
Rep Power: 8
dominik_christ is on a distinguished road
Dear Christian and Federico,

I could reproduce some of the issues and found some solutions:
1) The logic in Allwmake must be inverted: It tests for [ -z $CUDA_IGNORE], which is for the absence of CUDA_IGNORE. The logic has to be: If IGNORE is absent, do compile. The "then" and "else" part must therefore be swapped to:
if [ -z $CUDA_IGNORE ]
then
echo "Found nvcc -- enabling CUDA support."
wmake libso
else
echo "Found nvcc but \$CUDA_IGNORE set -- not enabling CUDA support."
fi

2) "wmake libso" should work independently from this. The compilation error that Christian is seeing comes from a broken dependency. To re-create dependencies, run "wclean" before "wmake libso" (but please read the points below before that).

3) The cufflink headers in "*.cu" and "include/*.H" in directory "cudaSolver/" have become messed up. For a quick fix, add a closing comment "*/" at the end of the descriptive header before the actual code for these files.

4) If you get any error with "gpu-architecture", then CUDA_ARCH is not set. As mentioned in the post above, you can permanently set it in etc/prefs.sh (use etc/prefs.sh-EXAMPLE as template). My configuration is:

export CUDA_SYSTEM=1
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib64
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

With these changes, I can compile cudaSolvers with the latest version of the git repository version. I will make a bugfix from above points and submit it to the foam-extend project.

Hope this solves the issues on your machines as well.

Best regards,
Dominik
dominik_christ is offline   Reply With Quote

Old   January 31, 2014, 12:10
Default
  #15
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
hi dominik

wow will try asap

manythx for your help and for your work

christian
Kojote is offline   Reply With Quote

Old   January 31, 2014, 14:35
Default
  #16
New Member
 
Krishna
Join Date: Jun 2012
Posts: 28
Blog Entries: 1
Rep Power: 5
rkc.cfd is on a distinguished road
Quote:
Originally Posted by dominik_christ View Post
Dear Christian and Federico,

The cufflink headers in "*.cu" and "include/*.H" in directory "cudaSolver/" have become messed up. For a quick fix, add a closing comment "*/" at the end of the descriptive header before the actual code for these files.

Dominik
That was funny when I found out that somebody really wanted us to read the code

Also when I compiled it was asking for the cusp definitions
Code:
lnInclude/cudaTypes.H:53:29: error: cusp/coo_matrix.h: No such file or directory
So I downloaded from the git repository and added the directory to make file options.

Thanks to all of you on this thread.
chery1986 likes this.
rkc.cfd is offline   Reply With Quote

Old   February 1, 2014, 06:39
Default
  #17
Member
 
xxxxx
Join Date: Feb 2013
Posts: 34
Rep Power: 4
xxxx is on a distinguished road
Thank you dominik_christ,

Actually I already fixed this:

Quote:
Originally Posted by dominik_christ View Post

3) The cufflink headers in "*.cu" and "include/*.H" in directory "cudaSolver/" have become messed up. For a quick fix, add a closing comment "*/" at the end of the descriptive header before the actual code for these files.
And this:

Quote:
Originally Posted by dominik_christ View Post

4) If you get any error with "gpu-architecture", then CUDA_ARCH is not set. As mentioned in the post above, you can permanently set it in etc/prefs.sh (use etc/prefs.sh-EXAMPLE as template). My configuration is:

export CUDA_SYSTEM=1
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib64
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20
with CUDA_ARCH=sm_30 (my machine!) before to try to compile with wmake libso but I still can't compile cudaSolvers.

Have you tested it? Do they work now? (the gpu solvers I mean)
__________________
Federico
xxxx is offline   Reply With Quote

Old   February 1, 2014, 12:44
Default
  #18
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
Hi

good to know that you were able to compile

Quote:
Originally Posted by rkc.cfd View Post
That was funny when I found out that somebody really wanted us to read the code

Also when I compiled it was asking for the cusp definitions
Code:
lnInclude/cudaTypes.H:53:29: error: cusp/coo_matrix.h: No such file or directory
So I downloaded from the git repository and added the directory to make file options.

Thanks to all of you on this thread.
could you upload your cuda solver folder here or somewhere else

thx

christian
Kojote is offline   Reply With Quote

Old   February 3, 2014, 03:18
Default
  #19
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 5
Kojote is on a distinguished road
Hi

I'm still running in trouble. I did the fix for the headers and the script. After "wclean" and "wmake libso" i still get this.

Making dependency list for source file cudaSolver/cudaSolver.C
make: *** Keine Regel vorhanden, um das Target »cudaCG/cgDiag.dep«,
benötigt von »Make/linux64Gcc46DPOpt/dependencies«, zu erstellen. Schluss.

Br

Christian
Kojote is offline   Reply With Quote

Old   February 3, 2014, 23:11
Default
  #20
New Member
 
Krishna
Join Date: Jun 2012
Posts: 28
Blog Entries: 1
Rep Power: 5
rkc.cfd is on a distinguished road
Hi,

Sorry for the late reply, some how I was not subscribed to the thread and did not get the message.

The link to my github is

https://github.com/mechysolv/openfoa...extend-3.0.git

I have downloaded the cusp solvers to

Code:
$WM_THIRD_PARTY_DIR/LocalDev
from

https://github.com/cusplibrary/cusplibrary

Hope that helps,

RKC
mm.abdollahzadeh likes this.
rkc.cfd is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
A smooth installation of foam-3.0-extend on OpenSUSE 13.1 64-bits cookcaptain OpenFOAM Installation 7 May 26, 2015 11:22
what should we do for the application of CUDA in CFD? aerosjc Main CFD Forum 6 February 25, 2013 12:01
CFX11 + Fortran compiler ? Mohan CFX 20 March 30, 2011 18:56
64bitrhel5 OF installation instructions mirko OpenFOAM Installation 2 August 12, 2008 18:07
Installation problems shellbell1999 OpenFOAM Installation 9 April 6, 2006 13:29


All times are GMT -4. The time now is 23:33.