CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

CUDA solvers of foam-extend-3.0

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree5Likes
  • 1 Post By bigphil
  • 1 Post By derkermit
  • 1 Post By derkermit
  • 2 Post By derkermit

Reply
 
LinkBack Thread Tools Display Modes
Old   May 1, 2014, 18:31
Question CUDA solvers of foam-extend-3.0
  #1
Member
 
A. Bernath
Join Date: Jun 2011
Location: Karlsruhe, Germany
Posts: 39
Rep Power: 7
derkermit is on a distinguished road
Hello,
has anybody suceeded up to now in using the new cuda solvers of foam-extend-3.0?

As an example, I changed in a tutorial case from PCG to cudaCG but the solver then complains about not knowing this type of solver. Also the printed list does not contain the cuda solvers. Before wasting too much time I thought it could be helpful to ask here for people which have or, even better, had the same problem.

Would be nice to get some help and share experiences!

Bye, Alex
derkermit is offline   Reply With Quote

Old   May 8, 2014, 03:41
Default
  #2
Member
 
A. Bernath
Join Date: Jun 2011
Location: Karlsruhe, Germany
Posts: 39
Rep Power: 7
derkermit is on a distinguished road
So nobody has used that solvers until now?
derkermit is offline   Reply With Quote

Old   May 8, 2014, 04:23
Default
  #3
Senior Member
 
bigphil's Avatar
 
Philip Cardiff
Join Date: Mar 2009
Location: Dublin,Ireland
Posts: 622
Rep Power: 22
bigphil will become famous soon enoughbigphil will become famous soon enough
Hi,

I don't think the cuda library is loaded by default in most solvers, so you would need to add the following to system/controlDict (on Linux):
Code:
libs                  ("libcudaSolvers.so");
or (on Mac)
Code:
libs                  ("libcudaSolvers.dylib");
Also, this is assuming that the cuda library was actually compiled; it is only compiled if your machine has a cuda capable device. You can check if it compiled by:
Code:
cd $FOAM_SRC/cudaSolvers
./Allwmake
Philip
mm.abdollahzadeh likes this.
bigphil is offline   Reply With Quote

Old   May 10, 2014, 16:54
Default
  #4
Member
 
A. Bernath
Join Date: Jun 2011
Location: Karlsruhe, Germany
Posts: 39
Rep Power: 7
derkermit is on a distinguished road
Hi Philip,
thanks for the hint. I got one step further but it crashes right when the cuda solver should start with the calculation. The error message reads

Code:
terminate called after throwing an instance of 'thrust::system::detail::bad_alloc'
  what():  std::bad_alloc: unknown error
I tried one rhoPimpleFoam tutorial first with the same error and changed to a smaller elasticSolidFoam tutorial case (bimaterialPlate) because I thought the memory size could be an issue (Geforce GTX570 with 1280MB) but that didn't make a change.

fvSolution looks like this:

Code:
    U
    {
        solver          cudaCG;
        preconditioner  diagonal;
        tolerance       1e-07;
        relTol          0.1;
    }
Do you have any idea, what is causing this error? Do I have to decompose the case in order to run the cuda solvers? Compilation of the cuda solvers finished with no errors.
I haven't checked if the cuda examples provided by nvidia work so that will be my next step.

Thanks, Alex
derkermit is offline   Reply With Quote

Old   May 11, 2014, 13:57
Default
  #5
Member
 
A. Bernath
Join Date: Jun 2011
Location: Karlsruhe, Germany
Posts: 39
Rep Power: 7
derkermit is on a distinguished road
Godd news, CUDA works now. I had the wrong driver version installed and after downgrading to the one which came with the cuda installer, everything works fine.

I tested some tutorial cases but actually it isn't faster at all when using the GPU. I checked the GPU utilization level and I could not get that value above 10% so I guess if I could optimize this, the calculation will be faster. But how to achieve this?

Is there anyone who actually has accelerated calculations with this solvers?

Do I have to tell the solver how many cuda cores should be used?
mm.abdollahzadeh likes this.
derkermit is offline   Reply With Quote

Old   May 12, 2014, 04:14
Default
  #6
Senior Member
 
bigphil's Avatar
 
Philip Cardiff
Join Date: Mar 2009
Location: Dublin,Ireland
Posts: 622
Rep Power: 22
bigphil will become famous soon enoughbigphil will become famous soon enough
Quote:
Originally Posted by derkermit View Post
Godd news, CUDA works now. I had the wrong driver version installed and after downgrading to the one which came with the cuda installer, everything works fine.

I tested some tutorial cases but actually it isn't faster at all when using the GPU. I checked the GPU utilization level and I could not get that value above 10% so I guess if I could optimize this, the calculation will be faster. But how to achieve this?

Is there anyone who actually has accelerated calculations with this solvers?

Do I have to tell the solver how many cuda cores should be used?
That's great that you have it working.

As regards speed-up (or lack thereof), if I were you I would first check that you can replicate the speed-up results of Dan Combest (see his slides for OFW6) who originally developed some of these solvers. He used laplacianFoam and a hot block test case, BUT the speed-ups found were only for single precision.

I'm not sure that the GPU linear solvers in the current form show any speed-up for double precision (please correct me if anyone knows better).

Also, a basic solver like laplacianFoam is best suited to these GPU linear solvers as explicit terms are not iteratively updated outside the line solver, like in the SIMPLE/PISO algorithms and in the solidMechanics solvers.

Philip
bigphil is offline   Reply With Quote

Old   June 8, 2014, 14:24
Default
  #7
Senior Member
 
mahdi abdollahzadeh
Join Date: Mar 2011
Location: Covilha,Portugal
Posts: 150
Rep Power: 7
mm.abdollahzadeh is on a distinguished road
Quote:
Originally Posted by derkermit View Post
Hi Philip,
thanks for the hint. I got one step further but it crashes right when the cuda solver should start with the calculation. The error message reads

Code:
terminate called after throwing an instance of 'thrust::system::detail::bad_alloc'
  what():  std::bad_alloc: unknown error
I tried one rhoPimpleFoam tutorial first with the same error and changed to a smaller elasticSolidFoam tutorial case (bimaterialPlate) because I thought the memory size could be an issue (Geforce GTX570 with 1280MB) but that didn't make a change.

fvSolution looks like this:

Code:
    U
    {
        solver          cudaCG;
        preconditioner  diagonal;
        tolerance       1e-07;
        relTol          0.1;
    }
Do you have any idea, what is causing this error? Do I have to decompose the case in order to run the cuda solvers? Compilation of the cuda solvers finished with no errors.
I haven't checked if the cuda examples provided by nvidia work so that will be my next step.

Thanks, Alex
Dear ALex

I had installed everything successfully.
but when I try to run a case I am getting a problem.

Quote:
terminate called after throwing an instance of 'thrust::system::system_error' what (): invalid device function
Aborted (core dumped)
I also try downgrading from cuda 6 to cuda 5.5. but the problem continue to exist.
do u know how to resolve it?

many thanks

best
mahdi
mm.abdollahzadeh is offline   Reply With Quote

Old   June 9, 2014, 15:58
Default
  #8
Member
 
A. Bernath
Join Date: Jun 2011
Location: Karlsruhe, Germany
Posts: 39
Rep Power: 7
derkermit is on a distinguished road
Hi Mahdi,
try to install the driver that comes with the cuda runfile. You can extract it by using

Code:
-extract=/absolute/path/to/extract/location/
as an option for the runfile like it is explained here: http://docs.nvidia.com/cuda/cuda-get...nfile-contents

Regarding the speed up, I did some testing in double precision and there is a speed up for large problems when compared to the CG and BiCG solvers but none in case of GAMG.
I also monitored the usage of the gpu during the calculations. It was only noteworthy high for the larger problems and I think this is the reason for the speed up.

Regards, Alex
mm.abdollahzadeh likes this.
derkermit is offline   Reply With Quote

Old   June 11, 2014, 05:15
Default
  #9
Senior Member
 
T Chourushi
Join Date: Jul 2009
Location: India
Posts: 319
Blog Entries: 1
Rep Power: 10
Tushar@cfd is on a distinguished road
Did anyone knows the information document for the CUDA solvers? A good research article/book/thesis?

--
Thanks in Advance
Tushar@cfd is offline   Reply With Quote

Old   June 11, 2014, 05:39
Default
  #10
Senior Member
 
mahdi abdollahzadeh
Join Date: Mar 2011
Location: Covilha,Portugal
Posts: 150
Rep Power: 7
mm.abdollahzadeh is on a distinguished road
Quote:
Originally Posted by derkermit View Post
Hi Mahdi,
try to install the driver that comes with the cuda runfile. You can extract it by using

Code:
-extract=/absolute/path/to/extract/location/
as an option for the runfile like it is explained here: http://docs.nvidia.com/cuda/cuda-get...nfile-contents

Regarding the speed up, I did some testing in double precision and there is a speed up for large problems when compared to the CG and BiCG solvers but none in case of GAMG.
I also monitored the usage of the gpu during the calculations. It was only noteworthy high for the larger problems and I think this is the reason for the speed up.

Regards, Alex
Dear Bernath

now cuda is working .
but I had tested for motorbike case it didnt show me no speed up.
thus i tried for more heavy case. but then it terminated with a warning regarding memory.
my graphic card is gforce 8400 GS.
is there any way to resolve it? or is it just running on one processers of the graphic card and can i alter it?

best
mahdi
mm.abdollahzadeh is offline   Reply With Quote

Old   June 11, 2014, 10:52
Default
  #11
Member
 
A. Bernath
Join Date: Jun 2011
Location: Karlsruhe, Germany
Posts: 39
Rep Power: 7
derkermit is on a distinguished road
Quote:
Originally Posted by mm.abdollahzadeh View Post
Dear Bernath

now cuda is working .
but I had tested for motorbike case it didnt show me no speed up.
thus i tried for more heavy case. but then it terminated with a warning regarding memory.
my graphic card is gforce 8400 GS.
is there any way to resolve it? or is it just running on one processers of the graphic card and can i alter it?

best
mahdi
Dear Mahdi,
did you actually change the solvers to e.g. cudaCG in fvSolution?
I don't think you can expect too much from your gpu as long as you're not running a super old cpu
I don't know if it's possible to give a number of cores that has to be used like it is the case when decomposing and running on cpu. The only thing I can tell you is that if more elements are used and more inner iteration have to be calculated, the gpu usage rises.

Regards, Alex
derkermit is offline   Reply With Quote

Old   June 11, 2014, 11:00
Default
  #12
Senior Member
 
mahdi abdollahzadeh
Join Date: Mar 2011
Location: Covilha,Portugal
Posts: 150
Rep Power: 7
mm.abdollahzadeh is on a distinguished road
Quote:
Originally Posted by derkermit View Post
Dear Mahdi,
did you actually change the solvers to e.g. cudaCG in fvSolution?
I don't think you can expect too much from your gpu as long as you're not running a super old cpu
I don't know if it's possible to give a number of cores that has to be used like it is the case when decomposing and running on cpu. The only thing I can tell you is that if more elements are used and more inner iteration have to be calculated, the gpu usage rises.

Regards, Alex

Dear Alex

Many thanks for ur answer. sure I had changed the solver to cudaCG and it runned. but as I told it was not efficient for thr motorbike case. then I changed to bigger case but it aborted with warning about memory in thrust.
I have a case for simulation of plasma dynamics. its unsteady my mesh is large and I have also extra inner iterations. speeding up solution is necessary in this case for me. but however, I'm wondering how much benefit i could gain in comparison of running that in cluster with 100 CPUS.

best
mahdi
mm.abdollahzadeh is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
error message with modeling a cube with a hold at the center hsingtzu OpenFOAM Native Meshers: blockMesh 2 March 14, 2012 10:56
mesh airfoil NACA0012 anand_30 OpenFOAM Meshing & Mesh Conversion 12 December 12, 2011 05:16
BlockMesh FOAM warning gaottino OpenFOAM Native Meshers: blockMesh 7 July 19, 2010 14:11
Axisymmetrical mesh Rasmus Gjesing (Gjesing) OpenFOAM Native Meshers: blockMesh 10 April 2, 2007 14:00
Import gmsh msh to Foam adorean Open Source Meshers: Gmsh, Netgen, CGNS, ... 24 April 27, 2005 08:19


All times are GMT -4. The time now is 19:05.