[derkermit]

Hello,
has anybody suceeded up to now in using the new cuda solvers of foam-extend-3.0?

As an example, I changed in a tutorial case from PCG to cudaCG but the solver then complains about not knowing this type of solver. Also the printed list does not contain the cuda solvers. Before wasting too much time I thought it could be helpful to ask here for people which have or, even better, had the same problem.

Would be nice to get some help and share experiences!

Bye, Alex

[derkermit]

So nobody has used that solvers until now?

[bigphil]

Hi,

I don't think the cuda library is loaded by default in most solvers, so you would need to add the following to system/controlDict (on Linux):

libs                  ("libcudaSolvers.so");

or (on Mac)

libs                  ("libcudaSolvers.dylib");

Also, this is assuming that the cuda library was actually compiled; it is only compiled if your machine has a cuda capable device. You can check if it compiled by:

cd $FOAM_SRC/cudaSolvers
./Allwmake

Philip

[derkermit]

Hi Philip,
thanks for the hint. I got one step further but it crashes right when the cuda solver should start with the calculation. The error message reads

terminate called after throwing an instance of 'thrust::system::detail::bad_alloc'
  what():  std::bad_alloc: unknown error

I tried one rhoPimpleFoam tutorial first with the same error and changed to a smaller elasticSolidFoam tutorial case (bimaterialPlate) because I thought the memory size could be an issue (Geforce GTX570 with 1280MB) but that didn't make a change.

fvSolution looks like this:

    U
    {
        solver          cudaCG;
        preconditioner  diagonal;
        tolerance       1e-07;
        relTol          0.1;
    }

Do you have any idea, what is causing this error? Do I have to decompose the case in order to run the cuda solvers? Compilation of the cuda solvers finished with no errors.
I haven't checked if the cuda examples provided by nvidia work so that will be my next step.

Thanks, Alex

[derkermit]

Godd news, CUDA works now. I had the wrong driver version installed and after downgrading to the one which came with the cuda installer, everything works fine.

I tested some tutorial cases but actually it isn't faster at all when using the GPU. I checked the GPU utilization level and I could not get that value above 10% so I guess if I could optimize this, the calculation will be faster. But how to achieve this?

Is there anyone who actually has accelerated calculations with this solvers?

Do I have to tell the solver how many cuda cores should be used?

[bigphil]

Originally Posted by derkermit

Godd news, CUDA works now. I had the wrong driver version installed and after downgrading to the one which came with the cuda installer, everything works fine.

I tested some tutorial cases but actually it isn't faster at all when using the GPU. I checked the GPU utilization level and I could not get that value above 10% so I guess if I could optimize this, the calculation will be faster. But how to achieve this?

Is there anyone who actually has accelerated calculations with this solvers?

Do I have to tell the solver how many cuda cores should be used?

That's great that you have it working.

As regards speed-up (or lack thereof), if I were you I would first check that you can replicate the speed-up results of Dan Combest (see his slides for OFW6) who originally developed some of these solvers. He used laplacianFoam and a hot block test case, BUT the speed-ups found were only for single precision.

I'm not sure that the GPU linear solvers in the current form show any speed-up for double precision (please correct me if anyone knows better).

Also, a basic solver like laplacianFoam is best suited to these GPU linear solvers as explicit terms are not iteratively updated outside the line solver, like in the SIMPLE/PISO algorithms and in the solidMechanics solvers.

Philip

[mm.abdollahzadeh]

Originally Posted by derkermit

Hi Philip,
thanks for the hint. I got one step further but it crashes right when the cuda solver should start with the calculation. The error message reads

terminate called after throwing an instance of 'thrust::system::detail::bad_alloc'
  what():  std::bad_alloc: unknown error

I tried one rhoPimpleFoam tutorial first with the same error and changed to a smaller elasticSolidFoam tutorial case (bimaterialPlate) because I thought the memory size could be an issue (Geforce GTX570 with 1280MB) but that didn't make a change.

fvSolution looks like this:

    U
    {
        solver          cudaCG;
        preconditioner  diagonal;
        tolerance       1e-07;
        relTol          0.1;
    }

Do you have any idea, what is causing this error? Do I have to decompose the case in order to run the cuda solvers? Compilation of the cuda solvers finished with no errors.
I haven't checked if the cuda examples provided by nvidia work so that will be my next step.

Thanks, Alex

Dear ALex

I had installed everything successfully.
but when I try to run a case I am getting a problem.

Quote:
terminate called after throwing an instance of 'thrust::system::system_error' what (): invalid device function
Aborted (core dumped)
I also try downgrading from cuda 6 to cuda 5.5. but the problem continue to exist.
do u know how to resolve it?

many thanks

best
mahdi

[derkermit]

Hi Mahdi,
try to install the driver that comes with the cuda runfile. You can extract it by using

-extract=/absolute/path/to/extract/location/

as an option for the runfile like it is explained here: http://docs.nvidia.com/cuda/cuda-get...nfile-contents

Regarding the speed up, I did some testing in double precision and there is a speed up for large problems when compared to the CG and BiCG solvers but none in case of GAMG.
I also monitored the usage of the gpu during the calculations. It was only noteworthy high for the larger problems and I think this is the reason for the speed up.

Regards, Alex

[Tushar@cfd]

Did anyone knows the information document for the CUDA solvers? A good research article/book/thesis?

--
Thanks in Advance

[mm.abdollahzadeh]

Originally Posted by derkermit

Hi Mahdi,
try to install the driver that comes with the cuda runfile. You can extract it by using

-extract=/absolute/path/to/extract/location/

as an option for the runfile like it is explained here: http://docs.nvidia.com/cuda/cuda-get...nfile-contents

Regarding the speed up, I did some testing in double precision and there is a speed up for large problems when compared to the CG and BiCG solvers but none in case of GAMG.
I also monitored the usage of the gpu during the calculations. It was only noteworthy high for the larger problems and I think this is the reason for the speed up.

Regards, Alex

Dear Bernath

now cuda is working .
but I had tested for motorbike case it didnt show me no speed up.
thus i tried for more heavy case. but then it terminated with a warning regarding memory.
my graphic card is gforce 8400 GS.
is there any way to resolve it? or is it just running on one processers of the graphic card and can i alter it?

best
mahdi

[derkermit]

Originally Posted by mm.abdollahzadeh

Dear Bernath

now cuda is working .
but I had tested for motorbike case it didnt show me no speed up.
thus i tried for more heavy case. but then it terminated with a warning regarding memory.
my graphic card is gforce 8400 GS.
is there any way to resolve it? or is it just running on one processers of the graphic card and can i alter it?

best
mahdi

Dear Mahdi,
did you actually change the solvers to e.g. cudaCG in fvSolution?
I don't think you can expect too much from your gpu as long as you're not running a super old cpu
I don't know if it's possible to give a number of cores that has to be used like it is the case when decomposing and running on cpu. The only thing I can tell you is that if more elements are used and more inner iteration have to be calculated, the gpu usage rises.

Regards, Alex

[mm.abdollahzadeh]

Originally Posted by derkermit

Dear Mahdi,
did you actually change the solvers to e.g. cudaCG in fvSolution?
I don't think you can expect too much from your gpu as long as you're not running a super old cpu
I don't know if it's possible to give a number of cores that has to be used like it is the case when decomposing and running on cpu. The only thing I can tell you is that if more elements are used and more inner iteration have to be calculated, the gpu usage rises.

Regards, Alex

Dear Alex

Many thanks for ur answer. sure I had changed the solver to cudaCG and it runned. but as I told it was not efficient for thr motorbike case. then I changed to bigger case but it aborted with warning about memory in thrust.
I have a case for simulation of plasma dynamics. its unsteady my mesh is large and I have also extra inner iterations. speeding up solution is necessary in this case for me. but however, I'm wondering how much benefit i could gain in comparison of running that in cluster with 100 CPUS.

best
mahdi