[Jonathan]

Hi everyone,

Anyone had any luck / experience yet with Fluent in ANSYS release 15.0 and the ability to use GPU acceleration with the PB coupled solver?

which cards / use of more than x1 card (i.e. splitting a problem over x2 or more cards) etc?

I have some good hardware to trial with:
a SuperMicro M2090 (16 cpu cores, x4 Tesla M2090 cards) but:

1) i am seeing a fairly significant increase in compute time,
2) the simulation which otherwise converges fine on CPU's only ends up blowing up with the GPU running, and
3) i get a complete programme crash when i try to use more than x1 GPU cards

Anyway, just wondering if anyone had had any luck so far on the forums and had any advice / experience to share.

thanks a lot in advance
cheers
jonathan

[enayath]

Jonathan,

I am totally new to run the case with GPU. can you please tell me how to use GPU in Fluent?
do I need to use it with CPus or alone?
My machine has 32 cores and Gcard is Nvidia K6000

Thank you,
Hooman

[Jonathan]

hi Hooman,

v150 should run with any Nvidia Fermi / Kepler architecture GPU. The GPU also needs to be CUDA 5 aware. So firstly check your hardware to make sure you meet these requirements.

After that, the process is simple:

1) Start fluent using: fluent 3d(dp) -txx -gpgpu=1

At this point, only problems using the pressure-based coupled solver (PBCS) will benefit from GPU usage as the overhead in getting information on and off of the GPU when using the segregated solver will cancel out the matrix solving time reductions.

Also, currently the GPU solver can only work with single phase problems.

Your biggest constraint will probably be free memory on the GPU - the ANSYS seminar i attended indicated ~4GB GPU memory is required per 1M cells in the mesh (for 3ddp calculations).

Also, i had an issue when i tried to solve the energy equation for my model, so i recommend turning this off when you run your solution. (There is a fix but i wont go into it unless you need it).

hope this helps,
cheers
jonathan

[enayath]

jonathan,

Thanks fo rthe quick reply,

my graphic card has 12G ram and the regular Ram is 128
I did not get what you mean by "txx in "Start fluent using: fluent 3d(dp) -txx -gpgpu=1.Can I have something similar to the attached jpeg file or the number of the GPGPus per machine should be equal to number of processes?

The other thing, I want to solve turbulence and heat transfer...I am afraid you said I am not going to be able o run my single phase calculation while enabling energ in Fluent!

Thank you,
Hooman

[Jonathan]

Hi,

The instructions i gave were for starting Fluent using the command line - I guess you are using Windows and therefore are all GUI based - no problem.

So, open the Fluent launcher window (your graphic) as normal
1) Select Parallel
2) Select no of parallel CPU threads you want to use (this will depend on you licence) (recommend you just choose what you normally use)
3) Select '1' in GPGPUs per Machine box

So, when you set up your problem - select the Pressure-based solver from the General tab, and then Coupled in the P-V-coupling window under Solution Methods

I was solving Energy but using constant density fluid properties, so i needed to change the solution of the Energy equation to use the Flexible multigrid cycle (Solution Controls -> Advanced - Multigrid tab -> Change Energy solution method to Flexible cycle from F-Cycle in drop down menu). Make sure your turbulence model equations are also set to the flexible cycle - they should be by default.

NB. Only the flow (pressure and momentum) equations are solved in a coupled fashion on the GPU. The turbulence and energy equations are solved on the CPU in a segregated fashion. The issue above is a bug and has apparently been fixed in v160.

You dont need to do anything else (you only have x1 GPU so everything should be fine). There are other expert settings you can play with if you have multiple GPU's and you want to specify explicitly to Fluent which ones you want to use.

Also, if you are using multiple GPU's (which you are not so dont worry - info just FYI), you need to make sure that the no of parallel solver processes you specify when you start Fluent, is evenly divisible by the no of GPGPU's you have specified.

i.e. x2 GPUS = 2 / 4 / 6 / 8 / 12 etc CPU threads
x3 GPUS = 3 / 6 / 12 / 18 / 24 etc CPU threads

Otherwise everything else will happen automatically - you will see when you start solving the GPU linear solver will kick in and start working.

hope this helps,
cheers
jonathan

[enayath]

Jonathan,

Thank you so very much for the explanation and time...
Got it...I defined my density as a function of temperature in the material section...I have license for Fluent 15 and not 16...so you mean still the energy and turbulence will be calculated on CPU and flow on Gpu?
Regards,
Hooman

[Jonathan]

hi - no problem, glad to help,

I have license for Fluent 15 and not 16

v160 has not been released yet so no problem there

...so you mean still the energy and turbulence will be calculated on CPU and flow on Gpu?

When you use the PBCS, energy and turbulence are always solved in a segregated fashion, after the flow equations have been solved. Practically speaking, because of this, (i.e. because the GPU adds benefit when solving large system matrices), the GPU acceleration will only really add benefit when solving the large pressure-momentum coupled system, and so the other additional equations (turb, energy) will not be solved on the GPU. The time it takes to build the matrices (for the non-flow equations) and transfer them onto the GPU vs the solving time, will cancel out and there will be no real benefit - that is why only the (coupled) flow equations are solved on the GPU.

You can force the other equations to solve on the GPU (as far as i know) from the TUI, but it would not be a good idea in terms of speeding up your simulations.

Incidently, if you get things working nicely - i.e. good speedups when using the GPU, can you let me know? I would appreciate that greatly as i did not get good results - in fact, my simulations ran slower when using GPUs.

thanks - take care,
cheers
jonathan

[enayath]

Hi,

I will do that for sure but it may take about 2 weeks as currently I am running my cases on 14.5 and need to install 15 on my main machine.After that I will keep you posted.

Hooman

[malay]

Originally Posted by Jonathan

in fact, my simulations ran slower when using GPUs.

also me !!! why ?

[enayath]

Hello folks,

Sorry I forgot to post here...
I got the same issue...mine is slower when using GPU...so I went back to regular calculation....

Hooman

[malay]

Originally Posted by enayath

Hello folks,

Sorry I forgot to post here...
I got the same issue...mine is slower when using GPU...so I went back to regular calculation....

Hooman

Is there any explanation for that ?

[enayath]

No idea...

[malay]

Originally Posted by enayath

No idea...

so that mean Ndidia+Ansys cheat us

[enayath]

I hope there is something wrong with our set up...I paid about $5000 to have a good Nvidia product...

[malay]

Originally Posted by enayath

I hope there is something wrong with our set up...I paid about $5000 to have a good Nvidia product...

I hope that also ..


please if you have any new solution , let me know here

[DungPham]

Originally Posted by enayath

I hope there is something wrong with our set up...I paid about $5000 to have a good Nvidia product...

I hope so .

[kimireggie]

Originally Posted by Jonathan

[SIZE=2]

1) i am seeing a fairly significant increase in compute time,
2) the simulation which otherwise converges fine on CPU's only ends up blowing up with the GPU running, and
3) i get a complete programme crash when i try to use more than x1 GPU cards

Hi Jonathan, glad to find your post by Google.

Some issues you met during GPGPU calculation also occurred under similar environment (16 CPU cores, 1 K40):
1) with some case, it takes longer for Fluent to solve with GPU, i guess there is something to do with problem size and parallel size, but still need to verify.
2) with some case (2 out of 5), it does diverge with GPU when convergence can be achieved with only CPUs. Change from DB to PB does help though.
3) by crash, you mean ...?

[kimireggie]

Originally Posted by Jonathan

Hi,

So, when you set up your problem - select the Pressure-based solver from the General tab, and then Coupled in the P-V-coupling window under Solution Methods

Hi Jonathan,

Does it have to be Coupled rather than SIMPLE under P-V-coupling setup? Thanks.

[LuckyTran]

Originally Posted by kimireggie

Hi Jonathan,

Does it have to be Coupled rather than SIMPLE under P-V-coupling setup? Thanks.

With GPU enabled, by default Fluent only uses the GPU for the coupled scheme. You can force the Fluent to use the GPU for SIMPLE scheme (through some commands in the TUI), but it slows down rather than accelerates the computation. Maybe in future releases Fluent will have better support for other schemes but right now it works sometimes only for Coupled and works badly for the others.

[kimireggie]

Originally Posted by LuckyTran

With GPU enabled, by default Fluent only uses the GPU for the coupled scheme.

Thanks for your quick reply.

Did you mean that once GPGPU acceleration is enabled, Fluent always adopts coupled scheme for P-V-Coupling, even if SIMPLE/SIMPLEC has been set in the case file? (Assumed no more parameters regarding GPGPU is set in command line or journal file except '-gpgpu=1')