CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > FLUENT

ANSYS Rel15.0 and GPU's

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree8Likes

Reply
 
LinkBack Thread Tools Display Modes
Old   January 15, 2014, 18:33
Default ANSYS Rel15.0 and GPU's
  #1
Senior Member
 
Join Date: Mar 2010
Location: Cape Town, SA
Posts: 156
Rep Power: 9
Jonathan is on a distinguished road
Hi everyone,

Anyone had any luck / experience yet with Fluent in ANSYS release 15.0 and the ability to use GPU acceleration with the PB coupled solver?

which cards / use of more than x1 card (i.e. splitting a problem over x2 or more cards) etc?

I have some good hardware to trial with:
a
SuperMicro M2090 (16 cpu cores, x4 Tesla M2090 cards) but:

1) i am seeing a fairly significant increase in compute time,
2) the simulation which otherwise converges fine on CPU's only ends up blowing up with the GPU running, and
3) i get a complete programme crash when i try to use more than x1 GPU cards

Anyway, just wondering if anyone had had any luck so far on the forums and had any advice / experience to share.

thanks a lot in advance
cheers
jonathan
DungPham likes this.
Jonathan is offline   Reply With Quote

Old   June 11, 2014, 11:17
Default
  #2
Member
 
Hoo
Join Date: Feb 2012
Location: USA
Posts: 81
Rep Power: 6
enayath is on a distinguished road
Jonathan,

I am totally new to run the case with GPU. can you please tell me how to use GPU in Fluent?
do I need to use it with CPus or alone?
My machine has 32 cores and Gcard is Nvidia K6000

Thank you,
Hooman
DungPham likes this.
enayath is offline   Reply With Quote

Old   June 11, 2014, 11:44
Default
  #3
Senior Member
 
Join Date: Mar 2010
Location: Cape Town, SA
Posts: 156
Rep Power: 9
Jonathan is on a distinguished road
hi Hooman,

v150 should run with any Nvidia Fermi / Kepler architecture GPU. The GPU also needs to be CUDA 5 aware. So firstly check your hardware to make sure you meet these requirements.

After that, the process is simple:

1) Start fluent using: fluent 3d(dp) -txx -gpgpu=1

At this point, only problems using the pressure-based coupled solver (PBCS) will benefit from GPU usage as the overhead in getting information on and off of the GPU when using the segregated solver will cancel out the matrix solving time reductions.

Also, currently the GPU solver can only work with single phase problems.

Your biggest constraint will probably be free memory on the GPU - the ANSYS seminar i attended indicated ~4GB GPU memory is required per 1M cells in the mesh (for 3ddp calculations).

Also, i had an issue when i tried to solve the energy equation for my model, so i recommend turning this off when you run your solution. (There is a fix but i wont go into it unless you need it).

hope this helps,
cheers
jonathan
DungPham likes this.
Jonathan is offline   Reply With Quote

Old   June 11, 2014, 11:54
Default
  #4
Member
 
Hoo
Join Date: Feb 2012
Location: USA
Posts: 81
Rep Power: 6
enayath is on a distinguished road
jonathan,

Thanks fo rthe quick reply,

my graphic card has 12G ram and the regular Ram is 128
I did not get what you mean by "txx in "Start fluent using: fluent 3d(dp) -txx -gpgpu=1.Can I have something similar to the attached jpeg file or the number of the GPGPus per machine should be equal to number of processes?

The other thing, I want to solve turbulence and heat transfer...I am afraid you said I am not going to be able o run my single phase calculation while enabling energ in Fluent!

Thank you,
Hooman
Attached Images
File Type: jpg john.jpg (50.5 KB, 63 views)
DungPham likes this.
enayath is offline   Reply With Quote

Old   June 11, 2014, 12:10
Default
  #5
Senior Member
 
Join Date: Mar 2010
Location: Cape Town, SA
Posts: 156
Rep Power: 9
Jonathan is on a distinguished road
Hi,

The instructions i gave were for starting Fluent using the command line - I guess you are using Windows and therefore are all GUI based - no problem.

So, open the Fluent launcher window (your graphic) as normal
1) Select Parallel
2) Select no of parallel CPU threads you want to use (this will depend on you licence) (recommend you just choose what you normally use)
3) Select '1' in GPGPUs per Machine box

So, when you set up your problem - select the Pressure-based solver from the General tab, and then Coupled in the P-V-coupling window under Solution Methods

I was solving Energy but using constant density fluid properties, so i needed to change the solution of the Energy equation to use the Flexible multigrid cycle (Solution Controls -> Advanced - Multigrid tab -> Change Energy solution method to Flexible cycle from F-Cycle in drop down menu). Make sure your turbulence model equations are also set to the flexible cycle - they should be by default.

NB. Only the flow (pressure and momentum) equations are solved in a coupled fashion on the GPU. The turbulence and energy equations are solved on the CPU in a segregated fashion. The issue above is a bug and has apparently been fixed in v160.

You dont need to do anything else (you only have x1 GPU so everything should be fine). There are other expert settings you can play with if you have multiple GPU's and you want to specify explicitly to Fluent which ones you want to use.

Also, if you are using multiple GPU's (which you are not so dont worry - info just FYI), you need to make sure that the no of parallel solver processes you specify when you start Fluent, is evenly divisible by the no of GPGPU's you have specified.

i.e. x2 GPUS = 2 / 4 / 6 / 8 / 12 etc CPU threads
x3 GPUS = 3 / 6 / 12 / 18 / 24 etc CPU threads

Otherwise everything else will happen automatically - you will see when you start solving the GPU linear solver will kick in and start working.

hope this helps,
cheers
jonathan
DungPham likes this.
Jonathan is offline   Reply With Quote

Old   June 11, 2014, 13:05
Default
  #6
Member
 
Hoo
Join Date: Feb 2012
Location: USA
Posts: 81
Rep Power: 6
enayath is on a distinguished road
Jonathan,

Thank you so very much for the explanation and time...
Got it...I defined my density as a function of temperature in the material section...I have license for Fluent 15 and not 16...so you mean still the energy and turbulence will be calculated on CPU and flow on Gpu?
Regards,
Hooman
DungPham likes this.
enayath is offline   Reply With Quote

Old   June 11, 2014, 13:15
Default
  #7
Senior Member
 
Join Date: Mar 2010
Location: Cape Town, SA
Posts: 156
Rep Power: 9
Jonathan is on a distinguished road
hi - no problem, glad to help,

Quote:
I have license for Fluent 15 and not 16
v160 has not been released yet so no problem there

Quote:
...so you mean still the energy and turbulence will be calculated on CPU and flow on Gpu?
When you use the PBCS, energy and turbulence are always solved in a segregated fashion, after the flow equations have been solved. Practically speaking, because of this, (i.e. because the GPU adds benefit when solving large system matrices), the GPU acceleration will only really add benefit when solving the large pressure-momentum coupled system, and so the other additional equations (turb, energy) will not be solved on the GPU. The time it takes to build the matrices (for the non-flow equations) and transfer them onto the GPU vs the solving time, will cancel out and there will be no real benefit - that is why only the (coupled) flow equations are solved on the GPU.

You can force the other equations to solve on the GPU (as far as i know) from the TUI, but it would not be a good idea in terms of speeding up your simulations.

Incidently, if you get things working nicely - i.e. good speedups when using the GPU, can you let me know? I would appreciate that greatly as i did not get good results - in fact, my simulations ran slower when using GPUs.

thanks - take care,
cheers
jonathan
DungPham likes this.
Jonathan is offline   Reply With Quote

Old   June 11, 2014, 13:30
Default
  #8
Member
 
Hoo
Join Date: Feb 2012
Location: USA
Posts: 81
Rep Power: 6
enayath is on a distinguished road
Hi,

I will do that for sure but it may take about 2 weeks as currently I am running my cases on 14.5 and need to install 15 on my main machine.After that I will keep you posted.

Hooman
enayath is offline   Reply With Quote

Old   August 8, 2014, 10:45
Default
  #9
Member
 
ahmad
Join Date: Feb 2012
Posts: 97
Rep Power: 6
malay is on a distinguished road
Quote:
Originally Posted by Jonathan View Post
in fact, my simulations ran slower when using GPUs.

also me !!! why ?
malay is offline   Reply With Quote

Old   August 8, 2014, 10:47
Default
  #10
Member
 
Hoo
Join Date: Feb 2012
Location: USA
Posts: 81
Rep Power: 6
enayath is on a distinguished road
Hello folks,

Sorry I forgot to post here...
I got the same issue...mine is slower when using GPU...so I went back to regular calculation....

Hooman
enayath is offline   Reply With Quote

Old   August 8, 2014, 10:52
Default
  #11
Member
 
ahmad
Join Date: Feb 2012
Posts: 97
Rep Power: 6
malay is on a distinguished road
Quote:
Originally Posted by enayath View Post
Hello folks,

Sorry I forgot to post here...
I got the same issue...mine is slower when using GPU...so I went back to regular calculation....

Hooman
Is there any explanation for that ?
malay is offline   Reply With Quote

Old   August 8, 2014, 10:52
Default
  #12
Member
 
Hoo
Join Date: Feb 2012
Location: USA
Posts: 81
Rep Power: 6
enayath is on a distinguished road
No idea...
enayath is offline   Reply With Quote

Old   August 8, 2014, 10:55
Default
  #13
Member
 
ahmad
Join Date: Feb 2012
Posts: 97
Rep Power: 6
malay is on a distinguished road
Quote:
Originally Posted by enayath View Post
No idea...
so that mean Ndidia+Ansys cheat us
malay is offline   Reply With Quote

Old   August 8, 2014, 10:58
Default
  #14
Member
 
Hoo
Join Date: Feb 2012
Location: USA
Posts: 81
Rep Power: 6
enayath is on a distinguished road
I hope there is something wrong with our set up...I paid about $5000 to have a good Nvidia product...
DungPham likes this.
enayath is offline   Reply With Quote

Old   August 8, 2014, 10:59
Default
  #15
Member
 
ahmad
Join Date: Feb 2012
Posts: 97
Rep Power: 6
malay is on a distinguished road
Quote:
Originally Posted by enayath View Post
I hope there is something wrong with our set up...I paid about $5000 to have a good Nvidia product...
I hope that also ..


please if you have any new solution , let me know here
malay is offline   Reply With Quote

Old   September 5, 2014, 11:40
Default
  #16
New Member
 
DungPham's Avatar
 
Dung A.Pham
Join Date: Sep 2011
Location: Anseong, Korea
Posts: 20
Rep Power: 6
DungPham is on a distinguished road
Quote:
Originally Posted by enayath View Post
I hope there is something wrong with our set up...I paid about $5000 to have a good Nvidia product...
I hope so .
DungPham is offline   Reply With Quote

Old   September 18, 2014, 02:22
Default
  #17
New Member
 
Kimi
Join Date: May 2012
Posts: 5
Rep Power: 6
kimireggie is on a distinguished road
Quote:
Originally Posted by Jonathan View Post
[SIZE=2]

1) i am seeing a fairly significant increase in compute time,
2) the simulation which otherwise converges fine on CPU's only ends up blowing up with the GPU running, and
3) i get a complete programme crash when i try to use more than x1 GPU cards
Hi Jonathan, glad to find your post by Google.

Some issues you met during GPGPU calculation also occurred under similar environment (16 CPU cores, 1 K40):
1) with some case, it takes longer for Fluent to solve with GPU, i guess there is something to do with problem size and parallel size, but still need to verify.
2) with some case (2 out of 5), it does diverge with GPU when convergence can be achieved with only CPUs. Change from DB to PB does help though.
3) by crash, you mean ...?
kimireggie is offline   Reply With Quote

Old   September 18, 2014, 02:36
Default
  #18
New Member
 
Kimi
Join Date: May 2012
Posts: 5
Rep Power: 6
kimireggie is on a distinguished road
Quote:
Originally Posted by Jonathan View Post
Hi,

So, when you set up your problem - select the Pressure-based solver from the General tab, and then Coupled in the P-V-coupling window under Solution Methods
Hi Jonathan,

Does it have to be Coupled rather than SIMPLE under P-V-coupling setup? Thanks.
kimireggie is offline   Reply With Quote

Old   September 18, 2014, 08:41
Default
  #19
Senior Member
 
Lucky Tran
Join Date: Apr 2011
Location: Orlando, FL USA
Posts: 1,271
Rep Power: 20
LuckyTran will become famous soon enough
Quote:
Originally Posted by kimireggie View Post
Hi Jonathan,

Does it have to be Coupled rather than SIMPLE under P-V-coupling setup? Thanks.
With GPU enabled, by default Fluent only uses the GPU for the coupled scheme. You can force the Fluent to use the GPU for SIMPLE scheme (through some commands in the TUI), but it slows down rather than accelerates the computation. Maybe in future releases Fluent will have better support for other schemes but right now it works sometimes only for Coupled and works badly for the others.
LuckyTran is offline   Reply With Quote

Old   September 18, 2014, 21:32
Default
  #20
New Member
 
Kimi
Join Date: May 2012
Posts: 5
Rep Power: 6
kimireggie is on a distinguished road
Quote:
Originally Posted by LuckyTran View Post
With GPU enabled, by default Fluent only uses the GPU for the coupled scheme.
Thanks for your quick reply.

Did you mean that once GPGPU acceleration is enabled, Fluent always adopts coupled scheme for P-V-Coupling, even if SIMPLE/SIMPLEC has been set in the case file? (Assumed no more parameters regarding GPGPU is set in command line or journal file except '-gpgpu=1')
kimireggie is offline   Reply With Quote

Reply

Tags
coupled solver, fluent, gpu, pbns, v15

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
GPU acceleration on ANSYS Fluent 14.5 Daveo643 FLUENT 19 October 20, 2015 14:35
Using GPUs with UDFs TMac Fluent UDF and Scheme Programming 1 January 11, 2013 13:24
DELL Tested Acceleration of OpenFOAM using GPUs Lukasz OpenFOAM Running, Solving & CFD 0 September 17, 2012 16:32
Is it possible to speed up FLUENT by: NVIDIA Tesla GPUs czaru Hardware 0 November 18, 2011 05:33
Will GPU's play a part in the future of CFD? Tom Main CFD Forum 15 January 22, 2009 06:04


All times are GMT -4. The time now is 23:58.