CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

SpeedIT Classic and OpenFOAM

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   September 13, 2010, 05:48
Default SpeedIT Classic and OpenFOAM
  #1
Senior Member
 
Andrea Pasquali
Join Date: Sep 2009
Location: Germany
Posts: 142
Rep Power: 7
andrea.pasquali is on a distinguished road
Hi,
I'm trying the SpeedIT Classic version with OpenFOAM plugin on OF 1.7.
I compiled the SpeedIT library and OF plugin well (I think) following all the README.txt files but when I try icoFoam with PCG_accel solver I obtain "ERROR : solver function returned -1".
My solver function is:
solver_result = sicl_gscsrcg( n_rows, pgpu_vals, pgpu_c_idx, pgpu_r_idx, pgpu_X, pgpu_B, precond, &n_iter, &eps) ;

Am I forgetting something?

Thanks in advance for any help

Andrea
__________________
Andrea Pasquali
andrea.pasquali is offline   Reply With Quote

Old   November 2, 2010, 01:43
Default Return -1
  #2
New Member
 
Fábio
Join Date: Sep 2010
Posts: 2
Rep Power: 0
fsantos is on a distinguished road
Hi andrea, have you recompiled OF in single precision?? It can be the reason of the solver error.

Best regard
Fábio Santos
fsantos is offline   Reply With Quote

Old   November 2, 2010, 17:21
Default maybe try to recompile the speedit code in CUDA
  #3
Senior Member
 
chegdan's Avatar
 
Daniel P. Combest
Join Date: Mar 2009
Location: St. Louis, USA
Posts: 543
Rep Power: 18
chegdan will become famous soon enough
Instead of recompiling OF in single precision (and you want double precision), if you have a 1.3 compute capable card for double precision on your computer, change the floats to double in the speedIt cuda code (if you know some CUDA) and then try it again.

to compile CUDA code in doubel precision at the -arch sm-1.3 flag to your nvcc calls. this might be in a compile script.

Dan
chegdan is offline   Reply With Quote

Old   November 16, 2010, 05:17
Default
  #4
New Member
 
Stephan Koerner
Join Date: Mar 2009
Posts: 4
Rep Power: 8
phan is on a distinguished road
Hello! I'm also trying to set up the OpenFoam Plugin. But as I'm not that familiar with C I don't understand how to insert the solver functions correctly.
If I delete the comment signs of the given functions and compile I get the following error message:

"si_... was not declared in this scope"

And if I don't insert any function it can be compiled without any problem but as soon I start the simulation I get the message:

"Call your own solver from PBiCG_accel.C file"

Do I need to edit the PBiCG_accel.C file at all?
I switched OpenFoam already to single precision.

I would really appreciate your help. Thanks in advance,
Stephan

Last edited by phan; November 16, 2010 at 07:50.
phan is offline   Reply With Quote

Old   November 16, 2010, 16:13
Default
  #5
Member
 
Lukasz Miroslaw
Join Date: Dec 2009
Location: Poland
Posts: 64
Rep Power: 7
Lukasz is on a distinguished road
Send a message via Skype™ to Lukasz
Please post the question to the forum at speedit.vratis.com as it does not concern OF users directly (you need to register first). We will be happy to answer your question.
Lukasz is offline   Reply With Quote

Old   November 17, 2010, 16:50
Default
  #6
New Member
 
Zhijun
Join Date: Nov 2010
Posts: 3
Rep Power: 6
xizhijun is on a distinguished road
dear lukasz ,please respect that openfoam is free software and that you should respect everybodys opinion if you take the freedom to advertise your products in this forum .i and my colleges don't like this style .can somebody please post the openfoam cuda plugin to the git repository so that we can evaluate the code without registering to this company?
thanks zhijun !
xizhijun is offline   Reply With Quote

Old   November 19, 2010, 14:54
Default
  #7
Member
 
Lukasz Miroslaw
Join Date: Dec 2009
Location: Poland
Posts: 64
Rep Power: 7
Lukasz is on a distinguished road
Send a message via Skype™ to Lukasz
Dear All,

We are happy to announce a new release of the OpenFOAM plugin 1.1 (GPL License).
Here is the list of features:

-Multi-GPU support.
-Tested on Fermi architecture (GTX460 and Tesla C2050).
-Automated submission of the domain to the GPU cards (using decomposePar from OpenFOAM).
-Optimized submission of computational tasks to the best GPU card in the system for any number of computational threads.
-Plugin picks the most powerful GPU card for a single thread cases.

You can freely download it at speedit.vratis.com. Enjoy!
Lukasz is offline   Reply With Quote

Old   December 13, 2010, 07:42
Default
  #8
Senior Member
 
Andrea Pasquali
Join Date: Sep 2009
Location: Germany
Posts: 142
Rep Power: 7
andrea.pasquali is on a distinguished road
Hi,
I'm coming back to use Speed IT + OpenFOAM plug in.
I installed the new CUDA toolkit 3.2, the Speed IT 1.0, and the OpenFOAM plug in 1.0.
When I run the "icoFoam" test with PCG_accel I have:

--> FOAM FATAL ERROR:
Can not allocate GPU memory

FOAM exiting

I don't know where is the probem,
could anyone help me ?

Thanks

Andrea
__________________
Andrea Pasquali
andrea.pasquali is offline   Reply With Quote

Old   December 14, 2010, 07:50
Default
  #9
Senior Member
 
Andrea Pasquali
Join Date: Sep 2009
Location: Germany
Posts: 142
Rep Power: 7
andrea.pasquali is on a distinguished road
I found Cuda Error number 38 = cudaErrorNoDevice = No available CUDA device.
I meet this error when PCG_accel.C calls the first cuda command (cudaMalloc).
What means "No available CUDA device" ?
I tested the examples in SDK and they work well.
Any help?

Thanks
__________________
Andrea Pasquali
andrea.pasquali is offline   Reply With Quote

Old   December 14, 2010, 13:10
Default
  #10
Senior Member
 
Andrea Pasquali
Join Date: Sep 2009
Location: Germany
Posts: 142
Rep Power: 7
andrea.pasquali is on a distinguished road
Hi,
I solve my (first) problem . Now I have problem in single / double precision. I'm trying to compile speed it in double (I changed the float in double inside .cu and .h files)
I run make ad I obtained:

si_classic.cu(226): error: argument of type "const double *" is incompatible with parameter of type "const float *"

si_classic.cu(242): error: argument of type "const double *" is incompatible with parameter of type "const float *"

si_classic.cu(242): error: argument of type "const double *" is incompatible with parameter of type "const float *"

226 is cublasSnrm2 that is float (in cublas.h)
226 is cublasSdot that is float (in cublas.h)

What I have to do? I'm trying to rechange they from double to float but I found other errors... have I to change the cublas.h file?

Thanks for any help

Andrea
__________________
Andrea Pasquali
andrea.pasquali is offline   Reply With Quote

Old   December 15, 2010, 07:18
Default
  #11
Senior Member
 
Andrea Pasquali
Join Date: Sep 2009
Location: Germany
Posts: 142
Rep Power: 7
andrea.pasquali is on a distinguished road
I recompiled the si_classic.cu and si_classic.h in double precision but my GPU is not still working. This is what I did:

- In .cu file I changed the "cublasSnrm2" and "cublasSdot" functions in "cublasDnrm2" and "cublasDdot", the "sicl_gs..." solver functions in "si_gd..." and all floats in double;
- In .h file I changed the "sicl_gs..." solver functions in "sicl_gd..." and and all floats in double;
- In Makefile I added at nvcc command the option "--gpu-architecture sm_13" and the compilation with "make" command is ok without any errors;
- Then I recompiled the PCG_accel.C lib adding the "sicl_gdcsrcg" solver function in "#elif defined(WM_DP) // scalar is double" statment without success. I have error "sicl_gdcsrcg was not declared in this scope";
- But if I add the "sicl_gdcsrcg" solver function in "#if defined(WM_SP) // scalar is float" statment the compilation is up to date! My OpenFOAM is in double precision! However the "icoFoam" test gives me this result at pressure iteartion:

ERROR : solver function returned -1
diagonalPCG_accel: Solving for p, Initial residual = 0, Final residual = 1e-06, No Iterations 1000

Am I forgetting anything else needs to be changed from float to double? Or am I wronging anything else?
Thanks for any help

Andrea
__________________
Andrea Pasquali
andrea.pasquali is offline   Reply With Quote

Old   February 9, 2011, 17:35
Default
  #12
Senior Member
 
Join Date: Mar 2009
Location: Austin, TX
Posts: 134
Rep Power: 9
kyle is on a distinguished road
Did anyone ever get this working? I got everything compiled and installed, but I am getting a seg fault immediately after "Time = 1". If I remove the "_accel" from the solvers then it runs fine, so everything is good FOAM wise.

I am using CUDA 3.2, OpenFOAM 1.6-ext, GCC 4.4.4 and RHEL 6.

To get the plugin compiled I had to go in and make several private members of "lduMatrix::solver" public. It seems like this should be documented if it is actually required, so I am wondering if I am using an incompatible set of software versions, or if I am just doing something else wrong.
kyle is offline   Reply With Quote

Old   February 9, 2011, 17:51
Default
  #13
Senior Member
 
Join Date: Mar 2009
Location: Austin, TX
Posts: 134
Rep Power: 9
kyle is on a distinguished road
Judging from this Japanese blog post, we are looking not looking at any benefit anyway. This guy saw a 3x slowdown when compared to unaccelerated GAMG. This is the only data I have seen that isn't from the developers themselves. He used extremely small cases, so that might be limiting the GPU solvers somehow. Also if Google translate is to be trusted, the GPU accelerated solutions are not even accurate.

http://www.geocities.co.jp/SiliconVa...T-Classic.html
kyle is offline   Reply With Quote

Old   February 9, 2011, 18:45
Default
  #14
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,301
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Kyle,

I've never used SpeedIT so I don't know why the code isn't working for you. But I have accompanied some of the posts about SpeedIT and I can at least advise about one detail: the SpeedIT Classic is for single precision only, so it can only be fairly comparable with the single precision build of OpenFOAM.
And by my experience, GAMG doesn't always work for every case.

So in either case, it really depends on the case you're trying to solve and if single precision is enough! It's like running the case decomposed in parallel with CPUs only: the usual measure for estimated performance is to have at least 50k cells per core. But this is only for some usual case scenarios that some people have. It all depends on the equations that need to be solved and how well it can be done in parallel.

As for putting GPU into the fighting ring, it also depends on the balance between the number of calculations to be done per iteration vs the amount of data to be transferred between the CPU's (system) RAM and GPU's (graphics card) RAM.

Either way, that same blogger has an interesting comparison with Inria's CNC: http://www.geocities.co.jp/SiliconVa...M/CNC/CNC.html

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   February 9, 2011, 19:00
Default
  #15
Senior Member
 
Join Date: Mar 2009
Location: Austin, TX
Posts: 134
Rep Power: 9
kyle is on a distinguished road
wyldckat,

I am running single precision, which is fine for my use case. Thanks for the link, I did not know there was another OpenFOAM solver that ran on the GPU. It looks like it might be a little better than speedit.

And as far as copying between system RAM and GPU RAM, this should only happen at the beginning and end of the run. If you are exchanging the entire solution over the PCI-e bus every iteration then any and all advantage the GPU might have had is gone. Unfortunately this means you need to be able to fit the entire mesh and solution into GPU RAM, which if you have a decent sized case requires a lot of video cards!
kyle is offline   Reply With Quote

Old   February 14, 2011, 18:13
Default
  #16
New Member
 
Jakub Pola
Join Date: Feb 2011
Posts: 17
Rep Power: 6
stainboy is on a distinguished road
Hi,

I'm from the Speedit team and I hope that I might help here

I just finished the tests of Speedit Classic with OpenFoam 1.7.1. Everything works fine.
I have Ubuntu 10.10 32 bit with Cuda 3.2 on GTX 460 and gcc 4.4.4.
  1. I have compiled OpenFoam from sources with single precision flag ($WM_PRECISION_OPTION variable set to SP in OpenFOAM/etc/basrc file). Speedit Classic offer solvers only single precision calculations.
  2. Compiled Speedit Classic available from the webpage. http://speedit.vratis.com/
  3. Compiled SpeeitOpenFoam plugin 1.1. Here some additional changes.
  4. Because classic version of Speedit allows you to run only single precision calculations you can run only PCG_accel solver and functions with prefix sicl_ from si_classic.h file.
  5. In PCG_accel.C and PBiCG_accel.C uncomment speedit classic header
    Code:
    #include "si_classic.h"
    //#include "speedit_ex.h"
  6. To be sure in file PBiCG_accel.c comment all available solvers and uncoment FoamFatalError lines. Those solvers are from Extreme version
    Code:
    lines (213-239)
    /*
    	INSERT SOLVER FUNCTION HERE
    */
    #if defined(WM_SP)     // scalar is float
    
    Foam::FatalError << "Call your own solver from PBiCG_accel.C file" << Foam::exit(Foam::FatalError, -1) ;
    
    //solver_result = si_cscsrbicgstab( n_rows, 
    //						   p_vals, p_c_idx, p_r_idx, 
    // 			 		 	   X, B,
    // 						   precond,
    // 					 	  &n_iter, &eps) ;
    
    //solver_result = si_gscsrbicgstab( n_rows, 
    //						pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
    // 			 			pgpu_X, pgpu_B,
    //						precond,
    // 						&n_iter, &eps) ;
    
    #elif defined(WM_DP)   // scalar is double
    		
    Foam::FatalError << "Call your own solver from PBiCG_accel.C file" << Foam::exit(Foam::FatalError, -1) ;
    
    	//	solver_result = si_cdcsrbicgstab( n_rows, 
    	//						p_vals, p_c_idx, p_r_idx, 
    	// 			 			X, B,
    	//						precond,
    	// 						&n_iter, &eps) ;
    
    //solver_result = si_gdcsrbicgstab( n_rows, 
    //					pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
    //			 		pgpu_X, pgpu_B,
    //					precond,
    //					&n_iter, &eps) ;
  7. Go to PCG_accel.C and make solver from classic version available
    Code:
    line 208
    /*
    	INSERT SOLVER FUNCTION HERE
    */
    #if defined(WM_SP)     // scalar is float
    
    //Foam::FatalError << "Call your own solver from PCG_accel.C file" << Foam::exit(Foam::FatalError, -1) ;
    
    //solver_result = si_cscsrcg( n_rows, 
    //				p_vals, p_c_idx, p_r_idx, 
    // 			        X, B,
    //				precond,
    //	 			&n_iter, &eps) ;
    
    //solver_result = si_gscsrcg( n_rows, 
    //					pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
    //		 			pgpu_X, pgpu_B,
    //					precond,
    //		 			&n_iter, &eps) ;
    
    solver_result = sicl_gscsrcg( n_rows, 
    					pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
    					pgpu_X, pgpu_B,
    					precond,
    					&n_iter, &eps) ;
    
    #elif defined(WM_DP)   // scalar is double
    		
    Foam::FatalError << "Call your own solver from PCG_accel.C file" << Foam::exit(Foam::FatalError, -1) ;
    
    	//	solver_result = si_cdcsrcg( n_rows, 
    	//			p_vals, p_c_idx, p_r_idx, 
    	// 			X, B,
    	//		        precond,
    	//			&n_iter, &eps) ;
    
    //	solver_result = si_gdcsrcg( n_rows, 
    //				pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
    //				pgpu_X, pgpu_B,
    //				precond,
    //				&n_iter, &eps) ;
    
    #endif
  8. compile plugin by typing
    Code:
    ~/OpenFOAM/OpenFOAM-1.7.1/OpenFOAM_SpeedIT_plugin$ wmake libso
  9. Copy the libraries libSpeedIT_Classic.so libcublas.so libcudart.so
    to $FOAM_LIBBIN directory. In my case in $FOAM_LIBBIN among many files you should have also
    Code:
    /home/kuba/OpenFOAM/OpenFOAM-1.7.1/lib/linuxGccSPOpt/
    libexternalsolv.so
    libSpeedIT_Classic.so
    libcudart.so
    libcublas.so
  10. Now modify the fvSolution file from icoFoam/cavity/system folder. Change solvers part to:
    Code:
    solvers
    {
        p
        {
            solver              PCG_accel;
            preconditioner  diagonal;
            tolerance         1e-06;
            relTol              0;
        }
    
        U
        {
            solver             PBiCG;
            preconditioner  DILU;
            tolerance         1e-05;
            relTol               0;
        }
    }
  11. Add following entry at the end of the controlDict to tell OpenFoam which library are necessary to use with PCG_accel solver
    Code:
    runTimeModifiable yes;
    
    libs
        (
            "libcudart.so"
            "libcublas.so"
            "libSpeedIT_Classic.so"
            "libexternalsolv.so"
        );
  12. Run the blockMesh and then icoFoam.

    I hope that this post helped.
    Kuba.
stainboy is offline   Reply With Quote

Old   July 27, 2011, 05:05
Default it is helpful
  #17
New Member
 
Di.Lee
Join Date: Jun 2011
Posts: 1
Rep Power: 0
Di.Lee is on a distinguished road
The plugin works. Thank Jacub very much.
Di.Lee is offline   Reply With Quote

Old   November 25, 2011, 06:56
Default
  #18
New Member
 
RDG
Join Date: Feb 2011
Posts: 29
Rep Power: 6
onyir is on a distinguished road
Quote:
Originally Posted by andrea.pasquali View Post
Hi,
I'm coming back to use Speed IT + OpenFOAM plug in.
I installed the new CUDA toolkit 3.2, the Speed IT 1.0, and the OpenFOAM plug in 1.0.
When I run the "icoFoam" test with PCG_accel I have:

--> FOAM FATAL ERROR:
Can not allocate GPU memory

FOAM exiting

I don't know where is the probem,
could anyone help me ?

Thanks

Andrea

Hi, Andrea, how did you manage to solve that issue? I'm having the same problem.

Thanks.
onyir is offline   Reply With Quote

Old   May 7, 2012, 05:00
Default
  #19
Member
 
Lukasz Miroslaw
Join Date: Dec 2009
Location: Poland
Posts: 64
Rep Power: 7
Lukasz is on a distinguished road
Send a message via Skype™ to Lukasz
We are about to release a new version of the OpenFOAM plugin where you could run SpeedIT Classic as well as SpeedIT 2.1/Multi-GPU in case of larger cases that do not fit into one GPU card. The new release should solve the problem.
Lukasz is offline   Reply With Quote

Old   August 9, 2012, 11:04
Default Some news ?
  #20
New Member
 
Matthieu Borgraeve
Join Date: Aug 2012
Posts: 17
Rep Power: 4
mborgraeve is on a distinguished road
Hello !
I have at least one question :
I am trying to use SpeedIT classic with the plugin2.1 for some benchmarks, and i don't manage to use the accel_solvers : the compilation of the plug-in crashes, and tell me he is missing some vector_type.h file.
I'm running on Kubuntu 12.04 64b fully up-to-date, installed cuda and OpenFoam managing an error concerning binutils and binutils-gold which give two version of l, each compile and run without problem)
Has anybody try or manage to use a recent version of it ?
Best regards,

EDIT Actually i found : i have to make a Allwmake FREE instead of COMMERCIAL

Last edited by mborgraeve; August 9, 2012 at 11:28. Reason: answered !
mborgraeve is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
OpenFOAM Feasibility... JasonG OpenFOAM 9 June 4, 2010 17:28
OpenFoam & Sobolev spaces Dan Main CFD Forum 4 October 4, 2008 15:50


All times are GMT -4. The time now is 00:52.