CFD Online Discussion Forums

CFD Online Discussion Forums (
-   Main CFD Forum (
-   -   PETSc Library (

kostaspan July 1, 2010 06:24

PETSc Library
I have installed PETSc libraries on Linux platform for solution of large sparse linear systems which derived by an implicit scheme at finite element code.
The installation integrated successful but i can use it.

I put as includes in C language #include"petscksp.h" but when i try to compile the program, i take errors. If someone know to tell me what flags must i write for compilation and what else muct be specified for right reading of all petsc libraries.

jed July 2, 2010 22:11

If the errors are about the missing header, you probably need -I$PETSC_DIR/include -I$PETSC_DIR/$PETSC_ARCH/include. One way to do this is to use PETSc makefiles (which can also handle linking). If it's something else, please send mail to (you will get a prompt reply).

kostaspan July 3, 2010 10:48

I use c of gcc. At ./configuration i put and --with-c-support. Its ok this for c of gcc?

jed July 5, 2010 02:30

You can use --with-cc=gcc, but it will usually pick up this compiler anyway. You don't need --with-c-support, that is only if you are compiling PETSc with a C++ compiler, but want to be able to call PETSc from C.

kostaspan July 6, 2010 03:21

Yes i put --with-c-support. Do you thing that i have problem with commands in code or compiling?
Now i reinstall --with-cc=gcc.

I want to ask you about functions for matrix construction. When i use MatSetValues function with very large matrix it take me so long time for finish. And is supposed that these constructions must be work quickly that to use in finite element code with very large systems. Have you any ideas for this? What i do?

jed July 6, 2010 11:19

MatSetValues is slow because you didn't preallocate correctly.

kostaspan July 6, 2010 13:24

Ok. I do some things which i read. But with 5000x5000 elements in matrix its take again much time. Below i give sou the construct of this matrix and please tell me what i do wrong. I have already lost a lot of days for this problem :(.
Below A: matrix , col3 : pointer to int for number of nonzeros at each row.

MatCreateSeqAIJ(PETSC_COMM_SELF,4000,4000,4000,col 3,&A);




More time take first function.

jed July 6, 2010 14:05

Looks like you have a dense matrix containing lots of explicit zeros, is possible that they are just very close to zero (but not exactly zero so they still get inserted)? The dimensions 4000x4000 are not at all large. How many nonzeros do you expect (what is in col3)? Run with -info | grep malloc, the allocation is almost certainly insufficient. To generate an error when inserting a value that has not been allocated, call

after creating the matrix, but before inserting values.

kostaspan July 6, 2010 16:53

Is there any function which convert a allocated pointer to Mat type matrix? Why to allocate memory for a matrix which i have already made in pointer (ex. double * or float *) ?

I did all above which you tell me but it takes same time...

jed July 6, 2010 16:59

Were there any mallocs during insertion? How long does it take for a particular size? If you have a dense matrix, you can use MatCreateSeqDense() and pass in your array. But it doesn't make sense to hold a sparse matrix with dense storage, usually people create the matrix using MatSetValues rather than assembling into their own structure and copying over.

kostaspan July 6, 2010 17:06

How many elements in matrix have you solved with petsc? I will try again other functions and i send you if i have problem.

Thanks for help

jed July 6, 2010 17:10

Matrix dimensions of a few hundred million, others have done 500 billion, these matrices usually have from a few 10s to 100s of nonzeros per row. Usually memory limits single-process matrix dimensions to a couple million for 3D problems, a bit more in 2D (many more nonzeros per row in 3D). It would be helpful if you describe what you are trying to do (e.g. where the matrix comes from and what you want to do with it).

kostaspan July 6, 2010 17:37

I have made code for spectral finite element analysis (DG method) in 3D for diffusion-convection equation. Until now i used only LU Dec direct method for steady or unsteady (explicit ) schemes because the DG method has an advantage of iterative procedure (i have many small systems with same left side at each element).
Before five days i make the implicit for above equation which requires the creation of stiffness matrix which include all DOFs of all elements. This matrix(by which we discuss) remain const for all time steps but change RHS where you know.
Therefore, at each time step i can solute with GMRES the system for to take the solution at next time. This matrix for 40000 elements and second order approximation (p2) become ~4000000 X4000000. But i here there is the problem. It take much time for allocate memory the petsc lib. :( I dont have such problems with malloc of c. It work quickly.
And documentation of Petsc dosnt give understandable things... If you can tell me which functions and which way you use...


proximityinfotech6 July 7, 2010 07:47

I put as includes in C language #include"petscksp.h" but when i try to compile the program, i take errors. If someone know to tell me what flags must i write for compilation and what else muct be specified for right reading of all petsc libraries. E20 fuel, which blends 20 percent ethanol with gasoline, reduces the tail pipe emissions of hydrocarbons and carbon monoxide, compared with traditional gasoline or E10 blends, according to a new study in the Journal of Automobile Engineering. In addition, the research found no measurable impact to vehicle drivability or maintenance in conventional internal combustion engines.
Used Auto Parts | Used Car Parts

jed July 7, 2010 08:08

You still haven't answered my question about number of mallocs during insertion, nor told me how slow is "slow". Your matrix dimension of 4M doesn't make sense, 40k Q_2 hex elements produces 27*40k=1.08M, 40k P_2 tet elements produces 10*40k=400k, both of which are significantly smaller than 4M. Additionally, your quoted code is clearly extracting from a dense matrix, which you certainly can't have for large sizes (and doesn't make sense for this problem). Of course creating a dense matrix bigger than available memory will cause lots of swapping and be tremendously slow. You should MatCreateSeqAIJ (or more generically, see the user's manual) with correct precallocation (the number of nonzeros per row depends on which elliptic flux you use) or at least an upper bound (e.g. 27*7 for structured hexes with a local flux), then loop over elements and faces, computing local matrices and inserting them.

kostaspan July 7, 2010 12:40

I did relatively many mallocs because i havent completed the optimization. I usually do this in order to have easy procedures and structures as far as possible. Now i correct this and the only values which saved for procedure are these in Sparse Mat . Of course, only non-zeros it must be saved. Finally, i found the petsc functions for efficient creation Mat. I give these lines below:
Therefore all values as blocks of size (sizeb) saved in A which have preallocation memory for only non-zeros blocks.

MatCreateSeqBAIJ(PETSC_COMM_SELF,sizeb,rows,cols,8 ,0,&A);

MatSetValuesBlocked(A,1,&cola,1,&colb,ImpEline,INS ERT_VALUES);




With this i run 100.000 x100000 matrices in a few sec...

All times are GMT -4. The time now is 10:39.