CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   Main CFD Forum (http://www.cfd-online.com/Forums/main/)
-   -   How to optimize the memory usage when using FEM (http://www.cfd-online.com/Forums/main/16215-how-optimize-memory-usage-when-using-fem.html)

vasilis January 15, 2009 10:15

How to optimize the memory usage when using FEM
 
Hi,

I am developing a 3D code utilizing the FEM. I am also using MPI.

Because the problem is 3D, my biggest problem is memory consumption and cpu time. The latter problem is partially taken care of with MPI.

My question is how to deal with the memory consumption. Currently, I have 28000 elements (755920 unknowns) and the code takes about 5GB of memory. Although this may seems to be a large number of elements, I need to further increase the number of elements. This amount of memory is used by the master CPU, the other CPUs are using about 150MB of memory.

Up to now, I am breaking the domain into pieces, and each CPU is solving the eqns on a portion of the domain. Then, I add all these matrices up without worrying about duplicated entries. Because of that, my code consumes a lot of memory.

I tried to fix the aforementioned problem by searching for duplicated entries in the global matrix. Although this has reduced the amount of required memory by a factor of 3, it takes for ever to construct the global matrix.

Does anybody know any other method to deal with this problem?

Thank you, Vasilis


Tom January 15, 2009 11:34

Re: How to optimize the memory usage when using FE
 
It sounds like you need to get rid of the master CPU (if I've understood correctly that the master is using a lot of memory while the others are not). Basically you should avoid having a master cpu in MPI in general - for optimal performance all cpu's should be doing exactly the same amount of work.

Peter Attar January 15, 2009 11:34

Re: How to optimize the memory usage when using FE
 
Are you are only storing the nonzero coefficients of your stiffness matrix? It would appear that you are not. You should be taking advantage of the sparsity of the stiffness matrix.


vasilis January 15, 2009 11:38

Re: How to optimize the memory usage when using FE
 
I use the master CPU to store the global matrix. This matrix requires a lot of memory.

vasilis January 15, 2009 11:42

Re: How to optimize the memory usage when using FE
 
Yes, I am storing only the non-zero coefficients. The global matrix requires a lot of memory because I have duplicated entries. Now, I am developing a code that deals with these duplicated entries, but it is extremely slow. It is running for about 50min, and it's not finished.

Jed January 15, 2009 13:17

Re: How to optimize the memory usage when using FE
 
He didn't say the matrix used 5 TB.

Jed January 15, 2009 13:23

Re: How to optimize the memory usage when using FE
 
The matrix should also be distributed. It's really a lot of work to do a good job of this. I recommend using PETSc to manage the parallel data structures and solvers. This way, you distribute the mesh and call a function to insert values into the matrix. At the end of the assembly, you call a function which performs necessary communication. This scales well (10^10 dofs, 10^5 processors).

vasilis January 16, 2009 02:38

Re: How to optimize the memory usage when using FE
 
The matrix is distributed, but the master cpu assembles the global matrix. I am solving the eqns using MUMPS, which is a parallel solver.

PETSc sounds like a promising program, I wander how difficult is to use it, though

Jed January 16, 2009 04:17

Re: How to optimize the memory usage when using FE
 
The matrix should never reside on one process. You should assemble for the owned elements on each process. PETSc can use MUMPS, but it can also use lots of other preconditioners (including BoomerAMG and ML). It is much easier to use PETSc than to not use PETSc. Read through the relevant sections of the manual and look at some examples.

vasilis January 16, 2009 09:17

Re: How to optimize the memory usage when using FE
 
I was assembling the local matrices to a global matrix (which demanded a huge amount of memory), when in fact MUMPS could solve the system with the local matrices. Surprisingly enough, now that I am using the local matrices its faster!!

Thank you all for your help!

jughead January 16, 2009 13:32

Re: How to optimize the memory usage when using FE
 
Use UMFPACK.

cra August 24, 2009 23:57

Mumps
 
Hi Vasilis
I use a 3D Ė FEM model (written in FORTRAN 90). The problem is it can take awful long time to run for large spatial-temporal cases. Some instances the solver can take up to 90-95% of the run time. Iím keen to use a parallel solver to overcome this issue. I had a look at the MUMPS and think that can help. But Iím not a programmer and bit hard to get my head around how to go about this. As youíve used the MUMPS with your FEM model (and MPI) just wonder whether you can advise me on how to do this. Is there any chance of giving me your source code so that I can see how should I modify my code?
Thank you in advance


All times are GMT -4. The time now is 17:36.