CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Main CFD Forum

How to optimize the memory usage when using FEM

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   January 15, 2009, 09:15
Default How to optimize the memory usage when using FEM
  #1
vasilis
Guest
 
Posts: n/a
Hi,

I am developing a 3D code utilizing the FEM. I am also using MPI.

Because the problem is 3D, my biggest problem is memory consumption and cpu time. The latter problem is partially taken care of with MPI.

My question is how to deal with the memory consumption. Currently, I have 28000 elements (755920 unknowns) and the code takes about 5GB of memory. Although this may seems to be a large number of elements, I need to further increase the number of elements. This amount of memory is used by the master CPU, the other CPUs are using about 150MB of memory.

Up to now, I am breaking the domain into pieces, and each CPU is solving the eqns on a portion of the domain. Then, I add all these matrices up without worrying about duplicated entries. Because of that, my code consumes a lot of memory.

I tried to fix the aforementioned problem by searching for duplicated entries in the global matrix. Although this has reduced the amount of required memory by a factor of 3, it takes for ever to construct the global matrix.

Does anybody know any other method to deal with this problem?

Thank you, Vasilis

  Reply With Quote

Old   January 15, 2009, 10:34
Default Re: How to optimize the memory usage when using FE
  #2
Tom
Guest
 
Posts: n/a
It sounds like you need to get rid of the master CPU (if I've understood correctly that the master is using a lot of memory while the others are not). Basically you should avoid having a master cpu in MPI in general - for optimal performance all cpu's should be doing exactly the same amount of work.
  Reply With Quote

Old   January 15, 2009, 10:34
Default Re: How to optimize the memory usage when using FE
  #3
Peter Attar
Guest
 
Posts: n/a
Are you are only storing the nonzero coefficients of your stiffness matrix? It would appear that you are not. You should be taking advantage of the sparsity of the stiffness matrix.

  Reply With Quote

Old   January 15, 2009, 10:38
Default Re: How to optimize the memory usage when using FE
  #4
vasilis
Guest
 
Posts: n/a
I use the master CPU to store the global matrix. This matrix requires a lot of memory.
  Reply With Quote

Old   January 15, 2009, 10:42
Default Re: How to optimize the memory usage when using FE
  #5
vasilis
Guest
 
Posts: n/a
Yes, I am storing only the non-zero coefficients. The global matrix requires a lot of memory because I have duplicated entries. Now, I am developing a code that deals with these duplicated entries, but it is extremely slow. It is running for about 50min, and it's not finished.
  Reply With Quote

Old   January 15, 2009, 12:17
Default Re: How to optimize the memory usage when using FE
  #6
Jed
Guest
 
Posts: n/a
He didn't say the matrix used 5 TB.
  Reply With Quote

Old   January 15, 2009, 12:23
Default Re: How to optimize the memory usage when using FE
  #7
Jed
Guest
 
Posts: n/a
The matrix should also be distributed. It's really a lot of work to do a good job of this. I recommend using PETSc to manage the parallel data structures and solvers. This way, you distribute the mesh and call a function to insert values into the matrix. At the end of the assembly, you call a function which performs necessary communication. This scales well (10^10 dofs, 10^5 processors).
  Reply With Quote

Old   January 16, 2009, 01:38
Default Re: How to optimize the memory usage when using FE
  #8
vasilis
Guest
 
Posts: n/a
The matrix is distributed, but the master cpu assembles the global matrix. I am solving the eqns using MUMPS, which is a parallel solver.

PETSc sounds like a promising program, I wander how difficult is to use it, though
  Reply With Quote

Old   January 16, 2009, 03:17
Default Re: How to optimize the memory usage when using FE
  #9
Jed
Guest
 
Posts: n/a
The matrix should never reside on one process. You should assemble for the owned elements on each process. PETSc can use MUMPS, but it can also use lots of other preconditioners (including BoomerAMG and ML). It is much easier to use PETSc than to not use PETSc. Read through the relevant sections of the manual and look at some examples.
  Reply With Quote

Old   January 16, 2009, 08:17
Default Re: How to optimize the memory usage when using FE
  #10
vasilis
Guest
 
Posts: n/a
I was assembling the local matrices to a global matrix (which demanded a huge amount of memory), when in fact MUMPS could solve the system with the local matrices. Surprisingly enough, now that I am using the local matrices its faster!!

Thank you all for your help!
  Reply With Quote

Old   January 16, 2009, 12:32
Default Re: How to optimize the memory usage when using FE
  #11
jughead
Guest
 
Posts: n/a
Use UMFPACK.
  Reply With Quote

Old   August 24, 2009, 23:57
Default Mumps
  #12
cra
New Member
 
Channa
Join Date: Aug 2009
Posts: 1
Rep Power: 0
cra is on a distinguished road
Hi Vasilis
I use a 3D Ė FEM model (written in FORTRAN 90). The problem is it can take awful long time to run for large spatial-temporal cases. Some instances the solver can take up to 90-95% of the run time. Iím keen to use a parallel solver to overcome this issue. I had a look at the MUMPS and think that can help. But Iím not a programmer and bit hard to get my head around how to go about this. As youíve used the MUMPS with your FEM model (and MPI) just wonder whether you can advise me on how to do this. Is there any chance of giving me your source code so that I can see how should I modify my code?
Thank you in advance
cra is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Estimation for memory usage by ICEM ftab Hardware 5 September 19, 2011 04:32
Fluent Parallel Memory Usage David Christopher FLUENT 1 March 3, 2011 02:20
How to increase memory usage by fluent Anastasios Georgoulas FLUENT 2 December 29, 2008 05:41
CFX CPU time & real time Nick Strantzias CFX 8 July 23, 2006 17:50
comments on FDM, FEM, FVM, SM, SEM, DSEM, BEM kenn Main CFD Forum 2 July 18, 2004 18:28


All times are GMT -4. The time now is 02:59.