SU2 and ParMETIS

bensky · February 4, 2014, 11:34

Greetings,

I am running into a memory limit when running SU2_DDC on a ~40M cell grid (SU2 version 3.0). If I understand correctly, because SU2_DDC/METIS only runs serially, there is an upper bound for the allocated RAM per CPU and thus size of grid to be decomposed. Here is the error I get when SU2_DDC is called:

------------------------ Divide the numerical grid ----------------------
[METIS Fatal Error] ***Memory allocation failed for METIS_MESHPARTNODAL: adjncy. Requested size: -1385723056 bytes

A 12M cell grid partitions and solves without issue. I have heard that it may be possible to utilize ParMETIS instead of METIS to parallelize the partitioning process in SU2_DDC? If so, is there any instructions/documentation on how to build SU2 with ParMETIS?

Thanks in advance and best regards,

Ben

fpalacios · February 4, 2014, 20:58

Hi Ben,

You are right. It seems that you don't have enough memory in your master node to perform the partition (we have done larger simulations but with more memory in the master node).

ParMETIS is a future plan for SU2, but we are not still there. So, at this point the quickest thing that you can do is to use a machine with more RAM memory. Another possibility, if the grid is easy to divide, is to bypass METIS and implement a geometry based grid partitioner, or run METIS several times in different parts of the domain.

The call to METIS is very simple (void CPhysicalGeometry::SetColorGrid(CConfig *config)
in geometry_structure.cpp)… you will see that METIS is nothing more than a method to color the grid, so you can modify the subroutine to reduce the size the grid that METIS is using.

Thanks for using Su2,
Best,
Francisco

February 4, 2014, 11:34	SU2 and ParMETIS	#1
bensky New Member Ben N. Join Date: Dec 2012 Location: San Jose, CA Posts: 1 Rep Power: 0	Greetings, I am running into a memory limit when running SU2_DDC on a ~40M cell grid (SU2 version 3.0). If I understand correctly, because SU2_DDC/METIS only runs serially, there is an upper bound for the allocated RAM per CPU and thus size of grid to be decomposed. Here is the error I get when SU2_DDC is called: ------------------------ Divide the numerical grid ---------------------- [METIS Fatal Error] ***Memory allocation failed for METIS_MESHPARTNODAL: adjncy. Requested size: -1385723056 bytes A 12M cell grid partitions and solves without issue. I have heard that it may be possible to utilize ParMETIS instead of METIS to parallelize the partitioning process in SU2_DDC? If so, is there any instructions/documentation on how to build SU2 with ParMETIS? Thanks in advance and best regards, Ben

February 4, 2014, 20:58		#2
fpalacios Super Moderator Francisco Palacios Join Date: Jan 2013 Location: Long Beach, CA Posts: 404 Rep Power: 15	Hi Ben, You are right. It seems that you don't have enough memory in your master node to perform the partition (we have done larger simulations but with more memory in the master node). ParMETIS is a future plan for SU2, but we are not still there. So, at this point the quickest thing that you can do is to use a machine with more RAM memory. Another possibility, if the grid is easy to divide, is to bypass METIS and implement a geometry based grid partitioner, or run METIS several times in different parts of the domain. The call to METIS is very simple (void CPhysicalGeometry::SetColorGrid(CConfig *config) in geometry_structure.cpp)… you will see that METIS is nothing more than a method to color the grid, so you can modify the subroutine to reduce the size the grid that METIS is using. Thanks for using Su2, Best, Francisco