ParMETIS
 

I have been using METIS for years in the front end of my face based cell centered unstructured CFD code. The front end reads the grid gereator file, builds the global vertex and adjacency list and runs METIS_PartGraph_Kway.

I have a large model now and METIS is exceeding the 32GB ram I have on our "fat" serial node, so I think I need to use ParMETIS.

Question is how does one "easily" build a starting graph for each processor? Simple spatial splitting? Put a edges each on n-1 procs and the rest on one proc?

Thanks! Rob

Re: ParMETIS
 

How do you generate your huge meshes? Ideally the mesh generator provides a preliminary partition and you distribute parts based on that. If you can't have a preliminary partition, you can just assign a contiguous block of <code>n/p</code> elements to each process as a preliminary partition. Of course, this requires that your file format gives you a way to determine connectivity for the elements in a part without reading the entire thing. You can use MPI-IO (perhaps through an interface like HDF5), read sequentially from the first node, or read separate files on each node. Which is fastest depends on the filesystem and network of your cluster.

Re: ParMETIS
 

Thanks Jed: I use ICEM which will not partition the mesh. Whatever my solution I qwill be faced with the I/O issue you mention and I will take your advise and look into MPI-IO Thank you Rob