[jollage]

Hi all,

I'm thinking to write a DNS code using finite difference scheme using OpenMPI. I am wondering what kind of the parallel strategy is the best?

[sbaffini]

What do you mean by parallel strategy? Once you select OpenMPI (which actually is an MPI implementation... or maybe you meant openmp?) then there is not so much left to choose in the parallel framework.

[jollage]

Originally Posted by sbaffini

What do you mean by parallel strategy? Once you select OpenMPI (which actually is an MPI implementation... or maybe you meant openmp?) then there is not so much left to choose in the parallel framework.

Hi sbaffini, thanks for your comments. By parallel strategy, I mean the way how people partition the computational domain. Like, I have a nx*ny rectangular domain and I use finite difference method. The computational domain is partitioned as (nx/N)*ny for each processor (N in total) when I take the derivative in the y direction or nx*(ny/N) when I take the derivative in the x direction. I have to transform the computational grids in each processor in either x or y direction.

This is what I learned from the code I currently have. It seems that the scalability of my code is not good (when the number of the processors exceeding 5, the gain is not so much). I'm wondering if there is any other way to do the same thing.

Thanks.

[mprinkey]

If you are using classic finite differences, there is no need to transpose the data to calculation approximate derivatives. You need to maintain ghost layers of data on each PE sufficient to feed the finite difference stencil. This is very basic MPI coding. I recommend that you go read the thoroughly comprehend the Jacobi iteration examples in the MPI Exercises here:

http://www.mcs.anl.gov/research/proj.../contents.html

That example uses Jacobi iteration to solve the 2D difference equations approximating a 2D Laplace equation. The same strategy applies to any finite difference scheme with a 5-point stencil. And the Jacobi iteration code can be trivially turned into the RHS calculation/time-step update for a transient calculation. The extension to 3D is obvious. If you are using high-order finite differences or multi-point upwinding schemes, you may need to add a second or even third ghost layer, but the essential strategy is the same.

[jollage]

Originally Posted by mprinkey

If you are using classic finite differences, there is no need to transpose the data to calculation approximate derivatives. You need to maintain ghost layers of data on each PE sufficient to feed the finite difference stencil. This is very basic MPI coding. I recommend that you go read the thoroughly comprehend the Jacobi iteration examples in the MPI Exercises here:

http://www.mcs.anl.gov/research/proj.../contents.html

That example uses Jacobi iteration to solve the 2D difference equations approximating a 2D Laplace equation. The same strategy applies to any finite difference scheme with a 5-point stencil. And the Jacobi iteration code can be trivially turned into the RHS calculation/time-step update for a transient calculation. The extension to 3D is obvious. If you are using high-order finite differences or multi-point upwinding schemes, you may need to add a second or even third ghost layer, but the essential strategy is the same.

Thanks a lot mprinkey, I'll look into it. I'll come back to this when I get stuck.