CFD Online Discussion Forums

CFD Online Discussion Forums (
-   OpenFOAM (
-   -   Parallel refineMesh with Cyclic Boundaries (

mchurchf July 20, 2011 14:46

Parallel refineMesh with Cyclic Boundaries
I want to make a really large block mesh with uniform grid resolution throughout and cyclic boundaries. When I say really large, I mean hundreds of millions or a billion cells. My machines do not have enough memory to run blockMesh and decomposePar, both of which are serial executable only.

Therefore, I have been experimenting with using blockMesh to make a coarse mesh, decomposing that, and then using refineMesh over the entire domain in parallel to refine each piece of the mesh to create the really large mesh. This makes the memory demands on blockMesh and decomposePar smaller. refineMesh is run in parallel, so memory is distributed over a lot of nodes making the job manageable.

The problem is when there are cyclic boundaries. refineMesh seems to not handle cyclic boundaries in parallel. Any suggestions on what I'm trying to do?

wyldckat July 20, 2011 18:22

Greetings Matthew,

I'm sorry for not going into details, but here goes:
  1. Preserve cyclic boundaries: Cyclic patches and parallel postprocessing problems post #8
  2. Now I can't remember very well, but I vaguely remember something about tolerance for compatibility between cyclic patches... maybe this one can shed some light: post #2
  3. Here's another, this time a whole thread dedicated to this:
For more, search the forum for something like "tolerance cyclic patches".

Good luck!

impecca June 6, 2012 11:49

Hi, Matthew.

Even you can refine the mesh using 'refineMesh' with parallel computing, presumably you still need to construct the subdomains in the end using 'reconstructPar'. and this is not running in parallel either as far as I know.

I am also strugling with deomposing and reconstructing large mesh. So I am wondering that you have an alternative solution to deal with a large mesh? (apart from buying more memory).
In my case, I can use upto 35 million cells with 48gb memory.


mchurchf June 6, 2012 12:19

A Working Method for Creating Large Meshes in Parallel

I do have a method that works now. I create a coarse mesh with blockMesh. Then I decompose the domain with decomposePar, making sure to list the cyclic boundaries in the "preservePatches" keyword. Then I run refineMesh or refineHexMesh in parallel until I get to the desired grid resolution. I do not have the computer memory to reconstruct the domain with reconstructPar. I deal with that by sampling slices, probes, etc. while the solver is running. If I really need to explore the full 3D flow field, Paraview's OpenFOAM reader will read in each processor directory and display the flow field as one contiguous field. You need to copy the system/controlDict to system/controlDict.foam. After you open Paraview, you choose the controlDict.foam file as the file to open, which invokes the OpenFOAM reader. Then you'll have the option to view the decomposed domain (where each processor directory is read in). If you have a very large domain that requires a lot of memory, then you can launch Paraview in parallel to spread the memory requirements over multiple compute nodes.


impecca June 6, 2012 16:47

Thank you for quick response. and I also agree that will be a good solution.
I did a quick test as you suggested and I have a related question.
I decomposed a coarse mesh into subdomains and then I ran

topoSet -parallel (for local refinement)
setsToZones -noFlipMap -parallel
refineMesh -dict -parallel

upto here, there is no problem. Then I realized that the refined mesh is
generated in a new time step within subdomains, e.g.

This means that the mesh is refined and decomposed but the fields
(U, p etc) are not. they remain the same in the old time step folder. So I am
wondering if there is a way to refine fields together with mesh? I've tried
to use couple of options such as '-overwrite' but it hasn't been successful so far.
Thank you in advance

mchurchf June 7, 2012 09:31


Yes, try using refineHexMesh. It is the newer refinement tool, and it will refine the solution fields, too. It's use is similar to refineMesh.


impecca June 20, 2012 13:06

Thanks for your valuable comments.
I forgot to mention this.

All times are GMT -4. The time now is 04:03.