Parallel refineMesh with Cyclic Boundaries
 

I want to make a really large block mesh with uniform grid resolution throughout and cyclic boundaries. When I say really large, I mean hundreds of millions or a billion cells. My machines do not have enough memory to run blockMesh and decomposePar, both of which are serial executable only.

Therefore, I have been experimenting with using blockMesh to make a coarse mesh, decomposing that, and then using refineMesh over the entire domain in parallel to refine each piece of the mesh to create the really large mesh. This makes the memory demands on blockMesh and decomposePar smaller. refineMesh is run in parallel, so memory is distributed over a lot of nodes making the job manageable.

The problem is when there are cyclic boundaries. refineMesh seems to not handle cyclic boundaries in parallel. Any suggestions on what I'm trying to do?

Greetings Matthew,

I'm sorry for not going into details, but here goes:

1. Preserve cyclic boundaries: Cyclic patches and parallel postprocessing problems post #8
2. Now I can't remember very well, but I vaguely remember something about tolerance for compatibility between cyclic patches... maybe this one can shed some light: http://www.cfd-online.com/Forums/ope...tml#post276931 post #2
3. Here's another, this time a whole thread dedicated to this: http://www.cfd-online.com/Forums/ope...lic-patch.html

For more, search the forum for something like "tolerance cyclic patches".

Good luck!
Bruno

Hi, Matthew.

Even you can refine the mesh using 'refineMesh' with parallel computing, presumably you still need to construct the subdomains in the end using 'reconstructPar'. and this is not running in parallel either as far as I know.

I am also strugling with deomposing and reconstructing large mesh. So I am wondering that you have an alternative solution to deal with a large mesh? (apart from buying more memory).
In my case, I can use upto 35 million cells with 48gb memory.
Regards

Yusik

A Working Method for Creating Large Meshes in Parallel
 

Yusik,

I do have a method that works now. I create a coarse mesh with blockMesh. Then I decompose the domain with decomposePar, making sure to list the cyclic boundaries in the "preservePatches" keyword. Then I run refineMesh or refineHexMesh in parallel until I get to the desired grid resolution. I do not have the computer memory to reconstruct the domain with reconstructPar. I deal with that by sampling slices, probes, etc. while the solver is running. If I really need to explore the full 3D flow field, Paraview's OpenFOAM reader will read in each processor directory and display the flow field as one contiguous field. You need to copy the system/controlDict to system/controlDict.foam. After you open Paraview, you choose the controlDict.foam file as the file to open, which invokes the OpenFOAM reader. Then you'll have the option to view the decomposed domain (where each processor directory is read in). If you have a very large domain that requires a lot of memory, then you can launch Paraview in parallel to spread the memory requirements over multiple compute nodes.

Matt

Thank you for quick response. and I also agree that will be a good solution.
I did a quick test as you suggested and I have a related question.
I decomposed a coarse mesh into subdomains and then I ran

topoSet -parallel (for local refinement)
setsToZones -noFlipMap -parallel
refineMesh -dict -parallel

upto here, there is no problem. Then I realized that the refined mesh is
generated in a new time step within subdomains, e.g.
processor#/'newtimestep'/polyMesh

This means that the mesh is refined and decomposed but the fields
(U, p etc) are not. they remain the same in the old time step folder. So I am
wondering if there is a way to refine fields together with mesh? I've tried
to use couple of options such as '-overwrite' but it hasn't been successful so far.
Thank you in advance
Yusik

Yusik,

Yes, try using refineHexMesh. It is the newer refinement tool, and it will refine the solution fields, too. It's use is similar to refineMesh.

Matt

Thanks for your valuable comments.
I forgot to mention this.

"'reconstructPar'. and this is not running in parallel either as far as I know." : So if we reconstructPar in parallel and then what we will have? have to reconstructPar in single processor to merge data together again?

Quick answer: reconstructPar is sort-of the opposite of decomposePar:

• decomposePar is meant to run in serial mode (not parallel) and it will decompose the domain into subdomains, so that the case can then be run in parallel; it decomposes both the mesh and the fields.
• reconstructPar is also meant to run in serial mode and it will reconstruct the subdomains back into the main domain, but it will only reconstruct the fields and a few other things. But it will not reconstruct the mesh, for that you must use reconstructParMesh.