CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

Parallel refineMesh with Cyclic Boundaries

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree5Likes
  • 1 Post By wyldckat
  • 2 Post By mchurchf
  • 2 Post By mchurchf

Reply
 
LinkBack Thread Tools Display Modes
Old   July 20, 2011, 14:46
Default Parallel refineMesh with Cyclic Boundaries
  #1
Member
 
Matthew J. Churchfield
Join Date: Nov 2009
Location: Boulder, Colorado, USA
Posts: 49
Rep Power: 8
mchurchf is on a distinguished road
I want to make a really large block mesh with uniform grid resolution throughout and cyclic boundaries. When I say really large, I mean hundreds of millions or a billion cells. My machines do not have enough memory to run blockMesh and decomposePar, both of which are serial executable only.

Therefore, I have been experimenting with using blockMesh to make a coarse mesh, decomposing that, and then using refineMesh over the entire domain in parallel to refine each piece of the mesh to create the really large mesh. This makes the memory demands on blockMesh and decomposePar smaller. refineMesh is run in parallel, so memory is distributed over a lot of nodes making the job manageable.

The problem is when there are cyclic boundaries. refineMesh seems to not handle cyclic boundaries in parallel. Any suggestions on what I'm trying to do?
mchurchf is offline   Reply With Quote

Old   July 20, 2011, 18:22
Default
  #2
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,516
Blog Entries: 34
Rep Power: 86
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Matthew,

I'm sorry for not going into details, but here goes:
  1. Preserve cyclic boundaries: Cyclic patches and parallel postprocessing problems post #8
  2. Now I can't remember very well, but I vaguely remember something about tolerance for compatibility between cyclic patches... maybe this one can shed some light: Defining "cyclic" B.C. problem post #2
  3. Here's another, this time a whole thread dedicated to this: simpleSRFoam, makePatch and cyclic patch
For more, search the forum for something like "tolerance cyclic patches".

Good luck!
Bruno
Pagoda likes this.
wyldckat is offline   Reply With Quote

Old   June 6, 2012, 11:49
Default
  #3
Member
 
Kim Yusik
Join Date: Dec 2009
Posts: 39
Rep Power: 7
impecca is on a distinguished road
Hi, Matthew.

Even you can refine the mesh using 'refineMesh' with parallel computing, presumably you still need to construct the subdomains in the end using 'reconstructPar'. and this is not running in parallel either as far as I know.

I am also strugling with deomposing and reconstructing large mesh. So I am wondering that you have an alternative solution to deal with a large mesh? (apart from buying more memory).
In my case, I can use upto 35 million cells with 48gb memory.
Regards

Yusik
impecca is offline   Reply With Quote

Old   June 6, 2012, 12:19
Default A Working Method for Creating Large Meshes in Parallel
  #4
Member
 
Matthew J. Churchfield
Join Date: Nov 2009
Location: Boulder, Colorado, USA
Posts: 49
Rep Power: 8
mchurchf is on a distinguished road
Yusik,

I do have a method that works now. I create a coarse mesh with blockMesh. Then I decompose the domain with decomposePar, making sure to list the cyclic boundaries in the "preservePatches" keyword. Then I run refineMesh or refineHexMesh in parallel until I get to the desired grid resolution. I do not have the computer memory to reconstruct the domain with reconstructPar. I deal with that by sampling slices, probes, etc. while the solver is running. If I really need to explore the full 3D flow field, Paraview's OpenFOAM reader will read in each processor directory and display the flow field as one contiguous field. You need to copy the system/controlDict to system/controlDict.foam. After you open Paraview, you choose the controlDict.foam file as the file to open, which invokes the OpenFOAM reader. Then you'll have the option to view the decomposed domain (where each processor directory is read in). If you have a very large domain that requires a lot of memory, then you can launch Paraview in parallel to spread the memory requirements over multiple compute nodes.

Matt
wyldckat and mgg like this.
mchurchf is offline   Reply With Quote

Old   June 6, 2012, 16:47
Default
  #5
Member
 
Kim Yusik
Join Date: Dec 2009
Posts: 39
Rep Power: 7
impecca is on a distinguished road
Thank you for quick response. and I also agree that will be a good solution.
I did a quick test as you suggested and I have a related question.
I decomposed a coarse mesh into subdomains and then I ran

topoSet -parallel (for local refinement)
setsToZones -noFlipMap -parallel
refineMesh -dict -parallel

upto here, there is no problem. Then I realized that the refined mesh is
generated in a new time step within subdomains, e.g.
processor#/'newtimestep'/polyMesh

This means that the mesh is refined and decomposed but the fields
(U, p etc) are not. they remain the same in the old time step folder. So I am
wondering if there is a way to refine fields together with mesh? I've tried
to use couple of options such as '-overwrite' but it hasn't been successful so far.
Thank you in advance
Yusik
impecca is offline   Reply With Quote

Old   June 7, 2012, 09:31
Default
  #6
Member
 
Matthew J. Churchfield
Join Date: Nov 2009
Location: Boulder, Colorado, USA
Posts: 49
Rep Power: 8
mchurchf is on a distinguished road
Yusik,

Yes, try using refineHexMesh. It is the newer refinement tool, and it will refine the solution fields, too. It's use is similar to refineMesh.

Matt
wyldckat and mgg like this.
mchurchf is offline   Reply With Quote

Old   June 20, 2012, 13:06
Default
  #7
Member
 
Kim Yusik
Join Date: Dec 2009
Posts: 39
Rep Power: 7
impecca is on a distinguished road
Thanks for your valuable comments.
I forgot to mention this.
impecca is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with cyclic boundaries in Openfoam 1.5 fs82 OpenFOAM 37 January 7, 2015 01:31
Cyclic patches and parallel postprocessing problems askjak OpenFOAM Bugs 18 October 27, 2010 03:35
refineMesh -parallel crashes (in cavity tutorial) andras OpenFOAM Mesh Utilities 0 August 8, 2010 14:52
Cyclic patch in parallel calculations didomenico OpenFOAM Running, Solving & CFD 4 March 7, 2007 06:46
Cyclic Boundaries -> Match Option -> Arbitrary Derek CD-adapco 1 August 4, 2004 22:06


All times are GMT -4. The time now is 06:44.