CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

Computations with eighty million cells

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   December 28, 2010, 11:01
Default Computations with eighty million cells
  #1
New Member
 
Armin Gh.
Join Date: Sep 2010
Location: Aachen,Germany
Posts: 29
Rep Power: 5
Armin is on a distinguished road
Hi Dear FOAMers,

I know this may sound a little bit strange, but has any one tried such big computations? I am having trouble using e.g. BlockMesh.

On the other hand, building a case with smaller amount of cells and using refinement is not possible because I need the data from processors(1000) to be plotted in tecPlot and it would be not manageable to do all that by hand.

So any suggestions?

Thanks alot,
Armin
Armin is offline   Reply With Quote

Old   December 28, 2010, 16:39
Default
  #2
Senior Member
 
BastiL
Join Date: Mar 2009
Posts: 462
Rep Power: 10
bastil is on a distinguished road
In general it should work. Of course you need to have sufficant hardware.
What problemes are you running into?
bastil is offline   Reply With Quote

Old   December 28, 2010, 18:44
Default
  #3
New Member
 
Armin Gh.
Join Date: Sep 2010
Location: Aachen,Germany
Posts: 29
Rep Power: 5
Armin is on a distinguished road
Hi and thanks for your response,

first of all , I have a pretty good set of hardware at my disposal, because I'm running on a cluster with 100 cpu's and an enough memory size for that concerns, so this should not be the problem.
But blockMesh cannot handle domains, which contain more than about 7-10 million cells , I have even tried it on our cluster , and it didn't work, it simply stops working with out any error message.

It actually runs for 24 hours with 100% power(cpu occupancy) and nothing happens, and the RAM is just occupied with 14% the whole time. So I'm guessing nor the stack removal, writing memory neither cpu usage is the problem.

And I would say blockMesh cannot communicate with the whole memory available. (Judging from CPU and memory allocation).

I hope that I provided you with enough information.

Cheers,
Armin
Armin is offline   Reply With Quote

Old   December 28, 2010, 20:29
Default
  #4
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 7,090
Blog Entries: 32
Rep Power: 70
wyldckat is a jewel in the roughwyldckat is a jewel in the roughwyldckat is a jewel in the rough
Greetings to all!

@Armin: I hope you've managed to solve the other issue you had with interFoam.
As for generating 8 million cells and over, I had to re-read your posts and try it myself. On the second half of my post is what I wrote before re-reading.
OK, I've made a cube with 200x200x200 simple hexahedra cells with blockMesh with an AMD 1055T x6 (but only used 1 core) CPU, 7.5GB of physical RAM (512MB are dedicated to the onboard GPU). It needed to send 400MB of stuff from RAM into swap, but in less than 2 minutes blockMesh generated the 8M cell mesh with success.
So Armin, I suggest you try and isolate the problem by parts:
  • try generating meshes with increasing complexity, i.e. make a simple cube mesh with increasing number of cells: 500k->1M->2M->4M->8M and see if you hit a barrier.
  • make sure you are using the 64bit version of OpenFOAM. This should tell you which version you are using:
    Code:
    echo $WM_ARCH_OPTION
    If you are using the 32 bit version, you either hit a limit of 2M or 4M cells.

________________________
Now, for what I wrote before re-reading:

As for generating 8 million cells and over, I've seen some discussion about this in the past in this thread: blockMesh for bigger mesh
Basically, it says that:
  • blockMesh can't be executed in parallel;
  • you could use snappyHexMesh to refine the mesh generated by blockMesh, since snappyHexMesh can be executed in parallel.
The other possibility - although you put it aside in the first post - is to use refineMesh, but in parallel mode. I haven't tried it yet, but in theory it should work very well. This of course only makes sense if a single machine doesn't have the required 8GB of RAM. Basically, you could refine an 1M cell mesh into an 8M cell mesh simply by doing:
Code:
blockMesh
cellSet
decomposePar
foamJob -p -s refineMesh -dict
Well, as long as the respective *Dict files are properly defined... I believe it should work.

Best regards and good luck!
Bruno

PS: by the way, just to make sure, which OpenFOAM version are you using and which gcc version did you use to build OpenFOAM? It's just that OpenFOAM 1.7.0 and 1.6.x do build with gcc 4.5.0 and above, but some things won't work as intended... and one of them is blockMesh.

Last edited by wyldckat; December 28, 2010 at 20:32. Reason: See "PS:"
wyldckat is offline   Reply With Quote

Old   December 29, 2010, 05:33
Default
  #5
New Member
 
Armin Gh.
Join Date: Sep 2010
Location: Aachen,Germany
Posts: 29
Rep Power: 5
Armin is on a distinguished road
Hi ,

As for the matter of the last problem I solved it , thanks Bruno

As for this matter, I'm using OpenFOAM 1.6 and gcc version 4.3.3, and I haven't had any problem with it till now.

I will try out the first part of your post and let yoe know how it went.

cheers,
Armin
Armin is offline   Reply With Quote

Old   January 5, 2011, 13:24
Default Follow-up
  #6
New Member
 
Armin Gh.
Join Date: Sep 2010
Location: Aachen,Germany
Posts: 29
Rep Power: 5
Armin is on a distinguished road
Hi FOAMers,

I tried the suggestions earlier , and understood that although blockMesh is capable of meshing indefinitely, it needs a lot of memory to that. And well despite the fact that our cluster has 48 gigabytes on the biggest node, it is not always free(Others do lunch calculations ).

So I decided to use blockMesh for a less complicated mesh and then refine it with refineMesh, which can in fact work in parallel, and so I could use the queuing system with out any crashes.

But now I have another problem, regarding refineMesh utility; because I am using cyclic booundary conditions, refineMesh somehow does not like it and gives the following error;

Code:
There are decomposed cyclics in this mesh with transformations
This is not supported. The results will be incorrect.
so any suggestions??

Thanks a bunch,
Armin
Armin is offline   Reply With Quote

Old   January 5, 2011, 20:18
Default
  #7
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 7,090
Blog Entries: 32
Rep Power: 70
wyldckat is a jewel in the roughwyldckat is a jewel in the roughwyldckat is a jewel in the rough
Hi Armin,

Disclaimer: my experience in this field of OpenFOAM (as in many other fields ) is very limited, so I ask of the more experienced users to also step in.
I'm only answering because I've been getting the feeling that this forum still isn't at full speed, since the new year has started less than 6 days ago


Cyclic boundaries... if I'm not mistaken, it means that the respective cyclic boundary on the other side needs to be an absolute mirror cell wise. With this in mind, it's only natural that refineMesh can't handle these special cases.

You might want to check out this thread: local mesh refinement at channel geometries
This same issue was discussed there a few months ago.

So, if I were in your situation, I would:
  1. have the final refinement level already done by blockMesh on those cyclic boundaries, but coarse on the remaining cells.
  2. then use cellSet or selectCells to pick the cells that still need refining.
  3. then finally use refineMesh to refine the selected cells.

Upon looking a bit further, I saw a couple more utilities that might aid on another way of seeing things (i.e. refine cyclics afterwards):
  • autoRefineMesh - "Utility to refine cells near to a surface." - I think it uses the same dictionary as refineMesh, but it can't be used in parallel.
  • refineWallLayer - "Utility to refine cells next to patches." - I think it doesn't use the same dictionary. Also, no parallel mode allowed. The advantage I see from this one, is that if you use refineMesh to handle the whole mesh, except the cell area of the cyclic patches, then you can use refineWallLayer to do a symmetric refinement of those patches.
For a more complete list of OpenFOAM utilities, see here.

Best regards and good luck!
Bruno

Last edited by wyldckat; January 5, 2011 at 20:19. Reason: typo...
wyldckat is offline   Reply With Quote

Old   January 6, 2011, 17:04
Default Making cyclics easy walls
  #8
New Member
 
Armin Gh.
Join Date: Sep 2010
Location: Aachen,Germany
Posts: 29
Rep Power: 5
Armin is on a distinguished road
Hi FOAMers,

yesterday it occurred to me that probably I could change the cyclic conditions to wall do the refine on parallel and after that change them to cyclic. It actually worked but I am not sure if I have plausible mesh now.

Any one care to shed a light on that, is welcome.
And give Bruno some credit, he has been answering all of my questions, isn't any one else out there??

Cheers,
Armin
Armin is offline   Reply With Quote

Old   January 7, 2011, 14:15
Default
  #9
New Member
 
Armin Gh.
Join Date: Sep 2010
Location: Aachen,Germany
Posts: 29
Rep Power: 5
Armin is on a distinguished road
Hi again,

discard my last post,that was my stupidest approach ever. I totally forgot that other files such as boundary and faces etc. should be changed too. This is not actually possible since they are all labels. so sorry.

cheers,
Armin
Armin is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
SnappyHexMesh for internal Flow vishwa OpenFOAM Native Meshers: snappyHexMesh and Others 23 August 6, 2014 03:50
Import netgen mesh to OpenFOAM hsieh Open Source Meshers: Gmsh, Netgen, CGNS, ... 32 September 13, 2011 05:50
Computations with eighty million cells Armin Main CFD Forum 0 December 28, 2010 06:47
snappyHexMesh aborting Tobi OpenFOAM Native Meshers: snappyHexMesh and Others 0 November 10, 2010 03:23
physical boundary error!! kris CD-adapco 2 August 3, 2005 00:32


All times are GMT -4. The time now is 21:12.