[Armin]

Hi Dear FOAMers,

I know this may sound a little bit strange, but has any one tried such big computations? I am having trouble using e.g. BlockMesh.

On the other hand, building a case with smaller amount of cells and using refinement is not possible because I need the data from processors(1000) to be plotted in tecPlot and it would be not manageable to do all that by hand.

So any suggestions?

Thanks alot,
Armin

[bastil]

In general it should work. Of course you need to have sufficant hardware.
What problemes are you running into?

[Armin]

Hi and thanks for your response,

first of all , I have a pretty good set of hardware at my disposal, because I'm running on a cluster with 100 cpu's and an enough memory size for that concerns, so this should not be the problem.
But blockMesh cannot handle domains, which contain more than about 7-10 million cells , I have even tried it on our cluster , and it didn't work, it simply stops working with out any error message.

It actually runs for 24 hours with 100% power(cpu occupancy) and nothing happens, and the RAM is just occupied with 14% the whole time. So I'm guessing nor the stack removal, writing memory neither cpu usage is the problem.

And I would say blockMesh cannot communicate with the whole memory available. (Judging from CPU and memory allocation).

I hope that I provided you with enough information.

Cheers,
Armin

[wyldckat]

Greetings to all!

@Armin: I hope you've managed to solve the other issue you had with interFoam.
As for generating 8 million cells and over, I had to re-read your posts and try it myself. On the second half of my post is what I wrote before re-reading.
OK, I've made a cube with 200x200x200 simple hexahedra cells with blockMesh with an AMD 1055T x6 (but only used 1 core) CPU, 7.5GB of physical RAM (512MB are dedicated to the onboard GPU). It needed to send 400MB of stuff from RAM into swap, but in less than 2 minutes blockMesh generated the 8M cell mesh with success.
So Armin, I suggest you try and isolate the problem by parts:

• try generating meshes with increasing complexity, i.e. make a simple cube mesh with increasing number of cells: 500k->1M->2M->4M->8M and see if you hit a barrier.
• make sure you are using the 64bit version of OpenFOAM. This should tell you which version you are using:

  echo $WM_ARCH_OPTION

  If you are using the 32 bit version, you either hit a limit of 2M or 4M cells.

________________________
Now, for what I wrote before re-reading:

As for generating 8 million cells and over, I've seen some discussion about this in the past in this thread: http://www.cfd-online.com/Forums/ope...gger-mesh.html
Basically, it says that:

• blockMesh can't be executed in parallel;
• you could use snappyHexMesh to refine the mesh generated by blockMesh, since snappyHexMesh can be executed in parallel.

The other possibility - although you put it aside in the first post - is to use refineMesh, but in parallel mode. I haven't tried it yet, but in theory it should work very well. This of course only makes sense if a single machine doesn't have the required 8GB of RAM. Basically, you could refine an 1M cell mesh into an 8M cell mesh simply by doing:

blockMesh
cellSet
decomposePar
foamJob -p -s refineMesh -dict

Well, as long as the respective *Dict files are properly defined... I believe it should work.

Best regards and good luck!
Bruno

PS: by the way, just to make sure, which OpenFOAM version are you using and which gcc version did you use to build OpenFOAM? It's just that OpenFOAM 1.7.0 and 1.6.x do build with gcc 4.5.0 and above, but some things won't work as intended... and one of them is blockMesh.

[Armin]

Hi ,

As for the matter of the last problem I solved it , thanks Bruno

As for this matter, I'm using OpenFOAM 1.6 and gcc version 4.3.3, and I haven't had any problem with it till now.

I will try out the first part of your post and let yoe know how it went.

cheers,
Armin

[Armin]

Hi FOAMers,

I tried the suggestions earlier , and understood that although blockMesh is capable of meshing indefinitely, it needs a lot of memory to that. And well despite the fact that our cluster has 48 gigabytes on the biggest node, it is not always free(Others do lunch calculations ).

So I decided to use blockMesh for a less complicated mesh and then refine it with refineMesh, which can in fact work in parallel, and so I could use the queuing system with out any crashes.

But now I have another problem, regarding refineMesh utility; because I am using cyclic booundary conditions, refineMesh somehow does not like it and gives the following error;

There are decomposed cyclics in this mesh with transformations
This is not supported. The results will be incorrect.

so any suggestions??

Thanks a bunch,
Armin

[wyldckat]

Hi Armin,

Disclaimer: my experience in this field of OpenFOAM (as in many other fields ) is very limited, so I ask of the more experienced users to also step in.
I'm only answering because I've been getting the feeling that this forum still isn't at full speed, since the new year has started less than 6 days ago


Cyclic boundaries... if I'm not mistaken, it means that the respective cyclic boundary on the other side needs to be an absolute mirror cell wise. With this in mind, it's only natural that refineMesh can't handle these special cases.

You might want to check out this thread: http://www.cfd-online.com/Forums/ope...eometries.html
This same issue was discussed there a few months ago.

So, if I were in your situation, I would:

1. have the final refinement level already done by blockMesh on those cyclic boundaries, but coarse on the remaining cells.
2. then use cellSet or selectCells to pick the cells that still need refining.
3. then finally use refineMesh to refine the selected cells.

Upon looking a bit further, I saw a couple more utilities that might aid on another way of seeing things (i.e. refine cyclics afterwards):

• autoRefineMesh - "Utility to refine cells near to a surface." - I think it uses the same dictionary as refineMesh, but it can't be used in parallel.
• refineWallLayer - "Utility to refine cells next to patches." - I think it doesn't use the same dictionary. Also, no parallel mode allowed. The advantage I see from this one, is that if you use refineMesh to handle the whole mesh, except the cell area of the cyclic patches, then you can use refineWallLayer to do a symmetric refinement of those patches.

For a more complete list of OpenFOAM utilities, see here.

Best regards and good luck!
Bruno

[Armin]

Hi FOAMers,

yesterday it occurred to me that probably I could change the cyclic conditions to wall do the refine on parallel and after that change them to cyclic. It actually worked but I am not sure if I have plausible mesh now.

Any one care to shed a light on that, is welcome.
And give Bruno some credit, he has been answering all of my questions, isn't any one else out there??

Cheers,
Armin

[Armin]

Hi again,

discard my last post,that was my stupidest approach ever. I totally forgot that other files such as boundary and faces etc. should be changed too. This is not actually possible since they are all labels. so sorry.

cheers,
Armin