Some questions on blockMesh, decomposePar and renumberMesh
 

Hi everyone,

actually I am working on a real big case. So i tried renumberMesh on the damBreak tutorial and it said that the bandwith was reduced and indeed the processing time was slightly lower. (for a fairly coarse mesh)
So I am about to include renumberMesh in my big case. This case is decomposed for 4 processors.
Question 1: Do i have to run renumberMesh before or after decomposePar?
To see what i am doing:
Question2: As i tried renumberMesh for the damBreak (without decomposing) i had to change the controlDict, so that interFoam starts from 0.001 instead of 0 (the renumbered Mesh is written one timestep later). For my big case i changed the controlDict, but interDyMFoam allways said "wrting mesh for time 0"? So i tried renumberMesh -overwrite (to overwrite the old mesh), and then renumberMesh says "writing mesh to constant" and interDyMFoam still starts from 0?
So what do i have to do, to decompose my case and to renumber the mesh so that interDyMFoam uses it?

Question3: renumberMesh and also subsetMesh offer an option -parallel in the help output. But when i try to run it with the -parallel flag there comes for boath an error: "FATAL ERROR: attempt to run parallel on 1 processor" ? What does that mean and what do i have to do to run these commands in parallel?


Thanks a lot!
Greetings
Leech


PS: i tried to run renumberMesh before decomposing. This seemed to work better, as decomposePar said writing mesh for time 0.01. But then interDyMFoam says: "number of points in mesh differs from ... blabla in /processor0/constant/polyMesh" ? Any idea ?

Hi Pierre,

OK, in a nutshell:

1. Code:

   ---

   renumberMesh -overwrite

   ---

   This is the quickest option, since this way you don't need to do any changes to the time management system. The mesh present in "constant/polyMesh" gets renumbered into a more diagonal form for the "A.x=B" system.
   If you don't use the overwrite option, then two things must be done:
   • You must also copy the other fields ("U", "p" and so on) from the folder "0" to the new folder, which in your case is "0.001".
   • You must also edit "system/controlDict" and state that "startFrom" is "latestTime": http://www.openfoam.org/docs/user/controlDict.php
2. You should run renumberMesh in parallel (after decomposePar), so that it is more likely that the processors will work with a higher probability of being in sync:
   Code:

   ---

   foamJob -s -p renumberMesh -overwrite

   ---

   or
   Code:

   ---

   mpirun -n 4 renumberMesh -overwrite -parallel

   ---

   or any other similar way, depending on the parallel launching system you are using.
3. Running renumberMesh both before and after decomposePar will only have significant effect after, simply because the decomposition method will not take into account the location of the cells in the matrix; the decomposition method will only care about the position on the XYZ space. Nonetheless, it's possible that for those more complex decomposition methods, it might prove to be slightly more efficient (i.e. faster) to decompose with a nice and neat matrix ;)

I hope that answers your questions!

Best regards,
Bruno

Hi Bruno,

thank you very much! I will try this out this evening.
I didn't had the idea to run renumberMesh with foamJob..

How do you know this stuff? I googled a lot and got problems to find something about these more special tools from openFOAM. When i write renumberMesh -doc the help says that there i can find a documentation. But i just get something like "blalba no documentation online"..

Thank you!
Greets
Leech

Hi Leech,

I found this out when I was trying to improve performance with running a case in parallel in a single machine. I went looking on these folders:

• applications/utilities/parallelProcessing
• applications/utilities/mesh

And I then tripped on renumberMesh. Tested running renumberMesh before and after decomposition, to compare which gave the better results and that's when I concluded that it would be better to renumber after decomposing.

edit: running renumberMesh in parallel mode felt natural, since I wanted to affect the decomposed mesh, not the base mesh. Additionally, the "-doc" option only works if you compile the Doxygen source code as well: Building Code Documentation on OpenFOAM 2.0.0/x

As for foamJob, I've been using it for so long now that I can't remember how I found out about it.

Either way, I've been dealing and messing with the OpenFOAM source code for quite some time now, including being responsible for one of the various unofficial ports for Windows, namely blueCFD:

• http://www.bluecape.com.pt/blueCFD
• http://code.google.com/p/bluecfd-singlecore/
• http://openfoamwiki.net/index.php/Ti...ows_with_MinGW

Additionally, I've been taking notes on how to run OpenFOAM in parallel, so when I get some time (or if someone else wants to get started on it before me) I can write about it on the openfoamwiki.net: Advanced tips for working with the OpenFOAM shell environment

In conclusion: by keeping up-to-date with what is written here on the forum (and help in what I can), monitoring the bug tracker, going through the code and tinkering with things, I've continued to gather information and try to compile it into my blog and/or openfoamwiki.net!

Best regards,
Bruno

single and parallel run
 

Dear Bruno,

I am facing another problem.... and thank you in advance!

I have a case, it's two cubes in two blocks respectively, one block will rotate with the inside cube, to simulate two cubes(cars) passing by each other. I run the case in a single processor, icoDyMFoam, it works fine for me: please find the animation in the link : http://www.youtube.com/watch?v=G0tIz...M5Gu5PwqPzpj8=

when I tried to run in parallel, decomposePar is fine. Then run the solver, it gave the error like this :


Could you please help on this? I don't know why...

Thank you so much!

Aqua

Hi Aqua,

Sorry, but I've never used GGI, so I don't know where to begin with it in parallel :(
All I can do is suggest that you study the case you used for the base mesh.

Good luck!
Bruno

Hi Aqua,
I'm supposed to have the same problem as yours. You use the 1.6-ext version of OpenFOAM with GGI patch, but the version of the renumberMesh is not seemed to be executed in parallel mode...

renumberMesh and runtime
 

Hi,

I tried out the renumberMesh utility on a case with 197945 cells. I decomposed the domain into 2 sub-domains (98894 + 99051). I ran the parallel case once without renumbering and once with renumbering.

renumberMesh displayed this

However, the renumbered case is significantly slower than the non-renumbered case.

Non-renumbered:



renumbered:


What could be the reason for this?

Hi Gerhard,

There are a few details that come into play:

1. Did you renumber before decomposition?
2. Or did you renumber after decomposition and executed renumberMesh with the parallel option?
3. If you did #1, try #2 as well. Or vice-versa.
4. How exactly was the mesh generated?
5. How structured is the mesh, or is it completely unstructured?
6. Which decomposition method did you use?
   1. Did you confirm how many faces were shared between sub-domains in each case? The more there are, the most likely it will take longer.
7. Last but not least: the renumbered mesh might be worst for your case, due to the memory access order that each sub-domain needs.
   I've had a test case where with a 6 core processor, would have greater performance if I over-scheduled it with 16 sub-domains, instead of using only 6 sub-domains!

Best regards,
Bruno

I tried #2 and ran

The mesh is generated by blockMesh and is structured. Maybe there is no or little room for improvements when using structured meshes?

I used the decomposition method scotch without any further parameters.

The number of shared faces is 3948. If i renumber after decomposing, this number will not change, will it?

It might be different if you renumber before decomposing.
And please try renumbering before decomposing, because as I said before, it may vary depending on the case.

renumberMesh after refineMesh
 

Hello all,

I have a quick question,

I build my grid using blockMesh, then I run snappyHexMesh -overwrite to add boundary layers and then refineMesh -overwrite to refine local area. And then if I run renumberMesh -overwrite I get the following error:

--> FOAM FATAL IO ERROR:
size 7680 is not equal to the given value of 122880

7680 is the size (or number of faces of one of the patches before refineMesh and the 122880 is the size after refineMesh)

Any idea how to either renumberMesh to read the size correctly? Or how to update the mesh record to reflect the updated size?

Thank you all.

I was able to work around it by doing the refineMesh step first before the snappyHexMesh step because the mesh size in the snappy generated files in the 0 directory don't get updated when executing refineMesh -overwrite

By switching the order, snappy generates those files for the final mesh.

However, I would still like to know if there is way to update all of the 0 directory (including the snappy files). Thank you.

Ahmed Ibrahim

Hi all


First off, I think you can reset the 0-folder with setFields? I think this will also work of you have changed the mesh, since setFields works with real distances rather than a number of cells?

Secondly, I am wondering whether there is any benefit in applying renumberMesh directly after using blockMesh? I am working on a case which is not in parallel (yet) and without any further additions to the mesh?


I was going to try this out myself and I am going the post the answer once I am done. But in case anyone already knows the answer, feel free to respond.



Best regards
Arne

Hi all


I tried to use renumberMesh immediately after blockMesh and it turns out that this does have an influence. I am not sure why this is not automatically done whenever using blockMesh?

I saw that the bandwidth does not change, but the envolope decreased by 11%, I am just not sure whether this will have a remarkable influence on the simulationtime afterwards?


Best regards
Arne

Quick answer: Depends on how the blocks were assembled with blockMesh.

• If you only have 2-3 blocks, then it is probable that they are already well sorted, or at least well sorted within those blocks.
• If you have dozens of blocks, created in nearly random order, I expect that renumberMesh will improve considerably the mesh organization in memory.

Nonetheless, this is what I expect, because I'm not entirely certain how much of an improvement I've gained from this with a mesh created with blockMesh, since I rarely create multi-block meshes with it.

I also work with only 1 block. I just gave it a try on one of the tutorial cases (pisoFoam -> RAS -> cavity), the mesh exists out of 1 block here. Without renumberMesh, the simulation took 8.98 second, while with renumberMesh it only took 8.86. This difference corresponds with approximately 1.5% decrease in time for a mesh of one block with 20x20 cells.

Furthermore, the only change I've noted so far is the 'time step continuity errors'.


Best regards
Arne

Quick answer: That's within the margin of error due to computational randomness... Examples:

• CPUs don't always run at the same exact frequency every time.
• The way the matrices are solved can affect slightly the end results.

In other words, there is certainly a difference (the matrices are solved in a different way), but this difference does not necessarily mean that the simulation would run faster (because the difference is within the margin error)?


Thanks for your help. I think I will be using renumberMesh from now on. It might not do any good, but it will certainly not do any bad.

Quick note: Except when it does... always look at how much the matrix band-width changes, don't assume it always works perfectly.
I've seen it get worse when running it in serial and then in parallel after decomposing... namely that renumbering in serial was already great and running it again in parallel did disturb the band-width... although it wasn't all that much and it was not clear how much it impacted performance by also running it in parallel or not.

Oh, and don't forget about the "-overwrite" option... otherwise you can risk having an extra time step that you don't notice that corrupted your results...