SnappyHexMesh in Parallel
 

Hi all,

I started first tries with snappyHexMesh in parallel instead of serial. Everything works quite good so far. However, I am wondering about stategy of decomposition. The way it is intended to work seems to be:
1. Underlying hex-mesh (blockmesh)
2. decompose hex-mesh into n parts (decomposepar)
3. Run snappyHexMesh parallel with n processes
4. get the final snapped mesh distributed into n parts

So I guess you should use the same number of partitions for meshing as you intend to use for calculation. If you do so there is no need for redecomposition.
If you want one mesh you may run reconstructParMesh. This worked for me.
If you may run the calculation on m processes and m<n redistributeParMesh may do the job, but I have not tried it.
More interesting I want to run calculation on m cores, m>n. redistributeParMesh does not seem to the able handling this redistribution to a larger number of domains, is it?
Another question: Due to lack of memory I do not want to start all parallel processes at ones but e.g. 2 out of 4 and the other two after finishing the first two. Of course during meshing there is no opportunity for dynamic load balancing. However, it will save RAM. Is it possible?

Regards

redistributeParMesh should be
 

redistributeParMesh should be able to handle m>n. Just start it with the larger of the two. Don't know if it reads a mesh from time directories or always from constant so you might have to move your polyMesh into constant beforehand.

snappyHexMesh needs parallel communications in all phases, not just in the load balancing phase so no you cannot run in sequence.

Matthijs, thanks for this a
 

Matthijs,

thanks for this answer. So far I only got it working for m=n. I will run some more tests for m>n and m<n this afternoon and I will get back to you afterwards.

redistributeParMesh seems to r
 

redistributeParMesh seems to read from time-directories. However, it does not work that way . I got it working to redistribute a mesh from 2 to 3 parts using redistributeMeshPar. It is litte tricky and only worked wih hierarchical or metis, not with parmetis. Using metis I get the warning:

You have selected decomposition method decompositionMethod which does
not synchronise the decomposition across processor patches.

I do not understand the meaning and consequences from that.

I managed to run snappyHexMesh
 

I managed to run snappyHexMesh with processes one after another manually. This also seams to work with some tricks. These leads to two questions for me:

- Why not replace decompsePar with redistributeParMesh. His also works for 1->many parts and is parallel. His helps saving memory....
- Why not implement the option to run snappyHexMesh in parallel with individual parts sequentialy one after another instead of all at once. This will also save memory. Clusters are rare for meshing because snappyHexMesh is the first tool I know that is able to use them.

Regards

You must be some kind of mirac
 

You must be some kind of miracle worker Basti, because there is a lot of communication required between processors in snappyHexMesh. Running separate processors sequentially will not produce anything remotely resembling what you would get it you ran it in parallel.

If you are going to solve a case on a cluster, why not mesh it there as well? Saves you a whole lot of trouble.

Eugene, two reasons that cu
 

Eugene,

two reasons that currently might be a problem:
- Our current environment has not been designed for that because current commercial meshers are not really parallel. Meshing with snappyHexMesh takes more RAM than solving. I am running out of memory. SnappyHexMesh needs lots of memory.
- On the other hand I have messing nodes with relatively large shared memory.

However this is a problem of current hardware status and may change in future.
What I did to get it working is quite simple:
1. Run blockMesh for underlying hexmesh
2. Distribute hexmesh
3. Run snappyHexMesh on each of the underlying parts individually instead of one run. (Of course you can run each part in parallel once again...) This is quite similar to what old "proAM" can do. You have to change the "processor" patch to type "patch" for this to work.
4. Assemble mesh. I sometimes had troubles with stitchmesh for that.

Maybe our next generation cluster will solve al this. Regards.

Well it sure is an interesting
 

Well it sure is an interesting approach, but I repeat, the meshes you generate like this will be substantially different (and probably poorer in quality) than ones generated with all components online. Specifically, you will get weird jumps where you match the processors.

If you cannot mesh the entire thing due to insufficient memory, then I guess this is the only way to go.

Hi, thanks for you advices.
 

Hi,

thanks for you advices. Let me sum the procedure.

- run "blockMesh" a usual
- decomposeMethode in decomposeParDict must be hirarcial
- run "decomposePar"
- run "foamJob -p -s snappyHexMesh"
- afterwards run "reconstructParMesh -mergeTol 1
e-06 -latestTime" (or -time 1; -time 2; ...)

Works perfect. ("Yes, it can!") ;-)

Hi,

I am having trouble running snappyHexMesh in parallel. I use hierachical decomposition and I can run snappyHexMesh in parallel as long as I don't run the "snap" phase, which I absolutely need! Running the snap phase causes an immediate error.

My case runs fine on a single processor, however wwhen I set it to parallel (mpirun or foamJob -p,) I get a

Smoothing patch points ...
Smoothing iteration 0
Found 0 non-mainfold point(s).
[louis-dell:32518] *** An error occurred in MPI_Recv
[louis-dell:32518] *** on communicator MPI_COMM_WORLD
[louis-dell:32518] *** MPI_ERR_TRUNCATE: message truncated
[louis-dell:32518] *** MPI_ERRORS_ARE_FATAL (goodbye)


and changing MPI_BUFFER_SIZE does not solve the problem. It either changes the error message to a segmentation fault or a "cannot satisfy memory request" error. Even on a 40K cells mesh!!

Thanks for any hints on solving this!

-Louis

How do you assign fields to the decomposed meshes that you generate with snappyhexmesh in parallel? Can it be done directly on the decomposed mesh or does the mesh need to be reconstructed and then decomposed again?

What do you mean by fields?

I was trying to identify a way to assign the specified U,p, k, epsilon, nut, etc. flow variable (field) boundary conditions to the decomposed mesh directly. The incentive was to be able to process large cases on a 32bit parallel machine, as I thought the reconstructed mesh would violate the 32bit memory limit, which is around 2.5gigs.

It turns out the serial processes for the reconstruction, flow field assignment and re-decomposition do not consume the memory for the whole mesh. So I do not need a solution to this issue at this point, although one would improve the overall process, avoiding unnecessary mesh reconstruction and re-decomposition steps with it.

Well maybe I don't understand properly but as far as I know you can set the field conditions in the "0" folder. As for patch names, you have to define them prior to decomposition.

Best of luck,

-Louis

Hi Louis,

Unfortunately snappyHexMesh adds new patches, however decomposePar doesn't pass on any field BCs for patches that don't exist at decomposition time. ie. even if you've defined the BC's for the new patch in 0 before decomposition, it won't copy these to the processor directories. You can do it manually, but for large numbers of processors its not optimal.

However, i think there may be a workaround. If you create an empty patch in constant/polyMesh/boundary with the same name as the patch(es) that snappy adds, the field will be decomposed, and all is fine.

To create the empty patch open constant/polyMesh/boundary find the last patch (which should look something like)

to add a new patch use, put startface as the sum of nFaces and startFace from the last patch, and nFaces as 0, ie

The BC information for new_empty_patch should now be passed on when decomposePar runs (in theory)

I'm about to test this, so I'll let you know if it works.

Cheers,
Andrew

Dear Andrew,

did your approach work?

Best regards,

-Louis

Hi Louis,

It worked in some ways, the empty patch worked for the decomposition, but the mesh was not in a state to run anything. (missing cellProcAddressing files). I had to use reconstructParMesh followed by decomposePar again.

This approach seemed to work, without running out of memory.

Cheers,
Andrew

triSurface directories
 

I have got some way with running snappyHexMesh in prallel (a great utility) but have a question. It seems that decomposePar does what you would normally expect to run a solver but snappyHexMesh will not run in parallel (I am using v1.6 and doing 'foamJob -p -s snappyHexMesh -overwrite') unless I manually copy the constant/triSurface directory into processor?/constant/. This is true where I have used an stl file but also in the iglooWithFridges tutorial where there is only an edge file in triSurface directory. Is this the expected behaviour? Is there something I am missing that would avoid this?

Thanks,
Simon

Hi Simon,

I don't know the answer to your question but I'm happy to know that snappy works in parallel with 1.6! Can't wait to try it :)

-Louis

parallel meshing on n processors to parallel solution on n processors?
 

I know the answer is probably here, but I think this needs to be explicitly discussed... has someone gotten parallel snappyHexMesh to parallel solver (in my case simpleFoam) to work? I understand that one must move the polymesh folder from the latest snappyHexMesh iteration folder (2 or 3 or ?), but I get the following error messages after attempting to run...

host2:/data/offToCluster # mpirun --mca btl openib,sm,self -np 10 -machinefile ~/machinelist.txt simpleFoam -parallel > log.simpleFoam &
[1] 7652
host2:/data/offToCluster #
host2:/data/offToCluster # [1]
[1]
[1] keyword OBJECT_patch0 is undefined in dictionary "/data/offToCluster/processor1/0/p::boundaryField"
[1]
[1] file: /data/offToCluster/processor1/0/p::boundaryField from line 26 to line 63.
[1]
[1] From function dictionary::subDict(const word& keyword) const
[1] in file db/dictionary/dictionary.C at line 449.
[1]


I thought that the workflow was basic input deck > blockMesh > decomposePar (n processes) > copy stl files to all processor folders > snappyHexMesh (in parallel on n processes) > simpleFoam (in parallel on n processes).

My workaround of using "reconstructParMesh -mergeTol 1e-06 -time 2" does work, but then that limits me to the RAM of only one node because reconstructParMesh doesn't run in parallel.

Do I need to copying boundary conditions around in some way?