CFD Online Discussion Forums

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)
-   OpenFOAM Running, Solving & CFD (https://www.cfd-online.com/Forums/openfoam-solving/)
-   -   Mesh too big for memory. How to perform decomposition in parallel? (https://www.cfd-online.com/Forums/openfoam-solving/182423-mesh-too-big-memory-how-perform-decomposition-parallel.html)

t.oliveira January 9, 2017 19:08

Mesh too big for memory. How to perform decomposition in parallel?
 
Hi,

I need to decompose a case to run it in many nodes because the mesh is too large for the memory of a single node. It is so large that running decomposePar in a single node is not possible.
How can I decompose the case in parallel, using distributed memory? Running decomposePar in parallel, if possible, would be a solution.

Best wishes,
Thomas

P.S.: A question like this was asked nine years ago. I am asking again because things may have changed since then.

akidess January 10, 2017 05:30

How did you generate the mesh? Can this be less memory consuming than the decomposition?

t.oliveira January 12, 2017 08:16

Dear Anton,

Indeed, your question makes sense. I have also had problems to generate the mesh using blockMesh. Since my geometry is sufficiently simple, I wrote a program which directly writes down the files points, faces, owner, neighbour and boundary without using OpenFOAM classes.

An alternative would be to write another program that write those files in processor* directories, but this would require investigating how decomposePar does so. Anyway, this alternative would not solve the problem for the cases in which I have had access to a large-memory machine when creating the mesh but that machine is not available anymore for decomposing it.

Best wishes,
Thomas

arjun January 12, 2017 23:55

Quote:

Originally Posted by t.oliveira (Post 632989)
Dear Anton,

Indeed, your question makes sense. I have also had problems to generate the mesh using blockMesh. Since my geometry is sufficiently simple, I wrote a program which directly writes down the files points, faces, owner, neighbour and boundary without using OpenFOAM classes.

An alternative would be to write another program that write those files in processor* directories, but this would require investigating how decomposePar does so. Anyway, this alternative would not solve the problem for the cases in which I have had access to a large-memory machine when creating the mesh but that machine is not available anymore for decomposing it.

Best wishes,
Thomas

I actually wrote an oct tree based mesh generator that uses hard disk memory and generated 150 million cell mesh using only 2GB ram laptop. So mesh generation without partitioning is possible.

I am surprised to learn that openFOAM does not do paritioning in parallel. In FVUS I do parallel partitioning using par metis but when i was porting the solver for windows par metis no longer reliably available and thus had to write partitioner myself. Since I am no expert the partitioner is serial only. I put it there very reluctantly because that was one step that handicapped the solver for large meshes and felt very bad about it.

openfoam is out there for years so its surprising.

akidess January 13, 2017 04:16

Quote:

Originally Posted by arjun (Post 633076)
openfoam is out there for years so its surprising.

Not that surprising. I guess 90% of all users either have relatively small meshes or use parallel meshing (snappyHexMesh). Did any of your FVUS customers complain about your partitioner being serial only?

arjun January 13, 2017 07:45

Quote:

Originally Posted by akidess (Post 633088)
Not that surprising. I guess 90% of all users either have relatively small meshes or use parallel meshing (snappyHexMesh). Did any of your FVUS customers complain about your partitioner being serial only?


No so far no-one complained. But all the people who are doing serious work using linux version that is parallel partitioning. Most people using it for VOF and multiphase so big sizes are avoided as much as possible. (There is someone going to try on more than 50 million cell cases but that would be in april).

Even on windows the only step which is serial is the partitioning part so only that graph structure that is used for partitioning is serial (that is collected on root process). Rest of the solver is parallel. That means FVUS loads the mesh into partitions user is running solver on, then partitioning is performed (serial on windows and parallel on linux) and redistribution takes place. Solver then continues.

Because only one graph structure is serial only the size of meshes user can run still be much higher than openFOAM.

Cyp January 13, 2017 18:41

one solution is to create a coarse background grid (with blockMesh), then to decompose the domain on X CPUs and refine the background grid in parallel with refineMesh. Then you can use snappyHexMesh in parallel.

crazzy.pirate43 July 20, 2017 07:42

decomposing seperat file after the decomposition process
 
Quote:

Originally Posted by arjun (Post 633129)
No so far no-one complained. But all the people who are doing serious work using linux version that is parallel partitioning. Most people using it for VOF and multiphase so big sizes are avoided as much as possible. (There is someone going to try on more than 50 million cell cases but that would be in april).

Even on windows the only step which is serial is the partitioning part so only that graph structure that is used for partitioning is serial (that is collected on root process). Rest of the solver is parallel. That means FVUS loads the mesh into partitions user is running solver on, then partitioning is performed (serial on windows and parallel on linux) and redistribution takes place. Solver then continues.

Because only one graph structure is serial only the size of meshes user can run still be much higher than openFOAM.

Hi,

As you are talking about the decomposition process I have a problem here, I'm preparing a new boundary condition but it refuse to be decomposed during the decompsePar so I have to redistribute it manually for all processors , is there any way to use specific command to distribute it directly or I have to paste it in each processor manually:

Thanks in advance


All times are GMT -4. The time now is 18:08.