[gera]

Hello everyone,

In CFX, I compute the flow when the mesh is reconstructed during the computation (unsteady flow with re-meshing). I have two computers with identical amount of memory.

If I run my simulation at one computer then the memory usage is 45 % .

If I run my simulation at two computers then the memory usage is 45 % at each computer, however, I expect 22.5 % or so.

Does anyone know how to reduce the memory usage when several computers compute in parallel?

All the best,
Gera

[ghorrocks]

There is a bit of an overhead in running parallel compared to serial. It has to keep track of all the partition interfaces. But for a large model this should be small, each partition should roughly be the expected fraction of the entire model.

[gera]

Ghorrocks, thank you for the reply.

Yes, it is true indeed for the problem without remeshing. I can see exactly what you wrote for stationary mesh.

However, I solve the flow problem with remeshing. The algorithm is as follow:
0. CFX runs on two computers. Each computer takes the memory amount equal to the memory amount if the computation were ran on one computer. In my case, CFX takes 45% of memory instead of 22.5% per a computer.
1. CFX changes cells. For example, CFX stretches some cells.
2. CFX stops computing when a some condition is valid to stop the modelling (the mesh quality is bad).
3. ICEM CFD prepares a new mesh.
4. CFX takes the new mesh from 3., then CFX interpolates the numerical solution, after that CFX makes new partitioning, and starts the next iteration (goes to 1.)

The memory usage problem appears only when I do remeshing according to the algorithm. The memory usage problem appears just after the computation starts.

[Opaque]

You are describing a multi-step simulation, and making a global conclusion w/o analyzing the details of each step.

When you say you are running in parallel, are you certain ALL the steps in the simulation are in parallel? Otherwise, your expectation is flawed.

Ansys CFX partitioning step is a serial process (check the documentation); therefore, if the partitioning step is the one taking the memory "hit", the global memory usage of your parallel simulation will always peak at the same peak as the serial simulation (at least on the main/master process).

Have you checked if the re-meshing step is running in parallel, or serial? That is another serial step in your multi-step simulation.

Keep in mind that parallel simulation scale the time/memory based on the worst step in the process.

Hope the above helps

[gera]

Opaque,
thank you for the reply.

Yes, you are right -- some steps are serial.

Serial processes are:
ICEM CFD reconstructs only on the master (host) computer.
CFX makes partitioning only on the master (host) computer.

Parallel process is:
CFD computation is a parallel process on two computers.

Not clear to me:
During the computation CFX changes the mesh: cells can be stretched or reduced in size, etc. I cannot understand how CFX makes it on both computers or on the master computer only.





Some important addition to my original question.
Mesh change in CFX is done through:
Domain -> Basic Settings -> Mesh deformation ->
- Option : Region of Motion Specified
- Displacement Rel. to: Previous Mesh.

Then CFX changes cells of the mesh and the CFX does not divide the total memory approximately by the number of computers. In this case, CFX changes the mesh, but it does not update the mesh in ICEM.


Re-meshing via ICEM CFD through:

Simulation control -> Configuration -> Remeshing tab -> Option -> ICEM CFD Replay

Again CFX does not divide the memory between computers. In this case, CFX updates the mesh via ICEM when the mesh quality becomes poor.


If I do not include "Mesh deformation" and "Configuration -> Remeshing" then CFX does divide the total memory between computers.