CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > ANSYS > CFX

CFX memory usage with re-meshing

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   June 1, 2022, 10:02
Default CFX memory usage with re-meshing
  #1
Member
 
Georgy
Join Date: Apr 2011
Location: Russia
Posts: 32
Rep Power: 14
gera is on a distinguished road
Hello everyone,

In CFX, I compute the flow when the mesh is reconstructed during the computation (unsteady flow with re-meshing). I have two computers with identical amount of memory.

If I run my simulation at one computer then the memory usage is 45 % .

If I run my simulation at two computers then the memory usage is 45 % at each computer, however, I expect 22.5 % or so.

Does anyone know how to reduce the memory usage when several computers compute in parallel?

All the best,
Gera
gera is offline   Reply With Quote

Old   June 1, 2022, 20:32
Default
  #2
Super Moderator
 
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,641
Rep Power: 142
ghorrocks is just really niceghorrocks is just really niceghorrocks is just really niceghorrocks is just really nice
There is a bit of an overhead in running parallel compared to serial. It has to keep track of all the partition interfaces. But for a large model this should be small, each partition should roughly be the expected fraction of the entire model.
__________________
Note: I do not answer CFD questions by PM. CFD questions should be posted on the forum.
ghorrocks is offline   Reply With Quote

Old   June 3, 2022, 13:25
Default
  #3
Member
 
Georgy
Join Date: Apr 2011
Location: Russia
Posts: 32
Rep Power: 14
gera is on a distinguished road
Ghorrocks, thank you for the reply.

Yes, it is true indeed for the problem without remeshing. I can see exactly what you wrote for stationary mesh.

However, I solve the flow problem with remeshing. The algorithm is as follow:
0. CFX runs on two computers. Each computer takes the memory amount equal to the memory amount if the computation were ran on one computer. In my case, CFX takes 45% of memory instead of 22.5% per a computer.
1. CFX changes cells. For example, CFX stretches some cells.
2. CFX stops computing when a some condition is valid to stop the modelling (the mesh quality is bad).
3. ICEM CFD prepares a new mesh.
4. CFX takes the new mesh from 3., then CFX interpolates the numerical solution, after that CFX makes new partitioning, and starts the next iteration (goes to 1.)

The memory usage problem appears only when I do remeshing according to the algorithm. The memory usage problem appears just after the computation starts.
gera is offline   Reply With Quote

Old   June 3, 2022, 18:20
Default
  #4
Senior Member
 
Join Date: Jun 2009
Posts: 1,775
Rep Power: 31
Opaque will become famous soon enough
You are describing a multi-step simulation, and making a global conclusion w/o analyzing the details of each step.

When you say you are running in parallel, are you certain ALL the steps in the simulation are in parallel? Otherwise, your expectation is flawed.

Ansys CFX partitioning step is a serial process (check the documentation); therefore, if the partitioning step is the one taking the memory "hit", the global memory usage of your parallel simulation will always peak at the same peak as the serial simulation (at least on the main/master process).

Have you checked if the re-meshing step is running in parallel, or serial? That is another serial step in your multi-step simulation.

Keep in mind that parallel simulation scale the time/memory based on the worst step in the process.

Hope the above helps
__________________
Note: I do not answer CFD questions by PM. CFD questions should be posted on the forum.
Opaque is offline   Reply With Quote

Old   June 5, 2022, 12:12
Default
  #5
Member
 
Georgy
Join Date: Apr 2011
Location: Russia
Posts: 32
Rep Power: 14
gera is on a distinguished road
Opaque,
thank you for the reply.

Yes, you are right -- some steps are serial.

Serial processes are:
ICEM CFD reconstructs only on the master (host) computer.
CFX makes partitioning only on the master (host) computer.

Parallel process is:
CFD computation is a parallel process on two computers.

Not clear to me:
During the computation CFX changes the mesh: cells can be stretched or reduced in size, etc. I cannot understand how CFX makes it on both computers or on the master computer only.





Some important addition to my original question.
Mesh change in CFX is done through:
Domain -> Basic Settings -> Mesh deformation ->
- Option : Region of Motion Specified
- Displacement Rel. to: Previous Mesh.

Then CFX changes cells of the mesh and the CFX does not divide the total memory approximately by the number of computers. In this case, CFX changes the mesh, but it does not update the mesh in ICEM.


Re-meshing via ICEM CFD through:

Simulation control -> Configuration -> Remeshing tab -> Option -> ICEM CFD Replay

Again CFX does not divide the memory between computers. In this case, CFX updates the mesh via ICEM when the mesh quality becomes poor.


If I do not include "Mesh deformation" and "Configuration -> Remeshing" then CFX does divide the total memory between computers.
gera is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
General recommendations for CFD hardware [WIP] flotus1 Hardware 19 Yesterday 13:48
CFX error: "enforce system memory limit" controls whether this is a fatal error windfire CFX 12 March 23, 2020 09:15
Fluent Parallel Memory Usage David Christopher FLUENT 1 March 3, 2011 03:20
How to optimize the memory usage when using FEM vasilis Main CFD Forum 11 August 25, 2009 00:57
Problem with meshing long, thin faces in CFX Martin CFX 3 January 8, 2009 21:51


All times are GMT -4. The time now is 23:39.