CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Running, Solving & CFD

Used Memory Accumulates During Course of Simulation Until interFoam gets Killed

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By HPE

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   February 12, 2021, 15:57
Default Used Memory Accumulates During Course of Simulation Until interFoam gets Killed
  #1
Senior Member
 
Claudio Boezio
Join Date: May 2020
Location: Europe
Posts: 137
Rep Power: 5
Ship Designer is on a distinguished road
I let a simulation run overnight and the next day I saw that it was aborted for no apparent reason other than this message some time after it started:
Code:
[…]

GAMG:  Solving for p_rgh, Initial residual = 0.114596, Final residual = 0.000297992, No Iterations 2
GAMG:  Solving for p_rgh, Initial residual = 0.000313041, Final residual = 2.35967e-06, No Iterations 7
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 0 on node 701f164b7b63 exited on signal 9 (Killed).
--------------------------------------------------------------------------
I use OpenFOAM v8 on a Mac with Docker and at the time, Docker was set to 8 GB of memory for containers. When I checked memory usage in macOS' Activity Monitor, it reported approx 600 MB for the com.docker.hyperkit process. This is the value I usually see when running similar computations and I wondered more than once, how OpenFOAM can use so little memory for cases with > 1M cells…

I restarted the computation from the last saved time step and it was killed again. Since this mesh had more cells then I usually deal with (~2.7M) I increased the setting from 8 GB to 12 GB in Docker. It took longer this time but it was killed again eventually. Confused by the much lower used memory reported by Activity Monitor, I decided that I needed a way to monitor the memory usage from within the container while the computation si running. I added this function object to controlDict
Code:
FreeMemorySystemCall {
             type  systemCall;
             libs  ("libutilityFunctionObjects.so");

     executeCalls  ();
       writeCalls  ("free");
         endCalls  ();
     writeControl  timeStep;
    writeInterval  10;
 }
which would output something like this every 10 time steps (values are in kB):
Code:
              total        used        free      shared  buff/cache   available
Mem:       12268828     7852356     3653360         768      763112     4214616
Swap:       2097148           0     2097148
I didn't analize the complete logs yet, but so far I have seen increases in used memory of anything between 0 and up to 240 MB every 10 time steps. The used memory keeps increasing until ~1 GB of memory is left available, at which point the used memory in Activity Monitor starts to increase. After a while though, interFoam gets killed again.

Obviously interFoam has to store in memory all the fields and other data at least until it writes them to permanent storage. The steady increase in memory usage however suggests that data is stored much longer than that or that this is some sort of memory leakage. I would assume that since the mesh is static and the simulation of a pseudo steady state type, the total amount of average data across time steps shouldn't increase. I can't think of any runtime-generated data — in such quantities at least — that would be needed to be accumulated in memory for the entire duration of the computation, or is there?

Is this behaviour intentional or is this an indication of memory-related bug?

If interFoam is stopped and restarted again (without restarting Docker), memory usage is reset (aaumulation disappears). Memory usage right after starting the OpenFOAM container is ~350 MB. After starting interFoam, it jumps to several GB which seems reasonable (depending on mesh size) but then keeps steadily increasing.

Some background info about the case: it is a ship hull resistance calculation whit static mesh, ~2.7M cells, solved by interFoam in parallel mode and local time stepping. RAS is used with turbulence switched off. Simulation and field values progress as expected with clear tendency to converge. Function objects used are forces, fieldMinMax and yPlus.

For those not familiar with macOS, Activity Monitor is the built in utility that reports the resources usage of the system (CPU, Memory, Energy etc.). I'm sure that the heavily underreported memory used by the container has its reasons is not an OpenFOAM issue. It rather misled me to believe that running OpenFOAM is not memory intense, also because I never encountered this problem before (killed siumlation due to low memory).

I would be grateful for any hints, thanks! Have a nice weekend,

Claudio
Ship Designer is offline   Reply With Quote

Old   February 12, 2021, 16:36
Default
  #2
HPE
Senior Member
 
HPE's Avatar
 
Herpes Free Engineer
Join Date: Sep 2019
Location: The Home Under The Ground with the Lost Boys
Posts: 932
Rep Power: 11
HPE is on a distinguished road
Hi,

- is it possible for you to run `Massif` on your application and share the output here (if you know how to use it)?
- or a tutorial case with the same settings that reproduces the issue?
- can you comment out the function objects and rerun the simulation? (please comment them out instead of using `enabled no;` keyword pair).
Ship Designer likes this.
HPE is offline   Reply With Quote

Old   February 13, 2021, 10:43
Default
  #3
Senior Member
 
Claudio Boezio
Join Date: May 2020
Location: Europe
Posts: 137
Rep Power: 5
Ship Designer is on a distinguished road
Thank you HPE for your quick reply. I have to look into "Massif" as I'm not familiar with it. If I manage to get it to work I'll be happy to share any results. I'm currently doing computations for a project and will be able to make some test calculations in a couple of days time. As you suggested I commented out most function objects except forces and fieldMinMax which are necessary and the problem persists. I'll get back to you as soon as I can run some test computations.

Cheers, Claudio
Ship Designer is offline   Reply With Quote

Old   February 24, 2021, 09:58
Default
  #4
Senior Member
 
Claudio Boezio
Join Date: May 2020
Location: Europe
Posts: 137
Rep Power: 5
Ship Designer is on a distinguished road
Just a quick update, I was able to narrow down the problem to the div(phi,alpha) Gauss MPLIC scheme that I used as the cause of the increasing memory usage over the course of the computation. Since both MPLIC and PLIC were inroduced with OpenFOAM v8, I did a run with PLIC as well. Below is the memory usage of the v8 DTC Hull tutorial case found in /tutorials/multiphase/interFoam/RAS/DTCHull running in parallel on 4 cores:
Code:
           Scheme  div(phi,alpha) Gauss interfaceCompression vanLeer 1 (original of tutorial)
Memory usage [kB]  
      at Time   0  1651880 
      at Time  10  2077792
      at Time 500  2043216


           Scheme  div(phi,alpha) Gauss PLIC interfaceCompression vanLeer 1
Memory usage [kB]  
      at Time   0  1677056 
      at Time  10  2577572
      at Time 500  4120516


           Scheme  div(phi,alpha) Gauss MPLIC
Memory usage [kB]  
      at Time   0  1677184 
      at Time  10  2601868
      at Time 500  4121904
I let the computation run for 500 time steps which is sufficient to expose the increase of memory usage over time. It can be seen that the memory usage with the first scheme remains roughly constant. I can look a bit more into it but if it is too time consuming, I won't be able to perform useful diagnostics. I understand that using massif can slow down program execution up to 20 times, which is prohibitive for me as I have other work to do with OpenFOAM. I'll keep you posted.

Cheers, Claudio
Ship Designer is offline   Reply With Quote

Old   April 1, 2021, 21:35
Default
  #5
Senior Member
 
Claudio Boezio
Join Date: May 2020
Location: Europe
Posts: 137
Rep Power: 5
Ship Designer is on a distinguished road
I apologize for my late follow-up on this and it appears that the problem was already solved. Since making any modifications to a Docker container is not very straightforward, I tried to do the profiling with Valgrind on an OpenFOAM v8 installation I have on Ubuntu 18 in VirtualBox. Surprise surprise, when I ran the case, memory usage didn't increase over time…

After a bit of research I then remembered that I had installed the OpenFOAM v8 on that virtual machine sometime after the v8 Docker initial release. It then occurred to me, that it was the patched version that I installed an the virtual machine. I then downloaded and installed the latest (patched) v8 Docker image — which I internally call OpenFOAM v8.1 — and the problem was gone. MPLIC now works like a charm, without increasing memory usage over time. The Docker images available for download are all called v8, even the patched one, and it's difficult to tell them apart. I would welcome the Docker image designations to be more specific.

I can't explain why the error disappeared, since except 1-2 source files that seem not directly related, the source files were last modified at the time of the v8 initial release and not later. I also couldn't find any release notes that suggest that such a bug was fixed and thus it never occurred to me to update my initial v8 Docker container. Go figure…

Below is again the memory usage of the v8 DTC Hull tutorial case found in /tutorials/multiphase/interFoam/RAS/DTCHull running in parallel on 4 cores as above, except this time running on the patched Docker OpenFOAM v8:

Code:
           Scheme  div(phi,alpha) Gauss interfaceCompression vanLeer 1 (original)
Memory usage [kB]  
      at Time   0  1651880 
      at Time  10  2077792
      at Time 500  2043216

           Scheme  div(phi,alpha) Gauss PLIC interfaceCompression vanLeer 1
Memory usage [kB]  
      at Time   0  1605536 
      at Time  10  2493776
      at Time 500  2532268 (okay)

           Scheme  div(phi,alpha) Gauss MPLIC
Memory usage [kB]  
      at Time   0  1612616 
      at Time  10  2477772
      at Time 500  2525464 (okay)
Cheers, Claudio
Ship Designer is offline   Reply With Quote

Old   July 21, 2022, 20:59
Default
  #6
Senior Member
 
Bobby
Join Date: Oct 2012
Location: Ames, Iowa
Posts: 454
Rep Power: 14
babakflame is on a distinguished road
Hi Claudio

Is it possible to elaborate how did you get the memory usage of each scheme??

Thank you
babakflame is offline   Reply With Quote

Reply

Tags
interfoam, killed, memory

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
interFoam nozzle simulation diverges redTo OpenFOAM Running, Solving & CFD 0 May 15, 2020 11:53
interFoam - sudden simulation crash fmerk OpenFOAM Running, Solving & CFD 3 October 25, 2017 04:13
Wrong BC for VOF simulation using interFoam for flow past plate with k-omega-sst Bashar OpenFOAM Running, Solving & CFD 20 March 31, 2017 08:35
GPU simulation Fluent out of memory xab Hardware 2 January 12, 2017 06:21
Lenovo C30 memory configuration and discussions with Lenovo matthewe Hardware 3 October 17, 2013 11:23


All times are GMT -4. The time now is 03:15.