CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Paraview & paraFoam

Paraview Free/Delete all memory

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   January 18, 2013, 06:26
Default Paraview Free/Delete all memory
  #1
New Member
 
Join Date: Feb 2012
Posts: 12
Rep Power: 6
eric_albin is on a distinguished road
Hello,

I am encountering some difficulties to remove all the data from paraview.
After loading several time steps of a pvd files (Multi-Block Data) using a python script, the pc runs out of memory and paraview crashes. However, after each time steps, I delete all the variables with instructions "Delete(var)" and "del var".
It is the same problem if I do a Edit/Delete all, I still have the memory used according to the Memory inspector "client"...
The only way I see to free the memory is to kill the process paraview from a terminal or to kill the client in the memory inspector. Nevertheless, this also closes paraview.

Does someone knows how to delete all the memory between the first time-step and the second load? Maybe there is a way using server connections.
My idea would be :
1) create a connection.
2) load the data and run a job.
3) close the server before reopening a new one to compute the second load...

Do you have a solution, suggestions or other ideas?
Have a good day,
Eric
eric_albin is offline   Reply With Quote

Old   January 19, 2013, 15:54
Default
  #2
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,531
Blog Entries: 39
Rep Power: 97
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Eric,

I know that ParaView is known to have memory leaking problems, but last reports I've seen indicate that ParaView 3.98.0 no longer has this problem... it's here: ParaView fills my RAM

Best regards,
Bruno
__________________
wyldckat is offline   Reply With Quote

Old   January 20, 2013, 08:34
Default
  #3
New Member
 
Join Date: Feb 2012
Posts: 12
Rep Power: 6
eric_albin is on a distinguished road
Dear Bruno,
Thank you a lot for your comment. Indeed, I read that too and I am currently using this version of paraview (3.98) because of this problem and because it has additional export possibilities in vectorial format...

Nevertheless, I still get memory leaking problems with this version. Maybe because I am controlling paraview from python scripts... In fact, I am using scripts because I did not find some capabilities like saving automatically contours in new vtk files using filters, or computing and saving computed area of surfaces or volumes but I know how to do that using scripts...

Thanks again and best regards,
Eric
eric_albin is offline   Reply With Quote

Old   September 12, 2015, 18:31
Default
  #4
New Member
 
Join Date: Jul 2013
Posts: 4
Rep Power: 5
komodo is on a distinguished road
Dear Foamers,

this discussion is a little old, but maybe somebody may still be interested in the topic.

I'm using Paraview 4.1.0 (the version that comes with OF 2.4.0) and I'm trying to export in batch some data (from vtk files to csv files) using pvpython.

The script loops over some 1000s of files in several directories and after 6000-7000 files a memory leak error happens, which is quite annoying considering that I'm using a 32 GB workstation...

In the script loop I delete all the objects I generated within a single loop instance to try free memory but this doesn't help.

Has anybody found a solution to this memory leak issue?
Thanks and Best Regards
Cristian
komodo is offline   Reply With Quote

Old   September 13, 2015, 15:50
Default
  #5
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,531
Blog Entries: 39
Rep Power: 97
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Quick answer: Without example code for reproducing the same error, it's pretty hard to try and diagnose the issue. Either way, have you seen this answer: http://stackoverflow.com/a/26514796 ?
wyldckat is offline   Reply With Quote

Old   September 13, 2015, 18:02
Default
  #6
New Member
 
Join Date: Jul 2013
Posts: 4
Rep Power: 5
komodo is on a distinguished road
Dear wyldckat,

thanks for your reply.

I was already aware of the solution proposed in the link you posted. I actually found it on the main Paraview wiki: http://www.paraview.org/Wiki/ParaView/Python_Scripting

I am not a python expert and maybe I made some mistakes with the Delete() and del functions. I here report my script (I hope I am doing it in the right way):

Code:
#Script used to export csv from vtk datafiles
import fnmatch
import os
import sys
import re
from paraview.simple import *
#servermanager.Connect()

# Directories where to run
#dirs = ["01mm", "02mm", "03mm", "04mm", "05mm"]
dirs = ["01mm"]

# VTK file names
rhoBasename = "rho_sampleSurf_"  
rhoOutName =  "rho_coord."          

# Generate the list of files on which to operate
# and the corresponding output csv names
files = []
for dd in dirs:
    for ff in os.listdir("./"+dd):
        if fnmatch.fnmatch(ff, rhoBasename+"*"):
            nameRead = "./" + dd + "/" + ff
            numb = (re.findall("\d+", nameRead))[1]
            nameWrite = "./" + dd + "/" + rhoOutName + numb + ".csv"
            files.append([nameRead,nameWrite])

#print files
#sys.exit()


print "Start export"
zz = 1
for ff in files:
    if (zz%1000==0):
        print "written " + str(zz) + " csv files\n"
    # Open the correct file
    reader = OpenDataFile(ff[0])
    # Apply filters to it
    pd2cd = PointDatatoCellData(Input = reader)
    cc = CellCenters(Input = pd2cd)
    # Prepare the output name

    # Write the file
    writer = CreateWriter(ff[1], cc)
    # NO NEED IF LOOP DONE EXPLICITLY writer.WriteAllTimeSteps = 1
    writer.FieldAssociation = "Points"
    writer.UpdatePipeline()

    # Delete the generated objects
    del writer
    cc.Input = []
    Delete(cc)
    del cc
    pd2cd.Input = []
    Delete(pd2cd)
    del pd2cd
    Delete(reader)
    del reader

    zz += 1

print "Export completed"
I run the script by using "pvpython script.py" without any particular option.

Thanks for the help,
Best Regards,
Cristian
komodo is offline   Reply With Quote

Old   September 15, 2015, 07:03
Default
  #7
Member
 
Mikko
Join Date: Jul 2014
Location: The Hague, The Netherlands
Posts: 82
Rep Power: 4
Flowkersma is on a distinguished road
Hi Cristian,

Probably a better way to loop over files is to first build the pipeline and then only change the reader's and writer's filenames. Here is an example script:
Code:
reader = OpenDataFile(files[0][0])
pd2cd = PointDatatoCellData(Input = reader)
cc = CellCenters(Input = pd2cd)
writer = CreateWriter(files[0][1], cc)
writer.FieldAssociation = "Points"
for ff in files:     
   reader.FileNames = ff[0]
   writer.FileName = ff[1]
   writer.UpdatePipeline()
Now you don't have to worry about deleting the filters because you are using same filters all the time.

-Mikko
Flowkersma is offline   Reply With Quote

Old   September 15, 2015, 09:22
Default
  #8
New Member
 
Join Date: Jul 2013
Posts: 4
Rep Power: 5
komodo is on a distinguished road
Dear Mikko,

Thanks for your suggestion.

Your snippet made my day!

Now paraview doesn't leak memory while the script is running!.

Maybe creating / deleting a reader and a writer at any loop instance (as I used to do) made paraview leaving some garbage in memory...

I report here the new version of the script since maybe it could be useful to others having similar needs/issues:

Code:
#Script used to export csv from rho datafiles
import fnmatch
import os
import sys
import re
from paraview.simple import *

# Directories where to run
dirs = ["01mm", "02mm", "03mm", "04mm", "05mm"]

# File rho names
rhoBasename = "rho_sampleSurf_"     #rho_sampleSurf_2576.vtk 
rhoOutName =  "rho_coord."          #rho_coord.550.csv

# Generate the list of files on which to operate
# and the corresponding output csv names
files = []
for dd in dirs:
    for ff in os.listdir("./"+dd):
        if fnmatch.fnmatch(ff, rhoBasename+"*"):
            nameRead = "./" + dd + "/" + ff
            numb = (re.findall("\d+", nameRead))[1]
            nameWrite = "./" + dd + "/" + rhoOutName + numb + ".csv"
            files.append([nameRead,nameWrite])

print "Start export"

# Generate the export pipeline
reader = OpenDataFile(files[0][0])
pd2cd = PointDatatoCellData(Input = reader)
cc = CellCenters(Input = pd2cd)
writer = CreateWriter(files[0][1], cc)
writer.FieldAssociation = "Points"

# Processed files counter
zz = 1

for ff in files:
    # Print out the counter value from time to time
    if (zz%1000==0):
        print "written " + str(zz) + " csv files\n"

    # Update the pipeline and write the csv file
    reader.FileNames = ff[0]
    writer.FileName = ff[1]
    writer.UpdatePipeline()

    # Update counter
    zz += 1

print "Export completed"
Thanks again,
Best regards,
Cristian
komodo is offline   Reply With Quote

Old   September 15, 2015, 10:05
Default
  #9
Member
 
Mikko
Join Date: Jul 2014
Location: The Hague, The Netherlands
Posts: 82
Rep Power: 4
Flowkersma is on a distinguished road
I'm glad that it worked.

I just wanted to add that I think Paraview is not leaking because I've made same kind of loops with big datasets and for many files. I think you are not deleting the writer. Thus at the end of your loop you should delete all the sources in inverse order than generating them:

Code:
    # Delete the generated objects     
   Delete(writer)     
   Delete(cc)     
   Delete(pd2cd)
   Delete(reader)
If you are deleting all the filters like in your case, you can also:
Code:
for key, value in GetSources().items():
   Delete(value)
   print 'Deleted:', key, value
Flowkersma is offline   Reply With Quote

Old   September 15, 2015, 11:33
Default
  #10
New Member
 
Join Date: Jul 2013
Posts: 4
Rep Power: 5
komodo is on a distinguished road
Dear Mikko,

Thanks again for your reply.

I actually tried several combination of del / Delete() deleting the pipeline objects in reverse order, but without success.

In particular paraview complained for the "Delete(writer)" line...
You can see an attempt of that in my second post where I just do "del writer"

Probably I made some mistakes somewhere....

I anyway prefer the solution you posted since changing with a loop the reader input seems to me more clever and efficient rather than creating/deleting the reader at every time.

I actually wasn't aware of that possibility...good to know for my next scripts

Thanks and Best Regards,
Cristian
komodo is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Paraview 3.12 memory allocation problem while saving animation jgomez OpenFOAM Paraview & paraFoam 17 September 14, 2013 17:03
Newbie: Install ParaView 3.81 on OF-1.6-ext/OpenSuse 11.2? lentschi OpenFOAM Installation 1 March 9, 2011 03:32
paraFoam reader for OpenFOAM 1.6 smart OpenFOAM Installation 13 November 16, 2009 22:41
Memory error? Young CFX 3 September 30, 2008 11:33
CFX CPU time & real time Nick Strantzias CFX 8 July 23, 2006 17:50


All times are GMT -4. The time now is 09:02.