CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Post-Processing

Postprocessing large data sets in parallel

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   June 26, 2009, 08:02
Default Postprocessing large data sets in parallel
  #1
New Member
 
Oleksiy Kurenkov
Join Date: Mar 2009
Location: Nueremberg
Posts: 16
Rep Power: 8
evrikon is on a distinguished road
Send a message via Skype™ to evrikon
Hello forum,

I have computed a some large case (16 M cells) and I am going to postprocess it. Unfortunately I have some troubles with utilities foamToVTK and sample. It seems to me that I have not enough memory on the node where I start these utilities. Here is the error message:

------------------
Time = 1000
new cannot satisfy memory request.
This does not necessarily mean you have run out of virtual memory.
It could be due to a stack violation caused by e.g. bad use of pointers or an out of date shared
-----------------

The problem is that I have only 16 Gb RAM per node and it seems to be not enough (16M cells in my case). I tryed to find a workaround:

1) In every case I restricted the amount of necessary data fields to as small as possible:

foamToVTK -latestTime -fields "(p)"

In this way I can get the pressure field. However, for the velocity field I cannot because I need more RAM.

2) Ok, I thought if foamToVTK needs more RAM I try sample utility. But I had no success. It crashes with the same error message.

3) I tryed to use the option -parallel with foamToVTK. But it seems to me that it only works in the distributed fields, i.e. it creates the VTK directories in processor0, processor1 etc.

4) For sample -parallel I get the strange error:
"given patch field does not correspond to the mesh. Field size: 20753 mesh size: 20754"


5) What really works is foamToVTK -noInternall. In that way I produced the pressure pictures on the patches. I still need to produce a sample line at least and sample surface (nice to have).

Does anybody had any more ideas how to proceed? The problem apperars always if large cases are proceeded. One needs a node with large RAM for postprocessing. Or does anybody know how to overcome this restriction?

Oleksiy
__________________
*************************
Cheers, Oleksiy
evrikon is offline   Reply With Quote

Old   June 29, 2009, 15:09
Default
  #2
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,915
Rep Power: 40
gschaider will become famous soon enoughgschaider will become famous soon enough
If this is sufficient for you, you could use mapFields to map your results to a coarser grid of the same geometry and then postprocess that
gschaider is offline   Reply With Quote

Old   June 29, 2009, 17:46
Default
  #3
Senior Member
 
Sandeep Menon
Join Date: Mar 2009
Location: Amherst, MA
Posts: 386
Rep Power: 15
deepsterblue will become famous soon enough
Takuya's FOAM reader for Paraview 3.x allows post-processing for decomposed cases. You shouldn't have to reconstruct the case, but rather, post-process in parallel. This is, after all, what paraview was designed for, isn' it?
__________________
Sandeep Menon
University of Massachusetts Amherst
https://github.com/smenon
deepsterblue is offline   Reply With Quote

Old   June 30, 2009, 04:06
Default
  #4
New Member
 
Oleksiy Kurenkov
Join Date: Mar 2009
Location: Nueremberg
Posts: 16
Rep Power: 8
evrikon is on a distinguished road
Send a message via Skype™ to evrikon
Thank you on both suggestions. Especially a parallel native reader for OpenFoam seems to be very interesting.
__________________
*************************
Cheers, Oleksiy
evrikon is offline   Reply With Quote

Old   May 2, 2011, 12:30
Default
  #5
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 12
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
Quote:
Originally Posted by gschaider View Post
If this is sufficient for you, you could use mapFields to map your results to a coarser grid of the same geometry and then postprocess that
I don't think so, my experience is if you can't go with foamToVTK (be it a scalar or vector), then you cannot make mapFields work.
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 26, 2011, 04:10
Default
  #6
Senior Member
 
Sandy Lee
Join Date: Mar 2009
Posts: 207
Rep Power: 9
sandy is on a distinguished road
Hi Daniel,

Did you solve this problem? I met the same error informations when I run " parafoam " with my 4.67 million grids. Please help out.

Sandy
sandy is offline   Reply With Quote

Old   May 26, 2011, 09:03
Default
  #7
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 12
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
How many ram do you actual have? You can't do it with just a 12G ram
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 26, 2011, 09:28
Default
  #8
Senior Member
 
Sandy Lee
Join Date: Mar 2009
Posts: 207
Rep Power: 9
sandy is on a distinguished road
DELL T3500 -- 24G ram, one cpu and six cores . What about it, you think?
sandy is offline   Reply With Quote

Old   May 26, 2011, 09:44
Default
  #9
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 12
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
Quote:
Originally Posted by sandy View Post
DELL T3500 -- 24G ram, one cpu and six cores . What about it, you think?
I think this is far from enough, even for a scalar field. Sorry. You need to find a larger ram.
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 26, 2011, 17:26
Default
  #10
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 369
Rep Power: 10
arjun is on a distinguished road
Quote:
Originally Posted by lakeat View Post
I think this is far from enough, even for a scalar field. Sorry. You need to find a larger ram.
you can post process 4.7 million of data with fieldview in 2Gb ram (32bit machine).

24G is too much (may be not enough for paraview though).
arjun is offline   Reply With Quote

Old   May 26, 2011, 20:41
Default
  #11
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 12
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
Sorry, my bad.

I thought she has 46.7 million mesh.
Ok, 4.67m is not a big mesh, your RAM is good to handle it, if you meet any difficulty with uprime2mean, just pick it out, or you can first use foamtovtk to transform them to vtk format, then there would be no problem.

Arjun, fieldview is not free!
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 26, 2011, 21:35
Default
  #12
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 369
Rep Power: 10
arjun is on a distinguished road
Quote:
Originally Posted by lakeat View Post
Sorry, my bad.

I thought she has 46.7 million mesh.
Ok, 4.67m is not a big mesh, your RAM is good to handle it, if you meet any difficulty with uprime2mean, just pick it out, or you can first use foamtovtk to transform them to vtk format, then there would be no problem.

Arjun, fieldview is not free!
yaa that is true.

Last few days I have been working on creating a cfd solver that could handle large meshes. A solver that could handle 1 billion or so cells. (3.6billion is my largest so far). And have done lots of experimentation.

Post processing large data is really a big problem.

My solution (which i will be working on implimenting in next few months) is that let the cfd solver handle post processing. If solver can run the calculation it can do post processing too.

Here is bit of explanation to above comment. We as a user mostly concerned about say contour plots on mid plane or some planes. Animations or them. Vector plots on planes or some well defined surfaces.
Iso surface values etc etc.

Now the point to note that even though data is very very large, all these plots require data that could be written as a very small file. For example if you want iso surface of say vorticity. Solver can calculate that and export it is set of triangles in a file that paraview can read and load. (say vtk format). You have to just load that file and export the image that you want.
arjun is offline   Reply With Quote

Old   May 26, 2011, 21:45
Default
  #13
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 12
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
3.6 billion, that's huge! Are you from fieldview?
I remember I have to borrow a 128G ram to post process my results, and even then i have to use vtk outputs.

And sorry, I failed to follow you, files like uprime2mean is a very huge file in it self, after it is constructed, it will larger than 15G compressed already for your 3.6billion mesh I guess, and it's just a tensor field with six components, and you also have to build its connection with points coordinates info, in all, how could you make it smaller.

I see no better way, to process this tensor field except you decompose the domain into some small parts.
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 26, 2011, 21:55
Default
  #14
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 12
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
Nowadays, US nsf has some computing resources available to apply and also they are aware the need to visualize the large data files. So they have some machines with very big rams etc., and they can render the contours on a web browser, and I think this is also a nice idea. I don;t know what about in euro and japan, do you have the same resources or not?
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 27, 2011, 01:00
Default
  #15
Senior Member
 
Sandy Lee
Join Date: Mar 2009
Posts: 207
Rep Power: 9
sandy is on a distinguished road
However, how can I reconstruct my datum distributed in every propessor by parallel? Can I directly foamtovtk ?

If I run "reconstruct", my ram is not enough too ....
sandy is offline   Reply With Quote

Old   May 27, 2011, 01:04
Default
  #16
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 12
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
pls read:
Code:
reconstruct -help
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 28, 2011, 06:55
Default
  #17
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 369
Rep Power: 10
arjun is on a distinguished road
Quote:
Originally Posted by lakeat View Post
3.6 billion, that's huge! Are you from fieldview?
I remember I have to borrow a 128G ram to post process my results, and even then i have to use vtk outputs.

And sorry, I failed to follow you, files like uprime2mean is a very huge file in it self, after it is constructed, it will larger than 15G compressed already for your 3.6billion mesh I guess, and it's just a tensor field with six components, and you also have to build its connection with points coordinates info, in all, how could you make it smaller.

I see no better way, to process this tensor field except you decompose the domain into some small parts.

Data will always be partitioned.
The idea is that if you can run the calculations, you can use the same number of processors to do the post processing. So each processor is responsible for the data it produces.

What i want to do is that make each processor do the post processing, at the moment paraview is one possibility (fieldview is out because can not afford 500 licenses of it).

The problem i am facing with paraview is that while cfd solver can run the job, paraview is asking for much more memory and hence can not post process data even with same number of processors.


You are correct though, when case is large, things are very different, data sizes are huge. All these are challenges we need to overcome.
arjun is offline   Reply With Quote

Old   May 28, 2011, 19:26
Default
  #18
Senior Member
 
Sandy Lee
Join Date: Mar 2009
Posts: 207
Rep Power: 9
sandy is on a distinguished road
Quote:
Originally Posted by lakeat View Post
pls read:
Code:
reconstruct -help
Hi lakeat, I get the error information again. Maybe I should try:

Danie -help

?
sandy is offline   Reply With Quote

Old   May 29, 2011, 03:11
Default
  #19
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,301
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all!

I'm just stepping in to point out the typo... it's reconstructPar, not just reconstruct.

In the terminal (at least if you are using bash) you can confirm the existence of an application by pressing the Tab key to auto-complete the name. For example:
Code:
reconstr<Tab key>
reconstructPar
In case it doesn't show the rest, try pressing the Tab key twice to see if it shows a list of visible applications that start with the same characters.

Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   June 10, 2011, 00:58
Default
  #20
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 369
Rep Power: 10
arjun is on a distinguished road
just an update to my comment.


I implemented iso surface generation part. It is working well. So now for generating animation of Q criteria for 3billion cells case, I only need 1 processor paraview. Post processing files are very small and thus, i can save them with high frequency and it also means quality of animation would be much better than what I would get from saved calculation data.


Sweet success. :-D
arjun is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Smallest binary file format to save large data Zonexo Main CFD Forum 2 June 2, 2008 20:25
Large data file to manage fluid FLUENT 0 December 13, 2007 13:46
PostChannel in parallel maka OpenFOAM Post-Processing 1 May 30, 2006 05:37
Postprocessing Pressure data spanwise over a wing Rick FLUENT 1 February 3, 2006 10:28
How to update polyPatchbs localPoints liu OpenFOAM Running, Solving & CFD 6 December 30, 2005 18:27


All times are GMT -4. The time now is 04:38.