|
[Sponsors] |
|
June 26, 2009, 08:02 |
Postprocessing large data sets in parallel
|
#1 |
New Member
|
Hello forum,
I have computed a some large case (16 M cells) and I am going to postprocess it. Unfortunately I have some troubles with utilities foamToVTK and sample. It seems to me that I have not enough memory on the node where I start these utilities. Here is the error message: ------------------ Time = 1000 new cannot satisfy memory request. This does not necessarily mean you have run out of virtual memory. It could be due to a stack violation caused by e.g. bad use of pointers or an out of date shared ----------------- The problem is that I have only 16 Gb RAM per node and it seems to be not enough (16M cells in my case). I tryed to find a workaround: 1) In every case I restricted the amount of necessary data fields to as small as possible: foamToVTK -latestTime -fields "(p)" In this way I can get the pressure field. However, for the velocity field I cannot because I need more RAM. 2) Ok, I thought if foamToVTK needs more RAM I try sample utility. But I had no success. It crashes with the same error message. 3) I tryed to use the option -parallel with foamToVTK. But it seems to me that it only works in the distributed fields, i.e. it creates the VTK directories in processor0, processor1 etc. 4) For sample -parallel I get the strange error: "given patch field does not correspond to the mesh. Field size: 20753 mesh size: 20754" 5) What really works is foamToVTK -noInternall. In that way I produced the pressure pictures on the patches. I still need to produce a sample line at least and sample surface (nice to have). Does anybody had any more ideas how to proceed? The problem apperars always if large cases are proceeded. One needs a node with large RAM for postprocessing. Or does anybody know how to overcome this restriction? Oleksiy
__________________
************************* Cheers, Oleksiy |
|
June 29, 2009, 15:09 |
|
#2 |
Assistant Moderator
Bernhard Gschaider
Join Date: Mar 2009
Posts: 4,225
Rep Power: 51 |
If this is sufficient for you, you could use mapFields to map your results to a coarser grid of the same geometry and then postprocess that
|
|
May 2, 2011, 12:30 |
|
#3 |
Senior Member
Daniel WEI (老魏)
Join Date: Mar 2009
Location: Beijing, China
Posts: 689
Blog Entries: 9
Rep Power: 21 |
I don't think so, my experience is if you can't go with foamToVTK (be it a scalar or vector), then you cannot make mapFields work.
__________________
~ Daniel WEI ------------- Boeing Research & Technology - China Beijing, China |
|
May 26, 2011, 04:10 |
|
#4 |
Senior Member
Sandy Lee
Join Date: Mar 2009
Posts: 213
Rep Power: 18 |
Hi Daniel,
Did you solve this problem? I met the same error informations when I run " parafoam " with my 4.67 million grids. Please help out. Sandy |
|
May 26, 2011, 09:03 |
|
#5 |
Senior Member
Daniel WEI (老魏)
Join Date: Mar 2009
Location: Beijing, China
Posts: 689
Blog Entries: 9
Rep Power: 21 |
How many ram do you actual have? You can't do it with just a 12G ram
__________________
~ Daniel WEI ------------- Boeing Research & Technology - China Beijing, China |
|
May 26, 2011, 09:28 |
|
#6 |
Senior Member
Sandy Lee
Join Date: Mar 2009
Posts: 213
Rep Power: 18 |
DELL T3500 -- 24G ram, one cpu and six cores . What about it, you think?
|
|
May 26, 2011, 09:44 |
|
#7 |
Senior Member
Daniel WEI (老魏)
Join Date: Mar 2009
Location: Beijing, China
Posts: 689
Blog Entries: 9
Rep Power: 21 |
I think this is far from enough, even for a scalar field. Sorry. You need to find a larger ram.
__________________
~ Daniel WEI ------------- Boeing Research & Technology - China Beijing, China |
|
May 26, 2011, 17:26 |
|
#8 |
Senior Member
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 1,273
Rep Power: 34 |
||
May 26, 2011, 20:41 |
|
#9 |
Senior Member
Daniel WEI (老魏)
Join Date: Mar 2009
Location: Beijing, China
Posts: 689
Blog Entries: 9
Rep Power: 21 |
Sorry, my bad.
I thought she has 46.7 million mesh. Ok, 4.67m is not a big mesh, your RAM is good to handle it, if you meet any difficulty with uprime2mean, just pick it out, or you can first use foamtovtk to transform them to vtk format, then there would be no problem. Arjun, fieldview is not free!
__________________
~ Daniel WEI ------------- Boeing Research & Technology - China Beijing, China |
|
May 26, 2011, 21:35 |
|
#10 | |
Senior Member
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 1,273
Rep Power: 34 |
Quote:
Last few days I have been working on creating a cfd solver that could handle large meshes. A solver that could handle 1 billion or so cells. (3.6billion is my largest so far). And have done lots of experimentation. Post processing large data is really a big problem. My solution (which i will be working on implimenting in next few months) is that let the cfd solver handle post processing. If solver can run the calculation it can do post processing too. Here is bit of explanation to above comment. We as a user mostly concerned about say contour plots on mid plane or some planes. Animations or them. Vector plots on planes or some well defined surfaces. Iso surface values etc etc. Now the point to note that even though data is very very large, all these plots require data that could be written as a very small file. For example if you want iso surface of say vorticity. Solver can calculate that and export it is set of triangles in a file that paraview can read and load. (say vtk format). You have to just load that file and export the image that you want. |
||
May 26, 2011, 21:45 |
|
#11 |
Senior Member
Daniel WEI (老魏)
Join Date: Mar 2009
Location: Beijing, China
Posts: 689
Blog Entries: 9
Rep Power: 21 |
3.6 billion, that's huge! Are you from fieldview?
I remember I have to borrow a 128G ram to post process my results, and even then i have to use vtk outputs. And sorry, I failed to follow you, files like uprime2mean is a very huge file in it self, after it is constructed, it will larger than 15G compressed already for your 3.6billion mesh I guess, and it's just a tensor field with six components, and you also have to build its connection with points coordinates info, in all, how could you make it smaller. I see no better way, to process this tensor field except you decompose the domain into some small parts.
__________________
~ Daniel WEI ------------- Boeing Research & Technology - China Beijing, China |
|
May 26, 2011, 21:55 |
|
#12 |
Senior Member
Daniel WEI (老魏)
Join Date: Mar 2009
Location: Beijing, China
Posts: 689
Blog Entries: 9
Rep Power: 21 |
Nowadays, US nsf has some computing resources available to apply and also they are aware the need to visualize the large data files. So they have some machines with very big rams etc., and they can render the contours on a web browser, and I think this is also a nice idea. I don;t know what about in euro and japan, do you have the same resources or not?
__________________
~ Daniel WEI ------------- Boeing Research & Technology - China Beijing, China |
|
May 27, 2011, 01:00 |
|
#13 |
Senior Member
Sandy Lee
Join Date: Mar 2009
Posts: 213
Rep Power: 18 |
However, how can I reconstruct my datum distributed in every propessor by parallel? Can I directly foamtovtk ?
If I run "reconstruct", my ram is not enough too .... |
|
May 28, 2011, 19:26 |
|
#15 |
Senior Member
Sandy Lee
Join Date: Mar 2009
Posts: 213
Rep Power: 18 |
||
May 29, 2011, 03:11 |
|
#16 |
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,975
Blog Entries: 45
Rep Power: 128 |
Greetings to all!
I'm just stepping in to point out the typo... it's reconstructPar, not just reconstruct. In the terminal (at least if you are using bash) you can confirm the existence of an application by pressing the Tab key to auto-complete the name. For example: Code:
reconstr<Tab key> reconstructPar Best regards, Bruno
__________________
|
|
June 10, 2011, 00:58 |
|
#17 |
Senior Member
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 1,273
Rep Power: 34 |
just an update to my comment.
I implemented iso surface generation part. It is working well. So now for generating animation of Q criteria for 3billion cells case, I only need 1 processor paraview. Post processing files are very small and thus, i can save them with high frequency and it also means quality of animation would be much better than what I would get from saved calculation data. Sweet success. :-D |
|
June 10, 2011, 01:14 |
|
#18 | |
Senior Member
Daniel WEI (老魏)
Join Date: Mar 2009
Location: Beijing, China
Posts: 689
Blog Entries: 9
Rep Power: 21 |
Quote:
Well, mind sharing your code, it must be a great code, how did you do that..
__________________
~ Daniel WEI ------------- Boeing Research & Technology - China Beijing, China |
||
June 10, 2011, 01:16 |
|
#19 |
Senior Member
Daniel WEI (老魏)
Join Date: Mar 2009
Location: Beijing, China
Posts: 689
Blog Entries: 9
Rep Power: 21 |
"Danie -help" would result an error msg, but "Daniel -help" will not
__________________
~ Daniel WEI ------------- Boeing Research & Technology - China Beijing, China |
|
June 10, 2011, 02:22 |
|
#20 |
Senior Member
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 1,273
Rep Power: 34 |
I can outline what I did, so if someone wants to impliment it in openFOAM he can do it too.
first of all it is not difficult thing to do. Implementation does depend on data structure of solver so efficiency might be different from solver to solver. First, lets assume that solver is has control volumes and each control volume is made of some faces. Each face in turn is made up of some nodes spread in 3d space. Face could be polygonal with nf nodes numbered as n1 n2 .... nf. Take variable phi whose iso surface you want to generate. Now first step is to transfer this variable to nodes. That is calculate the values of this variable on nodes. This you anyway do if you export data for post processing , as most of post processors want node based data. Second step is to loop through all the cells one by one. Lets say you are at cell c. For this cell c collect all the faces. Since each face is just a polygon , check its edges ie n1->n2, n2->n3 , nf-1 -> nf , where the iso value falls if it falls. Could be done with the help of equation of line. If iso value happens, store this new node (xi,yi,zi) . so after you loop through all the faces of cell c, you will have say m new nodes generated. They make a polygon that you have to visualise. Store it and finally when all the cells are looped through export this in a vtk file or some format that your program can understand. For parallel run, each process would generate a part of this file. So for each iteration you would have MAX Np files (Np = number of processors). Write a small code that will read all these files, merge them and create a single file in any format that post processor can understand. I am using vtk polygons. Side Note: For cell , m nodes are generated , you would have to arrange them in an order that when it is drawn by post processor it does not look mangled. Even for very large calculations, each file is only few thousand polygons, and for this reason 1 processor post processor is good enough to create animations. I compared results with fieldview and they look very similar to it. I will be adding this into parallel unstructured grid based solver that i am working on. Will be good addition, I guess. |
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
PostChannel in parallel | maka | OpenFOAM Post-Processing | 2 | November 4, 2015 23:05 |
Smallest binary file format to save large data | Zonexo | Main CFD Forum | 2 | June 2, 2008 20:25 |
Large data file to manage | fluid | FLUENT | 0 | December 13, 2007 12:46 |
Postprocessing Pressure data spanwise over a wing | Rick | FLUENT | 1 | February 3, 2006 09:28 |
How to update polyPatchbs localPoints | liu | OpenFOAM Running, Solving & CFD | 6 | December 30, 2005 17:27 |