CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Programming & Development

reconstructPar and a high number of snapshots

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   April 17, 2012, 02:29
Default reconstructPar and a high number of snapshots
  #1
Senior Member
 
Fabian Schlegel
Join Date: Apr 2009
Location: Dresden, Germany
Posts: 184
Rep Power: 8
fs82 is on a distinguished road
Dear Foam Developers,

I am currently experiencing a little problem with reconstructPar. The statistics of OpenFoam (LES) are limited to <Ui> and <Ui'Uj'>. For most of the eningeering purposes this is enough. But for meteorological applications one needs correlations, high order statistics and more. The most used method is to use a high number of snapshots (>900). I have done so and my case (11 million grid points) runs very good on 640 cores of a AMD Bulldozer Cluster. I saved 1800 snapshots and now they are stored in the processor directories: one file for each field and time and processor. One reconstructPar job for all times is much to slow for all of the 1800 snapshots, one reconstructPar job for each time is a big problem for the file system and does not work also. Limiting the number of reconstructPar jobs to 32 (still one job for one time) gives a annoying behavior: some went through within 1 hour, some not within 12 hours. I have to say, I am not alone on this machine and I have to share it. Therefore exists a queuing system which limits the execution time.
Most of the other codes I know uses one result file per field even in parallel runs. The fields of the processors are combined during runtime and than stored. May be it is possible to include such a feature into OpenFoam? This would reduce the amount of time for postprocessing very much?

Kind regards,
Fabian
fs82 is offline   Reply With Quote

Old   April 17, 2012, 15:15
Default
  #2
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,510
Blog Entries: 34
Rep Power: 86
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Fabian,

Wow, that's a whole lot of data. And I think you're not alone when it comes to post-processing this amount of OpenFOAM data from clusters, so until someone else more experience can give his/her very welcome hints, here are mine...

OK, what kind of post-processing are you going to need?
  • If it's all done with ParaView and memory-wise the case fits in a single workstation, then you can use the internal ".foam" plugin for opening the files. As of OpenFOAM 2.0.0 and/or ParaView 3.8, you can run like this:
    Code:
    paraFoam -builtin
    If this doesn't work, simply run:
    Code:
    touch case.foam
    paraview
    And then open the file "case.foam".
  • If you don't have a workstation that has that much memory and disk space, as well as only requiring ParaView... then it's possible but it's going to be a bit crazy to do.... perhaps it's best to see the idea after this list. Nonetheless, if this is the case, you can ask how to do it here as well
  • If you need to do post-processing with something other than ParaView... things are going to get tricky. If the post-processing tools also use OpenFOAM code, then you should also use them in parallel.
OK, the final idea - when it comes to supercomputing and OpenFOAM, it's useful to do either one or both of the following:
  • Go to one of the more extensive workshops about OpenFOAM.
  • Request commercial support from anyone that properly supports OpenFOAM on clusters.


Last but not least, if you want the official OpenFOAM developers to see your idea, either:
If you want this to be done "for free" (takes a whole lot longer in most cases, if at all) by the community, you can either try posting the request at:
Best regards,
Bruno
wyldckat is offline   Reply With Quote

Old   April 18, 2012, 04:37
Default
  #3
Senior Member
 
Fabian Schlegel
Join Date: Apr 2009
Location: Dresden, Germany
Posts: 184
Rep Power: 8
fs82 is on a distinguished road
Thx Bruno for your detailed answer. I am using OF now for 3 years always on HPCs and I am still very happy with it. I am with you that visiting a workshop or buying support would be the best solution. Nevertheless I am working at the university (TU Dresden) and as you might know money for support, licences and workshops is always a big problem. Therefore I have to help myself or ask the forum, which helped me a lot in the past.

Quote:
Originally Posted by wyldckat View Post
Wow, that's a whole lot of data.
Yes it is, but I think is looks also more complicated than it is. I need high order statistics (skewness, kurtosis, correlations) for postprocessing. This is not implemented so far but if one has enough snapshots this is an easy OF utility ... I hope so. I use Tecplot for visualization, which would be the answer if paraview would be an option.

Quote:
Originally Posted by wyldckat View Post
OK, what kind of post-processing are you going to need?
I am only interested in statistics of the velocity field. This means I only need the instantaneous velocity field of the snapshots. I will do the postprocessing on a single core linux machine with 96GB memory. I think this should be enough. OF library provides the basic IO (mesh loading, reading fields of varios times, writing and basic mathematics) and I just have to combine it to a little tool. I hope it would be so easy as I imagine but I will see.

Quote:
Originally Posted by wyldckat View Post
If you want this to be done "for free" (takes a whole lot longer in most cases, if at all) by the community, you can either try posting the request at:
This is was one of the reasons, why I posted it in this subforum. If I have some time I could write a bug report. This was a question to the forum if somebody has the same problems and a solution. Also it was a question to the developers if it is possible to add a advanced parallel IO which gives the user the requested field at the and of the computational run. Most of the academic codes I know (semtex, lesocc) have such a feature. May be it is possible to add it to OF in the future.

Now something new: I found a workaround yesterday. Our queuing-system allows array jobs and one can specify the number of jobs running at the same time. Also reconstructPar support time ranges. Combining such a time range with an array job with limited jobs running parallel reduces the load for the file system and the jobs seems to have not such a high difference in the runtime.

kind regards,
Fabian
fs82 is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
bash script for pseudo-parallel usage of reconstructPar kwardle OpenFOAM Post-Processing 32 July 19, 2015 21:25
reconstructPar --> fileName::stripInvalid() called for invalid fileName commandtouse adona058 OpenFOAM Bugs 18 June 12, 2015 18:47
Problem with reconstructPar Jochem OpenFOAM Post-Processing 3 March 24, 2011 13:44
Problem with reconstructPar, please HELP quartzian OpenFOAM 1 September 24, 2009 06:11
Problem with reconstructPar fabianpk OpenFOAM 5 August 14, 2007 09:17


All times are GMT -4. The time now is 17:11.