CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Programming & Development

Problems with very large IO dictionaries in parallel

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   May 11, 2021, 07:13
Default Problems with very large IO dictionaries in parallel
  #1
New Member
 
Ace Ventura
Join Date: Nov 2017
Posts: 4
Rep Power: 8
kingdomkarl is on a distinguished road
Hello everyone,

I'm experiencing weird behavior with OF when it comes to very large IO dictionaries. Basically, I am interpolating a lot of my fluid variables from a large multidimensional table. Due to the number of dimensions these tables tend to get quite large (easily 10GB+).

When running my case in serial everything is fine and the initial results look good. When running in parallel with smaller tables (< 2-3 GB) also everything is fine and the results are in agreement with experimental data. Now when I increase my table size above 2-3 GB and run the case in parallel the solver "hangs" when reading in the dictionary and does not advance nor give any kind of error message. Again on single core the large tables work fine, so the table file shouldn't be compromised.

RAM requirement should also be fine on the machine I'm using, although I'm guessing that what I'm doing is terribly inefficient since I'm utilizing only MPI and no shared memory message passing (eg. openMP).

The IO read in for the table as an IODictionary looks like the following:

IOdictionary control
(
IOobject
(
fileName_,
instance,
obr,
IOobject::MUST_READ,
IOobject::NO_WRITE
)
);
control.readEntry("fields", entries_);

So basically a direct copy of the readIn method in "interpolationLookUpTable".

For any hints on how to solve this issue I'd be really thankful
kingdomkarl is offline   Reply With Quote

Old   May 21, 2021, 16:09
Default
  #2
Senior Member
 
Klaus
Join Date: Mar 2009
Posts: 250
Rep Power: 21
klausb will become famous soon enough
Maybe this helps:

There's a section called "OptimisationSwitches" in /home/.../OpenFOAM/OpenFOAM-8/etc/controlDict which deals with Parallel IO handling. It could help to set it to "collated" and set a buffer size...

Here you can find a comprehensive explanation what it is about: https://github.com/OpenFOAM/OpenFOAM...ae435d4ab42e6f

Basically, you'll run the case in parallel but now, IO is only done and managed through the master process. If you write your own IO code for your purposes, it might help to leverage this functionality.
klausb is offline   Reply With Quote

Old   May 22, 2021, 10:20
Default
  #3
HPE
Senior Member
 
HPE's Avatar
 
Herpes Free Engineer
Join Date: Sep 2019
Location: The Home Under The Ground with the Lost Boys
Posts: 932
Rep Power: 12
HPE is on a distinguished road
There are also ADIOS and ADIOS2 in the OpenFOAM modules.
HPE is offline   Reply With Quote

Old   May 22, 2021, 14:13
Default
  #4
New Member
 
Ace Ventura
Join Date: Nov 2017
Posts: 4
Rep Power: 8
kingdomkarl is on a distinguished road
Hey,

they both seems like viable options, thank you very much for the quick replies I'll dig through it next week and will post my updates accordingly.
kingdomkarl is offline   Reply With Quote

Old   September 28, 2021, 11:28
Talking
  #5
New Member
 
Balaji
Join Date: May 2013
Posts: 21
Rep Power: 12
Nealcaffrey is on a distinguished road
Hi kingdomkarl,

Any updates on the collated cluster run?

Did it work on the cluster?

Please post your script for decompose, runnning and reconstructing.

I am trying the same on cluster to save some file limit. Would be nice if you could share your updates and how you overcome the issue

Regards
Nealcaffrey is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Partitioning problems in the parallel job Neo FLUENT 5 January 26, 2021 11:10
[OpenFOAM.com] problems running in parallel on Mac OS X and Windows: only 1 cpu windscion OpenFOAM Installation 20 June 5, 2020 17:27
directMapped + regionCoupling + parallel problems elisabet OpenFOAM Running, Solving & CFD 15 October 3, 2018 11:04
MPI parallel meet problems jiaojiao OpenFOAM 1 May 10, 2016 03:30
Search of large problems Rashid Faizullin Main CFD Forum 0 September 10, 1998 14:46


All times are GMT -4. The time now is 21:37.