CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Bugs

A possible bug: Unusual slow-down using collated I/O format

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree3Likes
  • 2 Post By shaoX
  • 1 Post By olesen

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   November 29, 2020, 05:39
Default A possible bug: Unusual slow-down using collated I/O format
  #1
New Member
 
Xiao Shao
Join Date: Jun 2020
Posts: 1
Rep Power: 0
shaoX is on a distinguished road
Hello Foamers,

We know that the collated I/O option in OpenFOAM can significantly reduce the total files number when we do parallel running. But recently I found this option can be much slower than uncollated I/O. I'm using v1912.

This happens both on HPC or my workstation and I've checked all the settings (eg. OptimisationSwitches) according to https://www.openfoam.com/releases/op...2/parallel.php

Here's a simple test:
Test#1
$TUTORIAL/cavity, np=10, 40,000 cells; 0-0.2s, ∆t=1e-5, 20000 steps in total; use icoFoam.
ClockTime: collated: 110s; uncollated: 61s.

We can see there's a big difference in clockTime.

It looks like this is an input-output problem. But the following test designed to stress the I/O workload shows that I/O is not the problem.

/************************************************** *******
Lid-driven cavity flow, 6M (200*200*200) cells, ∆t=1e-4, run from 0-0.01s, totally only 100 steps, np=128.
Test#2:
Write interval: every 20 steps
ClockTime(collated): 518s
ColockTime(uncollated):511s
Test#3:
Write interval: every 2 steps
ClockTime(collated): 559s
ColockTime(uncollated):526s
************************************************** ********/

We can see based on this test the difference between collated and uncollated is negligible even Test#3 writes so frequently and writes ~60GB data in total.

The difference only becomes significant when the total running steps are high and the simulation is not big size.

I did profiling to find where computational time is spent and found ++runTime, or Foam::Time:: operator++() can be very time-consuming::
Test#4
cavity, np=10, 40,000 cells; 0-0.1s, ∆t=1e-5, 10000 steps
ClockTime=1552s; ++runTime cost: 24%
this percentage can go up to 45% if I run from 0 to 0.2s, or 20000 steps.

I need to stress this difference can be small if you are running a big simulation without too many steps. But in my situation, I need to put ++runTime into a subcycling in which fast scale equations are solved with smaller ∆t while other equations are waiting.

So the total times of executing ++runTime can be very high (millions of steps) and I find this slow-down is a serious problem.

Can someone give me any suggestion on this, or point out that I just made some stupid mistakes? Thank you very much!
Alishaha2 and HVonSch like this.
shaoX is offline   Reply With Quote

Old   July 2, 2021, 11:47
Default
  #2
New Member
 
Ali Shahanaghi
Join Date: Nov 2016
Posts: 3
Rep Power: 8
Alishaha2 is on a distinguished road
Hi Xiao Shao,

I recently faced the same problem. I am using an explicit solver based on rhoCentralFoam and I've to use small time-steps due to high Mach flow. Apparently, as you mentioned the runtime ++ operation takes more and more time with more iterations and using the collated format. I tracked down the issue and it was from
Time.C, setTime() function. The issue is when using collated format it accumulates the time step names in a pointer list and sorts them every time during ++ operation. You can check masterUncollatedFileOperation.C and the setTime() function.
Yet I could not figure out why such is list is required for the collated format but certainly, the size of the pointer list increases with the number of iterations.
hope it helps.

Ali
Alishaha2 is offline   Reply With Quote

Old   May 6, 2022, 17:39
Default
  #3
New Member
 
Hendrik von Schöning
Join Date: May 2020
Posts: 6
Rep Power: 4
HVonSch is on a distinguished road
Hello everyone,


I stumbled over this too and came to the same end as shaoX.


I think this is a huge killer for LES or DNS on HPC clusters.
Collated file format is often required there because of file system limits and, depending on the number of time steps, used cpu time can be higher by factor two to ten.


I disabled the time step list accumulation and had no problems since. Does anyone know, what unwanted consequences that could have?


/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.C


OpenFOAM-9:
line 2335: if (iter != times_.end()) -> if (false)

OpenFOAM-v2112:
line 2246: if (iter.found()) -> if (false)


Best,
Hendrik
HVonSch is offline   Reply With Quote

Old   May 7, 2022, 06:22
Default
  #4
Senior Member
 
Mark Olesen
Join Date: Mar 2009
Location: https://olesenm.github.io/
Posts: 1,503
Rep Power: 35
olesen will become famous soon enougholesen will become famous soon enough
Nice pinpointed diagnosis - I've created an issue https://develop.openfoam.com/Develop.../-/issues/2461

Please add yourself there for follow up. There may well be other aspects to discuss as well.
HVonSch likes this.
olesen is offline   Reply With Quote

Old   May 7, 2022, 13:45
Default
  #5
New Member
 
Hendrik von Schöning
Join Date: May 2020
Posts: 6
Rep Power: 4
HVonSch is on a distinguished road
Nice, thank you for the quick feedback and the bug report!
Best,
Hendrik
HVonSch is offline   Reply With Quote

Reply

Tags
collated format

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Collated File Format (in Foam-Extend) Question AndreasPe OpenFOAM Programming & Development 3 July 20, 2021 00:04
[Discretizer] Installing Discretizer umassche OpenFOAM Community Contributions 26 January 25, 2017 06:33
[General] Paraview data format conversion from vtk to parallel prog data format. odho ParaView 0 September 20, 2016 08:01
Bug in layerAdditionRemoval change to OF15 mesh format andersking OpenFOAM Bugs 2 August 6, 2008 05:17
Fluent/UNS I/O format Francisco Canabal FLUENT 2 February 3, 2000 11:24


All times are GMT -4. The time now is 15:14.