CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Bugs

A possible bug: Unusual slow-down using collated I/O format

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   November 29, 2020, 04:39
Default A possible bug: Unusual slow-down using collated I/O format
  #1
New Member
 
Xiao Shao
Join Date: Jun 2020
Posts: 1
Rep Power: 0
shaoX is on a distinguished road
Hello Foamers,

We know that the collated I/O option in OpenFOAM can significantly reduce the total files number when we do parallel running. But recently I found this option can be much slower than uncollated I/O. I'm using v1912.

This happens both on HPC or my workstation and I've checked all the settings (eg. OptimisationSwitches) according to https://www.openfoam.com/releases/op...2/parallel.php

Here's a simple test:
Test#1
$TUTORIAL/cavity, np=10, 40,000 cells; 0-0.2s, ∆t=1e-5, 20000 steps in total; use icoFoam.
ClockTime: collated: 110s; uncollated: 61s.

We can see there's a big difference in clockTime.

It looks like this is an input-output problem. But the following test designed to stress the I/O workload shows that I/O is not the problem.

/************************************************** *******
Lid-driven cavity flow, 6M (200*200*200) cells, ∆t=1e-4, run from 0-0.01s, totally only 100 steps, np=128.
Test#2:
Write interval: every 20 steps
ClockTime(collated): 518s
ColockTime(uncollated):511s
Test#3:
Write interval: every 2 steps
ClockTime(collated): 559s
ColockTime(uncollated):526s
************************************************** ********/

We can see based on this test the difference between collated and uncollated is negligible even Test#3 writes so frequently and writes ~60GB data in total.

The difference only becomes significant when the total running steps are high and the simulation is not big size.

I did profiling to find where computational time is spent and found ++runTime, or Foam::Time:: operator++() can be very time-consuming::
Test#4
cavity, np=10, 40,000 cells; 0-0.1s, ∆t=1e-5, 10000 steps
ClockTime=1552s; ++runTime cost: 24%
this percentage can go up to 45% if I run from 0 to 0.2s, or 20000 steps.

I need to stress this difference can be small if you are running a big simulation without too many steps. But in my situation, I need to put ++runTime into a subcycling in which fast scale equations are solved with smaller ∆t while other equations are waiting.

So the total times of executing ++runTime can be very high (millions of steps) and I find this slow-down is a serious problem.

Can someone give me any suggestion on this, or point out that I just made some stupid mistakes? Thank you very much!
shaoX is offline   Reply With Quote

Reply

Tags
collated format

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Collated File Format (in Foam-Extend) Question AndreasPe OpenFOAM Programming & Development 2 June 8, 2018 09:11
[Discretizer] Installing Discretizer umassche OpenFOAM Community Contributions 26 January 25, 2017 05:33
[General] Paraview data format conversion from vtk to parallel prog data format. odho ParaView 0 September 20, 2016 07:01
Bug in layerAdditionRemoval change to OF15 mesh format andersking OpenFOAM Bugs 2 August 6, 2008 04:17
Fluent/UNS I/O format Francisco Canabal FLUENT 2 February 3, 2000 10:24


All times are GMT -4. The time now is 23:00.