[shaoX]

Hello Foamers,

We know that the collated I/O option in OpenFOAM can significantly reduce the total files number when we do parallel running. But recently I found this option can be much slower than uncollated I/O. I'm using v1912.

This happens both on HPC or my workstation and I've checked all the settings (eg. OptimisationSwitches) according to https://www.openfoam.com/releases/op...2/parallel.php

Here's a simple test:
Test#1
$TUTORIAL/cavity, np=10, 40,000 cells; 0-0.2s, ∆t=1e-5, 20000 steps in total; use icoFoam.
ClockTime: collated: 110s; uncollated: 61s.

We can see there's a big difference in clockTime.

It looks like this is an input-output problem. But the following test designed to stress the I/O workload shows that I/O is not the problem.

/************************************************** *******
Lid-driven cavity flow, 6M (200*200*200) cells, ∆t=1e-4, run from 0-0.01s, totally only 100 steps, np=128.
Test#2:
Write interval: every 20 steps
ClockTime(collated): 518s
ColockTime(uncollated):511s
Test#3:
Write interval: every 2 steps
ClockTime(collated): 559s
ColockTime(uncollated):526s
************************************************** ********/

We can see based on this test the difference between collated and uncollated is negligible even Test#3 writes so frequently and writes ~60GB data in total.

The difference only becomes significant when the total running steps are high and the simulation is not big size.

I did profiling to find where computational time is spent and found ++runTime, or Foam::Time:: operator++() can be very time-consuming::
Test#4
cavity, np=10, 40,000 cells; 0-0.1s, ∆t=1e-5, 10000 steps
ClockTime=1552s; ++runTime cost: 24%
this percentage can go up to 45% if I run from 0 to 0.2s, or 20000 steps.

I need to stress this difference can be small if you are running a big simulation without too many steps. But in my situation, I need to put ++runTime into a subcycling in which fast scale equations are solved with smaller ∆t while other equations are waiting.

So the total times of executing ++runTime can be very high (millions of steps) and I find this slow-down is a serious problem.

Can someone give me any suggestion on this, or point out that I just made some stupid mistakes? Thank you very much!