Greetings Arun,

I've had a test case with the cavity tutorial, expanded from 2D to 3D, which did not scale well with an AMD 1055T processor with 6 cores, but with 16 sub-domains it was still faster than with 6. Nonetheless, it was possibly a fluke, in the sense that the over-scheduling lead to a slightly better usage of memory and cache.

I've been keeping a blog post with notes on this subject: Notes about running OpenFOAM in parallel
In it you should find this note:
From that post #20 you should be able to learn a bit more about this subject of using a single machine with multiple parallel processes!

Best regards,
Bruno

Strange behaviour
 

Hi all,

I set up a case in OpenFoam and I tried to run it paralell. The simulation works quite fine with two cores. Than we tried to increase the number of cores and an error occures:
run/VAWT$ [6] #0 Foam::error::printStack(Foam::Ostream&) at ??:?
[6] #4 Foam::PBiCG::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
[6] #5 Foam::fvMatrix<double>::solveSegregated(Foam::dict ionary const&) at ??:?
[6] #6 Foam::fvMatrix<double>::solve(Foam::dictionary const&) at ??:?
[6] #7 Foam::SolverPerformance<double> Foam::solve<double>(Foam::tmp<Foam::fvMatrix<doubl e> > const&) at ??:?
[6] #8 Foam::incompressible::RASModels::kOmegaSST::correc t() at ??:?
[6] #9 ? at ??:?
[6] #10 __libc_start_main in "/lib/x86_64-linux-gnu/libc.so.6"
[6] #11 ? at ??:?
[openfoamtest:58577] *** Process received signal ***
[openfoamtest:58577] Signal: Floating point exception (8)
[openfoamtest:58577] Signal code: (-6)
[openfoamtest:58577] Failing at address: 0x3e80000e4d1


There were no errors while I decompose the mesh (neither for 2 nor for 16 cores) and the error also occures randomly (somtimes at timestep 1.82, sometimes later). It is also possible to reconstruct the case after the error occurs and restart it. Here my decomposedParDict:
FoamFile
{
version 2.0;
format ascii;
class dictionary;
object decomposeParDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 16;

method scotch;

Please can someone tell me why this case works on two cores but not on 16 cores? And why it is possible to restart the case?

Quick answer: There are a various possibilities, but since you didn't give any details about your case nor the version of OpenFOAM you're using, I'll list the possibilities that come to my head... then again, I've written about this in the past, let me see if I can find it... there you go: http://www.cfd-online.com/Forums/ope...tml#post324934 - post #28