CFD Online Discussion Forums - Running OpenFOAM in parallel

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM (https://www.cfd-online.com/Forums/openfoam/)

- - Running OpenFOAM in parallel (https://www.cfd-online.com/Forums/openfoam/126216-running-openfoam-parallel.html)

Running OpenFOAM in parallel

Dear Foamers,

I am dealing with a pretty-big mesh and I am having some troubles.

I am running it on CINECA, a super-computer center and my script is as follow:

Code:

#!/bin/sh



blockMesh

decomposePar

mpirun -np 16 snappyHexMesh

reconstructParMesh -latestTime -mergeTol 1E-06 -noZero



rm -r processor*

cp constant/polyMesh/blockMeshDict .

rm -r constant/polyMesh

cp -r 1/polyMesh constant

cp blockMeshDict constant/polyMesh

rm -r 1

rm blockMeshDict



decomposePar

mpirun -np 16 topoSet -dict ./system/topoSetDict

mpirun -np 16 refineMesh -dict ./system/refineMeshDict1

reconstructParMesh -latestTime -mergeTol 1E-06 -noZero



rm -r processor*

cp constant/polyMesh/blockMeshDict .

rm -r constant/polyMesh

cp -r 1/polyMesh constant

cp blockMeshDict constant/polyMesh

rm -r 1

rm blockMeshDict



decomposePar

mpirun -np 16 simpleFoam

recontructPar

Once I run it, I get the error in the file attached.

As far as the steps are concerned, I am pretty sure they're ok, since if I run them on my laptop (with a smaller mesh, of course) everything is fine.
What is this errore linked to, according to you?

Thanks a lot,
Samuele

Dear Samuele,

the first striking thing I saw is that you miss some
parallel statements (see bold code):

Code:

mpirun -np 16 snappyHexMesh -parallel

and this statement you have to use for all mpiruns otherwise
you will start up 16 different processes which will cause some troubles.

Let me know whether it worked
regards
Colin

Thanks a lot, Colin.

Let me try.

I'll get back to you asap.

Samuele

Hi Colin,

I've added the option -parallel, but I am always getting the same error.

Any other hints?

Thanks,
Samuele

Hi Samuele

well the last line says:

Code:

=>> PBS: job killed: mem 31250244kb exceeded limit 12582912kb

which means that the memory is exceeded.
I thought this came by starting 16 times the same thread and making 16
times the same mesh, but apparently not.

So my question is: How big is your mesh and how much RAM do you have
available?
As a guideline for me I'm using 800k cells per 1GB RAM

Another question I have: why is blockMesh started up several times
while you specified blockMesh to start up as a single thread and not
as a mpirun?
Is this because you are using CINECA?

regards
Colin