CFD Online Discussion Forums - OpenFoam with PBS job manager

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM Running, Solving & CFD (https://www.cfd-online.com/Forums/openfoam-solving/)

- - OpenFoam with PBS job manager (https://www.cfd-online.com/Forums/openfoam-solving/168533-openfoam-pbs-job-manager.html)

OpenFoam with PBS job manager

Is there any official tutorial or example regarding OpenFoam and PBS job scheduler (the qsub command)? I saw some pages on the web but it seems that they are trying to show how to solve a problem first.

I actually don't care about that since my job is to install (have already done) and check it users can submit their jobs to the cluster.

So, I need an example which shows how to submit a solved problem (meshing, ...) to the job scheduler.

Basically, I write a pbs script and put "ls" command in it and submit it to the scheduler to see if all nodes are working and communicating. So, I am searching for a similar simple command with respect to OpenFoam.

Any idea for that?

Hi mahmoodn,

You can use blockMesh and snappyHexMesh provided by OpenFOAM for meshing.

For testing PBS job scheduler, you can simply pick up a tutorial case from $FOAM_TUTORIALS directory and test with that case.

The $FOAM_TUTORIALS/incompressible/icoFoam/cavity is a good start. You need to copy it to your working directory, and run blockMesh inside the case directory. After meshing is done, run icoFoam to begin your simulation.

P.S.: don't forget to source OpenFOAM bashrc first :D

Do you have any idea about this:

Code:

mahmood@cluster:of-test$ source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc

mahmood@cluster:of-test$ blockMesh

-bash: blockMesh: command not found

mahmood@cluster:of-test$ /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64Gcc48DPOpt/bin/blockMesh

/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64Gcc48DPOpt/bin/blockMesh: error while loading shared libraries: libblockMesh.so: cannot open shared object file: No such file or directory

mahmood@cluster:of-test$

I can test with LD_LIBRARY_PATH and PATH, but just want to know if I am on the correct track.

Hi mahmoodn,

Have you modified the bashrc from OpenFOAM? What's the value of your $FOAM_INST_DIR and $WM_PROJECT_INST_DIR?

Yes, I modified it during the installation. Lines associated with that is

Code:

export WM_PROJECT=OpenFOAM

export WM_PROJECT_VERSION=2.3.0

foamInstall=/export/apps/mechanics/$WM_PROJECT

Therefore, I see

Code:

$ ls /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/

Allwmake      bin      doc  platforms    README.org  tutorials

applications  COPYING  etc  README.html  src         wmake

The values you asked are

Code:

$ echo $FOAM_INST_DIR

/export/apps/mechanics/OpenFOAM

$ echo $WM_PROJECT_INST_DIR

/export/apps/mechanics/OpenFOAM

Hmm. Seems the shell variables aren't exported correctly.

What's the output of export | grep -E '(FOAM_|WM_)'? What's the value of your $PATH and $LD_LIBRARY_PATH?

Check it

Code:

$ export | grep -E '(FOAM_|WM_)'

declare -x FOAM_APP="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/applications"

declare -x FOAM_APPBIN="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/bin"

declare -x FOAM_EXT_LIBBIN="/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64GccDPOpt/lib"

declare -x FOAM_INST_DIR="/export/apps/mechanics/OpenFOAM"

declare -x FOAM_JOB_DIR="/export/apps/mechanics/OpenFOAM/jobControl"

declare -x FOAM_LIBBIN="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib"

declare -x FOAM_MPI="openmpi-1.6.5"

declare -x FOAM_RUN="/home/mahmood/OpenFOAM/mahmood-2.3.0/run"

declare -x FOAM_SETTINGS=""

declare -x FOAM_SIGFPE=""

declare -x FOAM_SITE_APPBIN="/export/apps/mechanics/OpenFOAM/site/2.3.0/platforms/linux64GccDPOpt/bin"

declare -x FOAM_SITE_LIBBIN="/export/apps/mechanics/OpenFOAM/site/2.3.0/platforms/linux64GccDPOpt/lib"

declare -x FOAM_SOLVERS="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/applications/solvers"

declare -x FOAM_SRC="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/src"

declare -x FOAM_TUTORIALS="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/tutorials"

declare -x FOAM_USER_APPBIN="/home/mahmood/OpenFOAM/mahmood-2.3.0/platforms/linux64GccDPOpt/bin"

declare -x FOAM_USER_LIBBIN="/home/mahmood/OpenFOAM/mahmood-2.3.0/platforms/linux64GccDPOpt/lib"

declare -x FOAM_UTILITIES="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/applications/utilities"

declare -x WM_ARCH="linux64"

declare -x WM_ARCH_OPTION="64"

declare -x WM_CC="gcc"

declare -x WM_CFLAGS="-m64 -fPIC"

declare -x WM_COMPILER="Gcc"

declare -x WM_COMPILER_LIB_ARCH="64"

declare -x WM_COMPILE_OPTION="Opt"

declare -x WM_CXX="g++"

declare -x WM_CXXFLAGS="-m64 -fPIC"

declare -x WM_DIR="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/wmake"

declare -x WM_LDFLAGS="-m64"

declare -x WM_LINK_LANGUAGE="c++"

declare -x WM_MPLIB="OPENMPI"

declare -x WM_OPTIONS="linux64GccDPOpt"

declare -x WM_OSTYPE="POSIX"

declare -x WM_PRECISION_OPTION="DP"

declare -x WM_PROJECT="OpenFOAM"

declare -x WM_PROJECT_DIR="/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0"

declare -x WM_PROJECT_INST_DIR="/export/apps/mechanics/OpenFOAM"

declare -x WM_PROJECT_USER_DIR="/home/mahmood/OpenFOAM/mahmood-2.3.0"

declare -x WM_PROJECT_VERSION="2.3.0"

declare -x WM_THIRD_PARTY_DIR="/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0"







$ echo $PATH

/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/gperftools-svn/bin:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/ParaView-4.1.0/bin:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/openmpi-1.6.5/bin:/home/mahmood/OpenFOAM/mahmood-2.3.0/platforms/linux64GccDPOpt/bin:/export/apps/mechanics/OpenFOAM/site/2.3.0/platforms/linux64GccDPOpt/bin:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/bin:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/bin:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/wmake:/opt/openmpi/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bio/ncbi/bin:/opt/bio/mpiblast/bin:/opt/bio/EMBOSS/bin:/opt/bio/clustalw/bin:/opt/bio/tcoffee/bin:/opt/bio/hmmer/bin:/opt/bio/phylip/exe:/opt/bio/mrbayes:/opt/bio/fasta:/opt/bio/glimmer/bin:/opt/bio/glimmer/scripts:/opt/bio/gromacs/bin:/opt/bio/gmap/bin:/opt/bio/tigr/bin:/opt/bio/autodocksuite/bin:/opt/bio/wgs/bin:/opt/eclipse:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/maven/bin:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbin:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbin:/opt/condor/bin:/opt/condor/sbin:/opt/gridengine/bin/linux-x64:/export/apps/mathematics/matlab2012a/bin:/home/mahmood/bin







$ echo $LD_LIBRARY_PATH

/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/gperftools-svn/lib:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/ParaView-4.1.0/lib/paraview-4.1:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/openmpi-1.6.5:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64GccDPOpt/lib/openmpi-1.6.5:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/openmpi-1.6.5/lib:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64Gcc/openmpi-1.6.5/lib64:/home/mahmood/OpenFOAM/mahmood-2.3.0/platforms/linux64GccDPOpt/lib:/export/apps/mechanics/OpenFOAM/site/2.3.0/platforms/linux64GccDPOpt/lib:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib:/export/apps/mechanics/OpenFOAM/ThirdParty-2.3.0/platforms/linux64GccDPOpt/lib:/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64GccDPOpt/lib/dummy:/opt/gridengine/lib/linux-x64:/opt/openmpi/lib

Seems your applications aren't installed to $FOAM_APPBIN.

Have you exported WM_COMPILER=Gcc48 before building OpenFOAM? This may be the issue. If so, just rebuild OpenFOAM without exporting that.

I ran it again. Please see the variables first

Code:

root@cluster:ThirdParty-2.3.0# module load openmpi-x86_64 || export PATH=$PATH:/usr/lib64/openmpi/bin

root@cluster:ThirdParty-2.3.0# source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI

root@cluster:ThirdParty-2.3.0#



root@cluster:OpenFOAM-2.3.0# echo $WM_NCOMPPROCS

4

root@cluster:OpenFOAM-2.3.0# echo $WM_COMPILER

Gcc48

root@cluster:OpenFOAM-2.3.0# echo $WM_MPLIB

SYSTEMOPENMPI

root@cluster:OpenFOAM-2.3.0# echo $FOAM_APPBIN

/export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64Gcc48DPOpt/bin

Then I ran ./Allwmake again and you can see the lengthy output at pastebin

This time, when I run blockMesh, I see its usage :)

Code:

root@cluster:OpenFOAM-2.3.0# blockMesh

/*---------------------------------------------------------------------------*\

| =========                 |                                                 |

| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |

|  \\    /   O peration     | Version:  2.3.0                                 |

|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |

|    \\/     M anipulation  |                                                 |

\*---------------------------------------------------------------------------*/

Build  : 2.3.0-f5222ca19ce6

Exec   : blockMesh

Date   : Mar 25 2016

Time   : 11:03:08

Host   : "cluster.scu.ac.ir"

PID    : 10972

Case   : /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0

nProcs : 1

sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).

fileModificationChecking : Monitoring run-time modified files using timeStampMaster

allowSystemOperations : Disallowing user-supplied system call operations



// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

Create time







--> FOAM FATAL IO ERROR:

cannot find file



file: /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/system/controlDict at line 0.



    From function regIOobject::readStream()

    in file db/regIOobject/regIOobjectRead.C at line 73.



FOAM exiting

So it is working....
Question: What commands should I put in the .bashrc or /etc/profile (for all users)? These two?

Code:

root@cluster:ThirdParty-2.3.0# module load openmpi-x86_64 || export PATH=$PATH:/usr/lib64/openmpi/bin

root@cluster:ThirdParty-2.3.0# source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI

Hi mahmoodn,

Good to see it working :D

Quote:

Question: What commands should I put in the .bashrc or /etc/profile (for all users)? These two?

Well, if your users do not need to switch between various versions of OpenFOAM, you can just add this two lines to /etc/profile.d/openfoam.sh (create new one if it does not exist).

Just keep the user variables same with those when you built OpenFOAM :p.

Regards,
Weiwen

It seems that the following line has no effect in /etc/profile and ~/.bashrc

Code:

alias of230='module load openmpi-x86_64; source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI'

Instead, if I manually run

Code:

source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI

from the terminal, then I will be able to run blockMesh without full path

Quote:

Originally Posted by mahmoodn (Post 591588)

It seems that the following line has no effect in /etc/profile and ~/.bashrc

Code:

alias of230='module load openmpi-x86_64; source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI'

This line set alias for OpenFOAM, users who want to use OpenFOAM must execute of230 first.

The aliases are useful when there are multiple OpenFOAM versions as it wouldn't mess up the environment variables ;).

Regards,
Weiwen

OK I got it... So the correct thing is to first run of230 and then blockMesh.

Thanks for that. I will move on to the tutorial for PBS.

Hi again,
I ran blockMesh and then icoFoam in the $FOAM_TUTORIALS/incompressible/icoFoam/cavity. I saw some outputs without any error and it seems it is working.

Is there any multithreaded (needing more than core) example which I want to test with PBS?

Quote:

Originally Posted by mahmoodn (Post 591609)

Is there any multithreaded (needing more than core) example which I want to test with PBS?

Yes, for parallel computing, you need a more complex case, e.g, $FOAM_TUTORIALS/incompressible/pimpleDyMFoam/propeller

The following steps may be helpful.

1. Copy the propeller case to your working directory.

2. Run ./Allrun.pre to generate mesh and set initial conditions.

3. Run decomposePar to decompose the domain into 4 parts.

4. Prepare the PBS submit scripts. A simple example may look like this:

Code:

#!/bin/bash

#PBS -l nodes=1:ppn=4

#

#PBS -l walltime=00:30:00

#PBS -N myJobName

#PBS -A MyProjectName

#PBS -o outputFileName

#PBS -e errorFileName



module load openmpi-x86_64

source /export/apps/mechanics/OpenFOAM/OpenFOAM-2.3.0/etc/bashrc WM_NCOMPPROCS=4 foamCompiler=ThirdParty WM_COMPILER=Gcc48 WM_MPLIB=SYSTEMOPENMPI



mpirun -np 4 pimpleDyMFoam -parallel | tee log

5. Use qsub to submit your job.

Regards,
Weiwen

Some good news and some bad news...

Good:
I am able to run the mpi command on master node as well as the compute node. In other words, the following commands works as expected.

Code:

of230

mpirun -np 4 pimpleDyMFoam -parallel | tee log

check the output for the server node (where OpenFoam is installed) and the compute node. 4 cores are utilized.

However, when I submit the job, the output file contains the following error:

Code:

mahmood@cluster:propeller$ qsub submit.tor

49.cluster.scu

mahmood@cluster:propeller$ qstat

Job id                    Name             User            Time Use S Queue

------------------------- ---------------- --------------- -------- - -----

49.cluster                 myJobName        mahmood                0 R default

mahmood@cluster:propeller$ qstat

Job id                    Name             User            Time Use S Queue

------------------------- ---------------- --------------- -------- - -----

49.cluster                 myJobName        mahmood         00:00:00 C default

mahmood@cluster:propeller$ cat outputFileName

/*---------------------------------------------------------------------------*\

| =========                 |                                                 |

| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |

|  \\    /   O peration     | Version:  2.3.0                                 |

|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |

|    \\/     M anipulation  |                                                 |

\*---------------------------------------------------------------------------*/

Build  : 2.3.0-f5222ca19ce6

Exec   : pimpleDyMFoam -parallel

Date   : Mar 25 2016

Time   : 20:20:19

Host   : "compute-0-0.local"

PID    : 2102

[0]

[0]

[0] --> FOAM FATAL ERROR:

[0] Cannot read "/home/mahmood/system/decomposeParDict"

[0]

FOAM parallel run exiting

[0]

--------------------------------------------------------------------------

MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD

with errorcode 1.



NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.

You may or may not see output from other processes, depending on

exactly when Open MPI kills them.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun has exited due to process rank 0 with PID 2102 on

node compute-0-0.local exiting improperly. There are two reasons this could occur:



1. this process did not call "init" before exiting, but others in

the job did. This can cause a job to hang indefinitely while it waits

for all processes to call "init". By rule, if one process calls "init",

then ALL processes must call "init" prior to termination.



2. this process called "init", but exited without calling "finalize".

By rule, all processes that call "init" MUST call "finalize" prior to

exiting or it will be considered an "abnormal termination"



This may have caused other processes in the application to be

terminated by signals sent by mpirun (as reported here).

--------------------------------------------------------------------------

Something is wrong with the MPI communicator.

Regarding the error

Code:

Cannot read "/home/mahmood/system/decomposeParDict"

I have to say, the file actually is available but in a different place. As suggested by you, I copied the tutorial folder to my home. So:

Code:

mahmood@cluster:propeller$ pwd

/home/mahmood/of-test/propeller



mahmood@cluster:propeller$ ls

0         Allrun.pre     log.createPatch            log.topoSet     processor1  system

0.org     constant       log.renumberMesh           outputFileName  processor2

Allclean  log            log.snappyHexMesh          postProcessing  processor3

Allrun    log.blockMesh  log.surfaceFeatureExtract  processor0      submit.tor



mahmood@cluster:propeller$ ls system/

controlDict                        forces      readFields                         surfaces

createInletOutletSets.topoSetDict  fvSchemes   snappyHexMeshDict

createPatchDict                    fvSolution  surfaceFeatureExtractDict

decomposeParDict                   Q           surfaceFeatureExtractDictDefaults

So, system/decomposeParDict is present. I suspect that someone (PBS or OpenFoam or MPI) wrongly creates $HOME/system/decomposeParDict which is considered as /home/mahmood/system/decomposeParDict

Any idea for that?

OK. I think I found the problem. According to Basics: Set-up for different desktop machines (post #7 and after that), I found that my home has not been exported to the NFS! (have to say that I created my account on the server only).

So, let me test something and I will be back.

Hi mahmoodn,

You need to setup NIS server and client for sharing account info. Moreover, the home of compute nodes should be mounted via NFS.

Regards,
Weiwen

OK I found the problem:)

In the PBS script, we have to add cd $PBS_O_WORKDIR abd this is an important thing.

Without that and by adding pwd before the mpi command in the PBS script, I saw that the working directory was /home/mahmood which is actually $HOME and for that reason it appended system/decomposeParDict to that.

With cd $PBS_O_WORKDIR, the working directory is correctly set to the location where propeller exists.

So, everything is OK now and I am able to run a OpenFoam job. Have to tell the guys for that.
Thanks.:)