OpenFOAM 2.2.1 on OpenSUSE 10.3 on Cluster?

CFDnewbie147 · July 26, 2013, 05:08

Hello together,

I'm trying to set up OpenFOAM 2.2.1 on a openSUSE 10.3 Cluster(x86 -64) to do some cfd-work for my study.

I have downloaded the Source Packs.tgz for OpenFOAM 2.2.1 and ThirdParty 2.2.1 and have, as descriped in the OF Source Pack Installation Guide, unpacked them in my $HOME/OpenFOAM directory.

Because it's running in tcsh I've added " alias opf 'source $HOME/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc' " to my .tcsh- file.

Then i want to use the Intel compiler(13.1.2), so I've changed the following in my cshrc- file in OpenFoam/etc:

#- Compiler:
# WM_COMPILER = Gcc | Gcc43 | Gcc44 | Gcc45 | Gcc46 | Clang | Icc (Intel icc)
setenv WM_COMPILER icc
setenv WM_COMPILER_ARCH # defined but empty
unsetenv WM_COMPILER_LIB_ARCH

I thought this shuld be all and i did ./Allwmake in my OpenFOAM directory. But after compiling in the "xterm window" I do the foamInstallationTest it looks like the picture "fehler"...

What's going wrong or what actions have i forgotten?
Please help me!!!

Thank u for ur replies...

wyldckat · August 21, 2013, 13:30

Greetings CFDnewbie147 and welcome to the forum!

Run Allwmake like this:

Code:

./Allwmake > make.log 2>&1

It will log the build process into the file "make.log".
Then look for the first occurrences of "Error ", that is if your system is in English; don't forget to not use the quotes in the search, but make sure you do use the space after the word 'error'.

If you can't figure out what the problem is, search and replace any sensitive information on that "make.log" file. Then compress it and attach the compressed file to your next post.

Best regards,
Bruno

CFDnewbie147 · August 22, 2013, 03:54

Thank you for your answer Bruno.

I've given the job to my supervisior, he's also the admin of the cluster. We had to change alle the make files and had to update a lot programms like gcc, qt and so on...

Now it's running on the frontends of the cluster and now it's a bit tricki, to run it via PBS (Portable batch system) on the processors of each node. Have u any suggestions for this problem?

Best regards and thank u for your help!

wyldckat · August 22, 2013, 05:33

Hi CFDnewbie147,

The problem with "PBS" is that there are several variants of PBS and each one has it's own way of setting up the number of cores and machines to be used.

The best you can do is:

Check with your systems administrator what is the conventional way to launch applications on your cluster, using the PBS you've got, in order to define the number of cores and machines to be assigned to each job.
Then search here on the forum and online for people that are using a similar PBS job nomenclature.
Among those, see if any of them are using OpenFOAM.

Best regards,
Bruno

CFDnewbie147 · August 26, 2013, 07:45

Hello,

I looked for the version of PBS we are using and it's TORQUE, i hope u know it.

The preprocessing is done on the frontend of the cluster and not on the nodes: the blockMesh and decomposePar that it should run on 8 processors, this means, that only one node is running with it's 8 processors on it.

I'm trying to parallize the cavity tutorial:

Here's my qsk.proc file which i push in the queu with qsub qsk.proc after doing the preprocessing:

Code:

#!/bin/sh
###############################################################
##
##  script qsub.opf for starting OpenFOAM within pbs-System on server...
##  
##
##  Author: ##  Date:   19.08.13
##
##  in this script: PBS-settings are done in script
##  start job with 'qsub qsk.opf'
##
##- Basic PBS-options are: ----------------
##  -l: ressources (ppn: procs/nodes; ....)
##  -q: queue to be used:
##  -N: for name of pbs-job:
##  -M,-m: for Email-notification:
##  -oe: log-files    (needs rcp/rsh?)
##  -S: is used due to a bug in the 'normal' shells
##
#######PBS-Definitions must'!preceed all other stuff!:
#
#PBS -N OpenFOAM_test
#PBS -l nodes=1:ppn=8
#PBS -q batch1
#PBS -S /bin/sh
#
#-----general user definitions:
#
#source OpenFOAM
#. /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc
#
PROG="Processing"
#
####################### end of user defs. #####
#
## look for no. of processors:
PROCS=`wc -l < $PBS_NODEFILE`
#
##### set intel-libs (needed if qsub -S... is executed)
##if [ -z "${LD_LIBRARY_PATH}" ]
##then
##  LD_LIBRARY_PATH="/opt/intel/cc_80/lib"
##  export LD_LIBRARY_PATH
##else
##  LD_LIBRARY_PATH="/opt/intel/cc_80/lib:$LD_LIBRARY_PATH"
##  export LD_LIBRARY_PATH
##fi
#
##if [ -z "${LD_LIBRARY_PATH}" ]
##then
##  LD_LIBRARY_PATH="/opt/intel/fc_80/lib"
##  export LD_LIBRARY_PATH
##else
##  LD_LIBRARY_PATH="/opt/intel/fc_80/lib:$LD_LIBRARY_PATH"
##  export LD_LIBRARY_PATH
##fi
#
##### set intel-libs (needed if qsub -S... is executed)
echo initial LD_LIBRARY_PATH was $LD_LIBRARY_PATH
## if gcc: comment it out
LD_LIBRARY_PATH="/opt/intel/composerxe/lib/intel64"
export LD_LIBRARY_PAT
#
##### set actual mpi-path:
PATH=/opt/openmpi-intel/bin:$PATH
export PATH
MPI_HOME=/opt/openmpi-intel
export MPI_HOME 
## for openmpi:
LD_LIBRARY_PATH="/opt/openmpi-intel/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH
#
#### run mpi 
cd $PBS_O_WORKDIR
echo "WorkingDirectory is: " $PBS_O_WORKDIR
echo "running:" `which mpirun` " with lib: " $LD_LIBRARY_PATH 
echo "Job is running on:" 
cat $PBS_NODEFILE 
## env > env-vars
#
##mpirun -nolocal -np $PROCS -machinefile $PBS_NODEFILE $PROG
##Note: 1) if complete Path is given for mpirun, MPI_HOME nad LD_LIBRARAY_PATH is not used!
##      2) don't use the machinefile-def., openmpi and Torque arrange this topic by intern comm.
/opt/openmpi-intel/bin/mpirun -np $PROCS $PROG xx

This file starts the Processing file which looks like:

Code:

#!/bin/sh
cd ${0%/*} || exit 1    # run from this directory
# Source tutorial run functions
. /opt/OpenFOAM/OpenFOAM-2.2.1/bin/tools/RunFunctions
# run icoFoam on front-nd
runParallel icoFoam 8

Then i get following logfiles:
log.icoFoam:

Code:

--------------------------------------------------------------------------
Failed to find the following executable:
Host:       magnus24
Executable: icoFoam
Cannot continue.
--------------------------------------------------------------------------

log OpenFOAM_test.e12936:

Code:

[magnus24:31966] [0,0,2] ORTE_ERROR_LOG: Not found in file odls_default_module.c at line 1191
[magnus24:31966] [0,0,2] ORTE_ERROR_LOG: Not found in file orted.c at line 626

log OpenFOAM_test.o12936:

Code:

initial LD_LIBRARY_PATH was
WorkingDirectory is:  /.../.../.../.../OpenFOAM/cavity_parallel_knoten
running: /opt/openmpi-intel/bin/mpirun  with lib:  /opt/openmpi-intel/lib:/opt/intel/composerxe/lib/intel64
Job is running on:
magnus24.....
magnus24.....
magnus24.....
magnus24....
magnus24....
magnus24....
magnus24....
magnus24....
Running icoFoam in parallel on /.../.../.../.../OpenFOAM/cavity_parallel_knoten using 8 processes
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run
icoFoam already run on /.../.../.../.../OpenFOAM/cavity_parallel_knoten: remove log file to re-run

It looks like, that each processor on the node "magnus24" tries to do the icoFoam completely and not only for the decomposed part. How can i fix this, that every processor only does the word for one part of the whole simulation???

Please help me!
Thank u for your replies!

wyldckat · August 26, 2013, 17:39

Hi CFDnewbie147,

If you are using PBS and by looking at the way your script is made, you should not use the "Processing" script file like that.

If you Google for:

Code:

OpenFOAM PBS

you'll see that people call the solver directly. In your case, the last or near last line in "qsk.proc" should be like this:

Code:

/opt/openmpi-intel/bin/mpirun -np $PROCS icoFoam -parallel

This is because the runParallel function being used inside the "Processing" script, which also calls mpirun, leading to an attempt of launching 64 icoFoam processes, although only the first 8 (might) work.

Best regards,
Bruno

CFDnewbie147 · August 27, 2013, 02:58

Hello Bruno,

thanks for your answer.
I added to my qsk.proc:

Code:

source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc

and changed the last line to:

Code:

/opt/openmpi-intel/bin/mpirun -np $PROCS icoFoam -parallel xx

Now i don't start the Processing- script, but run the code icoFoam directly, as mentioned above.

But now i get the following error message in my OpenFOAM_test.e12942- file:

Code:

/opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc: line 35: setenv: command not found
/opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc: line 36: setenv: command not found
/opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc: line 61: syntax error near unexpected token `setenv'
/opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc: line 61: `if ( ! $?FOAM_INST_DIR ) setenv FOAM_INST_DIR $foamInstall'
[magnus16:16418] [0,0,1] ORTE_ERROR_LOG: Not found in file odls_default_module.c at line 1191
--------------------------------------------------------------------------
Failed to find the following executable:
Host:       magnus16
Executable: icoFoam
Cannot continue.
--------------------------------------------------------------------------
[magnus16:16418] [0,0,1] ORTE_ERROR_LOG: Not found in file orted.c at line 626

:/

What's wrong again?
I hope u can help me again...Thank You very much!

wyldckat · August 27, 2013, 17:25

Quick answer - change:

Code:

source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc

To:

Code:

source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/bashrc

CFDnewbie147 · August 28, 2013, 01:39

Sorry, but doesn't work...

Then i get the following error message in my OpenFOAM_test.e12980:

Code:

[magnus16:14346] mca: base: component_find: unable to open paffinity hwloc: file not found (ignored)
[magnus16:14346] mca: base: component_find: unable to open errmgr default: file not found (ignored)
[magnus16:14346] mca: base: component_find: unable to open rml oob: file not found (ignored)
[magnus16:14346] [NO-NAME] ORTE_ERROR_LOG: Not found in file runtime/orte_init_stage1.c at line 182
[magnus16:14346] *** Process received signal ***
[magnus16:14346] Signal: Segmentation fault (11)
[magnus16:14346] Signal code: Address not mapped (1)
[magnus16:14346] Failing at address: 0xfffffff3
[magnus16:14346] [ 0] /lib64/libpthread.so.0 [0x2b6e5ba7bfb0]
[magnus16:14346] [ 1] /lib64/libc.so.6(strlen+0x20) [0x2b6e5bcfebb0]
[magnus16:14346] [ 2] /lib64/libc.so.6(_IO_vfprintf+0x37bd) [0x2b6e5bcce84d]
[magnus16:14346] [ 3] /lib64/libc.so.6 [0x2b6e5bccf8b3]
[magnus16:14346] [ 4] /lib64/libc.so.6(_IO_vfprintf+0x9e) [0x2b6e5bccb12e]
[magnus16:14346] [ 5] /opt/openmpi-intel/lib/libopen-pal.so.0(opal_show_help+0x485) [0x2b6e5ad85455]
[magnus16:14346] [ 6] /opt/openmpi-intel/lib/libopen-rte.so.0(orte_init_stage1+0x10b) [0x2b6e5aaf2b03]
[magnus16:14346] [ 7] /opt/openmpi-intel/lib/libopen-rte.so.0(orte_system_init+0x7) [0x2b6e5aaf672b]
[magnus16:14346] [ 8] /opt/openmpi-intel/lib/libopen-rte.so.0(orte_init+0x4f) [0x2b6e5aaf2843]
[magnus16:14346] [ 9] /opt/openmpi-intel/bin/mpirun(orterun+0x1ab) [0x4042e5]
[magnus16:14346] [10] /opt/openmpi-intel/bin/mpirun(main+0x20) [0x403d88]
[magnus16:14346] [11] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b6e5bca6b54]
[magnus16:14346] [12] /opt/openmpi-intel/bin/mpirun [0x403cd9]
[magnus16:14346] *** End of error message ***
/var/spool/torque/mom_priv/jobs/12980.karma.SC: line 70: 14346 Segmentation fault      /opt/openmpi-intel/bin/mpirun -np $PROCS icoFoam -parallel

My OpenFOAM_test.o12980 says:

Code:

Run started at
Wed Aug 28 07:30:13 CEST 2013
initial LD_LIBRARY_PATH was /opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/gperftools-svn/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/paraview-3.12.0/lib/paraview-3.12:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib/openmpi-1.6.3:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48DPOpt/lib/openmpi-1.6.3:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/lib64:/opt/intel/composerxe/lib/intel64:/opt/Python-2.7.2-infini-intel/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/qt-4.8.5/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/flex-2.5.37/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/mpc-1.0.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/mpfr-3.1.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gmp-5.1.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gcc-4.8.1/lib64:/home/zachjoer/OpenFOAM/zachjoer-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/site/2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib/dummy
WorkingDirectory is:  /z/pro/cfdtmp03/zachjoer/OpenFOAM/cavity_parallel_knoten
running: /opt/openmpi-intel/bin/mpirun  with lib:  /opt/openmpi-intel/lib:/opt/intel/composerxe/lib/intel64
Job is running on:
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
magnus16.cfd.lfk.eads.net
Run completed at 
Wed Aug 28 07:30:13 CEST 2013

I don't know what's wrong, because if i make

Code:

echo $shell

i get

Code:

/bin/tcsh

So I've normally to use the cshrc of OpenFOAM i think...
My administrator told me, that there is on each node OpenFOAM or the link... so why does it not work???

wyldckat · August 31, 2013, 15:10

Hi CFDnewbie147,

I didn't manage to look with much attention into this the other day and only now did I managed to look into this...

Basically the problem is that in the "qsk.proc" file, you have right at the start this line:

Code:

#!/bin/sh

which means that it will use either dash or bash (or similar) by default, which is why you were getting the error about "setenv" not existing.
If you change that line to:

Code:

#!/bin/csh

and use the original sourcing command you had:

Code:

source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc

then it should work!

Best regards,
Bruno

CFDnewbie147 · September 16, 2013, 09:23

Hello Bruno,

thanks for your answer but it doesn't work...I get the following error output:

Code:

PROCS=8: Command not found.
LD_LIBRARY_PATH=/opt/intel/composerxe/lib/intel64: Command not found.
export: Command not found.
PATH=/opt/openmpi-intel/bin:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/paraview-3.12.0/bin:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/bin:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/flex-2.5.37/bin:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gcc-4.8.1/bin:/home/zachjoer/OpenFOAM/zachjoer-2.2.1/platforms/linux64Gcc48DPOpt/bin:/opt/OpenFOAM/site/2.2.1/platforms/linux64Gcc48DPOpt/bin:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/bin:/opt/OpenFOAM/OpenFOAM-2.2.1/bin:/opt/OpenFOAM/OpenFOAM-2.2.1/wmake:/usr/local/bin/:/home/zachjoer/util:/opt/intel/composer_xe_2013.4.183/bin/intel64:/opt/intel/composer_xe_2013.4.183/mpirt/bin/intel64:/opt/intel/composer_xe_2013.4.183/bin/intel64_mic:/opt/intel/composer_xe_2013.4.183/debugger/gui/intel64:/opt/torque/bin:/opt/netcdf-gcc/bin:/opt/openmpi-gcc/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib64/jvm/jre/bin:/opt/ofed/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/ofed/sbin:/usr/lib/qt3/bin: Command not found.
export: Command not found.
MPI_HOME=/opt/openmpi-intel: Command not found.
export: Command not found.
LD_LIBRARY_PATH=/opt/openmpi-intel/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/paraview-3.12.0/lib/paraview-3.12:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib/openmpi-1.6.3:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48DPOpt/lib/openmpi-1.6.3:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/openmpi-1.6.3/lib64:/opt/intel/composerxe/lib/intel64:/opt/Python-2.7.2-infini-intel/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48/qt-4.8.5/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/flex-2.5.37/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/mpc-1.0.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/mpfr-3.1.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gmp-5.1.2/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64/gcc-4.8.1/lib64:/home/zachjoer/OpenFOAM/zachjoer-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/site/2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/ThirdParty-2.2.1/platforms/linux64Gcc48DPOpt/lib:/opt/OpenFOAM/OpenFOAM-2.2.1/platforms/linux64Gcc48DPOpt/lib/dummy:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.4.183/compiler/lib/intel64:/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64:/opt/intel/composer_xe_2013.4.183/mpirt/lib/intel64:/opt/intel/composer_xe_2013.4.183/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.4.183/ipp/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.4.183/tbb/lib/intel64/gcc4.1:/opt/openmpi-gcc/lib:/opt/netcdf-gcc/lib: Command not found.
export: Command not found.
PROCS: Undefined variable.

What's wrong again?
Best regards :-)

wyldckat · September 17, 2013, 18:21

Hi CFDnewbie147,

OK, so from what I re-read in an attempt to remember what was discussed here, what I can see is that when you followed my instruction from post #8, you got closer to the solution.
But then the other idea from #10 ended up in a worse situation.

Now that I reviewed the output from your post #9, there is one clear indication that one very important detail has been ignored: you built OpenFOAM with Open-MPI 1.6.3 that is provided in OpenFOAM's ThirdParty folder, when you should in fact have indicated to use the system's MPI.
In other words, the variable "WM_MPLIB" should be set to "SYSTEMOPENMPI" and not "OPENMPI". This is defined inside OpenFOAM's "etc/bashrc" file.

You need to change the setting for that variable in said file, start a new terminal and run Allwmake once again. It will re-build only the libraries that depend on the new MPI.

Best regards,
Bruno

CFDnewbie147 · September 23, 2013, 13:04

Hello Bruno,

sorry for this late answer. I solved the problem. As u said, i used the system open mpi, but i have to use the mpi of OF.

So i deleted the lines which get the system open mpi. In my script, there's only the source OF-cshrc and the last line, run mpi..... and now it works!

Thank you for your help!!!
Best regards
CFDNewbie147

August 21, 2013, 13:30		#2
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,975 Blog Entries: 45 Rep Power: 128	Greetings CFDnewbie147 and welcome to the forum! Run Allwmake like this: Code: ./Allwmake > make.log 2>&1 It will log the build process into the file "make.log". Then look for the first occurrences of "Error ", that is if your system is in English; don't forget to not use the quotes in the search, but make sure you do use the space after the word 'error'. If you can't figure out what the problem is, search and replace any sensitive information on that "make.log" file. Then compress it and attach the compressed file to your next post. Best regards, Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

August 22, 2013, 05:33		#4
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,975 Blog Entries: 45 Rep Power: 128	Hi CFDnewbie147, The problem with "PBS" is that there are several variants of PBS and each one has it's own way of setting up the number of cores and machines to be used. The best you can do is: Check with your systems administrator what is the conventional way to launch applications on your cluster, using the PBS you've got, in order to define the number of cores and machines to be assigned to each job. Then search here on the forum and online for people that are using a similar PBS job nomenclature. Among those, see if any of them are using OpenFOAM. Best regards, Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

August 26, 2013, 17:39		#6
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,975 Blog Entries: 45 Rep Power: 128	Hi CFDnewbie147, If you are using PBS and by looking at the way your script is made, you should not use the "Processing" script file like that. If you Google for: Code: OpenFOAM PBS you'll see that people call the solver directly. In your case, the last or near last line in "qsk.proc" should be like this: Code: /opt/openmpi-intel/bin/mpirun -np $PROCS icoFoam -parallel This is because the runParallel function being used inside the "Processing" script, which also calls mpirun, leading to an attempt of launching 64 icoFoam processes, although only the first 8 (might) work. Best regards, Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

September 17, 2013, 18:21		#12
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,975 Blog Entries: 45 Rep Power: 128	Hi CFDnewbie147, OK, so from what I re-read in an attempt to remember what was discussed here, what I can see is that when you followed my instruction from post #8, you got closer to the solution. But then the other idea from #10 ended up in a worse situation. Now that I reviewed the output from your post #9, there is one clear indication that one very important detail has been ignored: you built OpenFOAM with Open-MPI 1.6.3 that is provided in OpenFOAM's ThirdParty folder, when you should in fact have indicated to use the system's MPI. In other words, the variable "WM_MPLIB" should be set to "SYSTEMOPENMPI" and not "OPENMPI". This is defined inside OpenFOAM's "etc/bashrc" file. You need to change the setting for that variable in said file, start a new terminal and run Allwmake once again. It will re-build only the libraries that depend on the new MPI. Best regards, Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

September 23, 2013, 13:04		#13
CFDnewbie147 Member Join Date: Jul 2013 Posts: 62 Rep Power: 12	Hello Bruno, sorry for this late answer. I solved the problem. As u said, i used the system open mpi, but i have to use the mpi of OF. So i deleted the lines which get the system open mpi. In my script, there's only the source OF-cshrc and the last line, run mpi..... and now it works! Thank you for your help!!! Best regards CFDNewbie147 wyldckat likes this.

August 22, 2013, 03:54		#3
CFDnewbie147 Member Join Date: Jul 2013 Posts: 62 Rep Power: 12	Thank you for your answer Bruno. I've given the job to my supervisior, he's also the admin of the cluster. We had to change alle the make files and had to update a lot programms like gcc, qt and so on... Now it's running on the frontends of the cluster and now it's a bit tricki, to run it via PBS (Portable batch system) on the processors of each node. Have u any suggestions for this problem? Best regards and thank u for your help!

August 27, 2013, 17:25		#8
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,975 Blog Entries: 45 Rep Power: 128	Quick answer - change: Code: source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc To: Code: source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/bashrc __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

August 31, 2013, 15:10		#10
wyldckat Retired Super Moderator Bruno Santos Join Date: Mar 2009 Location: Lisbon, Portugal Posts: 10,975 Blog Entries: 45 Rep Power: 128	Hi CFDnewbie147, I didn't manage to look with much attention into this the other day and only now did I managed to look into this... Basically the problem is that in the "qsk.proc" file, you have right at the start this line: Code: #!/bin/sh which means that it will use either dash or bash (or similar) by default, which is why you were getting the error about "setenv" not existing. If you change that line to: Code: #!/bin/csh and use the original sourcing command you had: Code: source /opt/OpenFOAM/OpenFOAM-2.2.1/etc/cshrc then it should work! Best regards, Bruno __________________ OpenFOAM: FAQ \| Getting started Forum: How to get help, to post code/output and forum guide Read this before sending me PM

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
OpenSuse 12.3 x64 with OpenFoam 2.2.0	aylalisa	OpenFOAM Installation	2	May 5, 2013 11:29
OpenFOAM 1.7 - openSUSE 11.3 - gcc 4.5.0	alberto	OpenFOAM	12	July 28, 2010 11:59
Modified OpenFOAM Forum Structure and New Mailing-List	pete	Site News & Announcements	0	June 29, 2009 05:56
OpenFOAM 1.5-dev installation in OpenSuSe 10.3	darenyang	OpenFOAM Installation	0	April 28, 2009 04:30
OpenFOAM 15 with ParaView 340 on OpenSuSE 110	zairja	OpenFOAM Installation	7	February 10, 2009 08:31