CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Installation (http://www.cfd-online.com/Forums/openfoam-installation/)
-   -   Compiling on ROCKS Cluster 5.4.3 (http://www.cfd-online.com/Forums/openfoam-installation/97530-compiling-rocks-cluster-5-4-3-a.html)

ciberbob1987 February 19, 2012 20:24

Compiling on ROCKS Cluster 5.4.3
 
Hi, I'm a student of Politecnico di Torino, and I'm working with OpenFOAM on the HPC cluster of my university.
The system is equipped with ROCKS Cluster Linux Distribution 5.4.3 OS and SGE scheduler. I'm working via SSH and my account is obviously not SU.

Actually I don't know if this is the right way, anyway I have done these steps:

Code:

[~]snft3@cclix7$ cd Marco
[Marco]snft3@cclix7$ wget http://downloads.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz?r=http%3A%2F%2Fopenfoam.org%2Fdownload%2Fsource.php&ts=1329690292&use_mirror=dfn
[1] 18979
[2] 18980
[Marco]snft3@cclix7$ --2012-02-20 00:05:08--  http://downloads.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz?r=http%3A%2F%2Fopenfoam.org%2Fdownload%2Fsource.php
Resolving downloads.sourceforge.net... 216.34.181.59
Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://garr.dl.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz [following]
--2012-02-20 00:05:08--  http://garr.dl.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz
Resolving garr.dl.sourceforge.net... 193.206.140.34, 2001:760:ffff:b0::34
Connecting to garr.dl.sourceforge.net|193.206.140.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32518358 (31M) [application/x-tar]
Saving to: `OpenFOAM-2.1.0.tgz'

100%[================================================================>] 32,518,358  7.81M/s  in 4.2s   

2012-02-20 00:05:13 (7.31 MB/s) - `OpenFOAM-2.1.0.tgz' saved [32518358/32518358]


[1]-  Done                    wget http://downloads.sourceforge.net/project/foam/foam/2.1.0/OpenFOAM-2.1.0.tgz?r=http%3A%2F%2Fopenfoam.org%2Fdownload%2Fsource.php
[2]+  Done                    ts=1329690292
[Marco]snft3@cclix7$ tar -xzf OpenFOAM-2.1.0.tgz
[Marco]snft3@cclix7$ mkdir OpenFOAM
[Marco]snft3@cclix7$ mv OpenFOAM-2.1.0 ./OpenFOAM/
[Marco]snft3@cclix7$ cd OpenFOAM
[Marco]snft3@cclix7$ export FOAM_INST_DIR=/share/apps/snft/snft3/Marco/OpenFOAM
[Marco]snft3@cclix7$ foamDotFile=$FOAM_INST_DIR/OpenFOAM-2.1.0/etc/bashrc
[Marco]snft3@cclix7$ [ -f $foamDotFile ] && . $foamDotFile



[Marco]snft3@cclix7$ cd $WM_PROJECT_DIR
[OpenFOAM-2.0.1]snft3@cclix7$ chmod +x Allwmake
[OpenFOAM-2.0.1]snft3@cclix7$ ./Allwmake | tee ../compile.log

After about an hour I accidentally aborted the compiling process.
In the log there were some errors e.g.

Code:

make[2]: Entering directory `/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/applications/solvers/heatTransfer/buoyantPimpleFoam'
/bin/sh: /share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/wmake/platforms/linux64Gcc/wmkdep: No such file or directory
make[2]: Leaving directory `/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/applications/solvers/heatTransfer/buoyantPimpleFoam'
make[2]: Entering directory `/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/applications/solvers/heatTransfer/buoyantPimpleFoam'
SOURCE=buoyantPimpleFoam.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3  -DNoRepository -ftemplate-depth-100 -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/thermophysicalModels/basic/lnInclude -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/turbulenceModels/compressible/turbulenceModel -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/finiteVolume/lnInclude -IlnInclude -I. -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OpenFOAM/lnInclude -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OSspecific/POSIX/lnInclude  -fPIC -c $SOURCE -o Make/linux64GccDPOpt/buoyantPimpleFoam.o
/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OpenFOAM/lnInclude/globalIndexAndTransformI.H: In member function 'Foam::labelList Foam::globalIndexAndTransform::transformIndicesForPatches(const Foam::labelHashSet&) const':
/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OpenFOAM/lnInclude/globalIndexAndTransformI.H:515: warning: converting to 'Foam::label' from 'double'
/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/finiteVolume/lnInclude/readTimeControls.H: In function 'int main(int, char**)':
/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/finiteVolume/lnInclude/readTimeControls.H:38: warning: unused variable 'maxDeltaT'
g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3  -DNoRepository -ftemplate-depth-100 -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/thermophysicalModels/basic/lnInclude -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/turbulenceModels/compressible/turbulenceModel -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/finiteVolume/lnInclude -IlnInclude -I. -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OpenFOAM/lnInclude -I/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/src/OSspecific/POSIX/lnInclude  -fPIC -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64GccDPOpt/buoyantPimpleFoam.o -L/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib \
            -lmeshTools -lbasicThermophysicalModels -lspecie -lcompressibleTurbulenceModel -lcompressibleRASModels -lcompressibleLESModels -lfiniteVolume -lOpenFOAM -ldl  -lm -o /share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/bin/buoyantPimpleFoam
/usr/bin/ld: cannot find -lmeshTools
collect2: ld returned 1 exit status
make[2]: *** [/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/bin/buoyantPimpleFoam] Error 1
make[2]: Leaving directory `/share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.1.0/applications/solvers/heatTransfer/buoyantPimpleFoam'
make[1]: *** [buoyantPimpleFoam] Error 2

Other ones are very similar.
I'm now trying to compile again, I'll post the log.
I want to know:
  1. what kind of errors are these?
  2. are they critical?
  3. how to solve them?

Thank you. Marco

wyldckat February 20, 2012 05:31

Greetings Marco and welcome to the forum!

AFAIK, Rocks 5.4 is based on CentOS 5.4, which means that you probably don't have decent Gcc compiler... or in other words, have a version older than Gcc 4.3.3.

In these situations where CentOS and relatives are involved, I generally point people to CentFOAM: http://sourceforge.net/apps/mediawik...itle=Main_Page

But if you want to waste more time trying to diagnose the issue:
  • Code:

    gcc -v
    will give you the version you've got installed. Remember, anything below 4.3.3 will not work out-of-the-box.
  • To really see what goes wrong:
    Code:

    ./Allwmake > make.log 2>&1
    This way the whole compilation is logged into the file "make.log".
Best regards,
Bruno

ciberbob1987 February 22, 2012 12:19

1 Attachment(s)
Hi Bruno, thank you for the reply. I have downloaded CentFOAM and made some tests, but something related to MPI don't work well.

In the user guide of the cluster (http://dauin-hpc.polito.it/?p=user-guide) is reported that the path of mpirun is /opt/openmpi/bin, but if I use it with

Code:

### Defining openmpi executable
MPIRUN=/opt/openmpi/bin/mpirun
#
#
### Set run directory
DIR_FOAM_RUN=/share/apps/snft/snft3/Marco/OpenFOAM/run
#
## Solver and case
SOLVER=snappyHexMesh
CASO=turbineSiting
#
source /share/apps/snft/snft3/Marco/OpenFOAM/OpenFOAM-2.0.x/etc/bashrc
cd $DIR_FOAM_RUN
$MPIRUN $SOLVER -case $CASO -parallel

without any changes in OF config the system returns an error:

Code:

[compute-0-0.local:32745] mca: base: component_find: unable to open /share/apps/snft/snft3/Marco/OpenFOAM/ThirdParty-2.0.x/platforms/linux64Gcc45/openmpi-1.5.3/lib/openmpi/mca_ess_slurmd: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
/opt/openmpi/bin/mpirun: symbol lookup error: /share/apps/snft/snft3/Marco/OpenFOAM/ThirdParty-2.0.x/platforms/linux64Gcc45/openmpi-1.5.3/lib/openmpi/mca_plm_rshd.so: undefined symbol: orte_plm_base_rsh_launch_agent_setup

So, I tried to set WM_MPLIB=SYSTEMOPENMPI in bashrc and edit the settings.sh file from:

Code:

SYSTEMOPENMPI)
    # Use the system installed openmpi, get library directory via mpicc
    export FOAM_MPI=openmpi-system

    # Set compilation flags here instead of in wmake/rules/../mplibSYSTEMOPENMPI
    export PINC="`mpicc --showme:compile`"
    export PLIBS="`mpicc --showme:link`"
    libDir=`echo "$PLIBS" | sed -e 's/.*-L\([^ ]*\).*/\1/'`

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    export MPI_ARCH_PATH="${libDir%/*}"

    if [ "$FOAM_VERBOSE" -a "$PS1" ]
    then
        echo "Using system installed MPI:"
        echo "    compile flags : $PINC"
        echo "    link flags    : $PLIBS"
        echo "    libmpi dir    : $libDir"
    fi

    _foamAddLib    $libDir
    unset libDir
    ;;

to

Code:

SYSTEMOPENMPI)
    # Use the system installed openmpi, get library directory via mpicc
    export FOAM_MPI=openmpi-system

    # Set compilation flags here instead of in wmake/rules/../mplibSYSTEMOPENMPI
    export PINC="`/opt/openmpi/bin/mpicc --showme:compile`"
    export PLIBS="`/opt/openmpi/bin/mpicc --showme:link`"
    libDir=/opt/openmpi/lib

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    export MPI_ARCH_PATH=/opt/openmpi

    if [ "$FOAM_VERBOSE" -a "$PS1" ]
    then
        echo "Using system installed MPI:"
        echo "    compile flags : $PINC"
        echo "    link flags    : $PLIBS"
        echo "    libmpi dir    : $libDir"
    fi

    _foamAddLib    $libDir
    unset libDir
    ;;

But also in this case the system reports some errors (one for each processor):
Code:

--> FOAM FATAL ERROR:
Trying to use the dummy Pstream library.
This dummy library cannot be used in parallel mode

    From function UPstream::init(int& argc, char**& argv)
    in file UPstream.C at line 37.

FOAM exiting

Finally, I restored the default bashrc and settings.sh to use the OpenMPI provided with CentFOAM. Now the program runs (snappyHexMesh), but I can't use more than one node and before the execution some warnings are shown (see attached logs).

wyldckat February 22, 2012 17:08

Hi Marco,

:eek: Those errors almost look painful...

You were going in the right direction with "WM_MPLIB=SYSTEMOPENMPI". All that was missing was the dedicated Pstream and PTScotch libraries:
  1. Pstream:
    Code:

    foam
    wmake all src/Pstream

  2. PTScotch:
    Code:

    foam3rdParty
    ./Allwmake

Well, running Allwmake once more on the OpenFOAM folder should do the trick as well, perhaps even better:
Code:

foam
./Allwmake

Best regards,
Bruno

ciberbob1987 February 22, 2012 17:17

Hi Bruno,
I will try as soon as possible your solution, but I have gcc version 4.1.2 and flex is not installed. Is it sufficient to compile third party without errors?

wyldckat February 22, 2012 17:35

Hi Marco,

You should have gotten a shiny new gcc 4.4 or 4.5 with CentFOAM! For using strictly with Cent/OpenFOAM... i.e., while you have OpenFOAM's environment active (a shell where you're able to run OpenFOAM applications).

Best regards,
Bruno

linnemann February 23, 2012 07:22

Hi

Why do you want to change the openmpi location?

centFOAM comes with its own openmpi install.

Youre trying to mix a version of openmpi (most likely compiled with gcc 4.1.2) with a newer version compiled with gcc 4.5 or higher.

This is bound to give you errors :-). If you want to use system openmpi you have to recompile OF.

If your cluster dont have infiniband, the provided openmpi in centFOAM should be just as good as any system openmpi.

EDIT: Just saw that your cluster indeed have inifiniband so you will have to recompile as Brunos suggest using the system openmpi.

Best

wyldckat February 23, 2012 09:20

Hi Linnemann,
Quote:

Originally Posted by linnemann (Post 345895)
EDIT: Just saw that your cluster indeed have inifiniband so you will have to recompile as Brunos suggest using the system openmpi.

If CentFOAM delivers a complete installation (including the "lnInclude" folders with the correct time stamps), then it should only be necessary to compile Pstream and PTScotch, due to OpenFOAM's building structure :)

Best regards,
Bruno

ciberbob1987 February 25, 2012 20:21

Hi everybody,I have re-compiled the PStream code and now all seems to work fine.
First simulation with posousSimpleFoam runned in parallel on 7 CPU with no errors.
Many thanks to all, you saved me!


All times are GMT -4. The time now is 12:42.