[aow]

Hello CFD Online,

I've been trying to setup my own installation of OpenFOAM-5.x in an HPC cluster using Intel compiler and mpi modules provided in the cluster, particularly icc 18.0.2 and impi 18.0.2.

To begin with, I modified some of the environment variables in $WORK/OpenFOAM-5.x/etc/bashrc as follows, where $WORK is the directory containing the OpenFOAM-5.x and ThirdParty-5.x directories.

export FOAM_INST_DIR=$WORK
export WM_COMPILER=Icc
export WM_MPLIB=INTELMPI

Also, within the file $WORK/OpenFOAM-5.x/etc/config.sh/mpi, I added the following bold lines in the INTELMPI section.

INTELMPI)
	# Stampede2 location of Intel MPI 18.0.2
	export MPI_ROOT=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64
    
	# No trailing slash
    [ "${MPI_ROOT%/}" = "${MPI_ROOT}" ] || MPI_ROOT="${MPI_ROOT%/}"

    export FOAM_MPI="${MPI_ROOT##*/}"
    export MPI_ARCH_PATH=$MPI_ROOT
	_foamAddPath $MPI_ARCH_PATH/bin
	_foamAddPath $MPI_ARCH_PATH/include
	_foamAddPath $MPI_ARCH_PATH/lib

Then I wrote the following Slurm batch script file to submit to a dev node in the HPC cluster for compiling the ThirdParty and OpenFOAM software by submitting it to a queue using the command sbatch compileOpenFOAM500.sh.

#!/bin/bash
#SBATCH --job-name=compileOF5           # job name
##SBATCH --output=compileOF5.o$j         # standard output file name
##SBATCH --error=compileOF5.e$j          # standard error file name
#SBATCH --account=DesignSafe-Motley-UW  # project allocation name (required if you have >1)
#SBATCH --partition=skx-dev             # queue name
#SBATCH --time=2:00:00                  # run time (D-HH:MM:SS)
#SBATCH --nodes=1                       # total number of nodes
#SBATCH --ntasks=48                     # total number of MPI tasks
##SBATCH --chdir=$WORK                   # path to job work directory
##SBATCH --mail-type=all                 # send emails when the job begins, ends, or fails
##SBATCH --mail-user=andrew25@uw.edu     # job alert emails are sent to this address

# Remove old log files
for object in 'log.'*; do
	if [ -f $object ]; then
		rm $object
	fi
done

# Remove old slurm output files
for object in 'slurm-'*; do
	if [ -f $object ]; then
		rm $object
	fi
done

# COMPILER:
# --------
# According to https://openfoam.org/download/source/software-for-compilation/ gcc version
# 4.8 or higher and icc version 17.0.4 or higher are required for OpenFOAM-5.x to compile.
module load intel/18.0.2 # options: intel/16.0.3, intel/17.0.4, intel/18.0.0, intel/18.0.2

# PARALLEL PROCESSING:
# -------------------
# To find the Intel MPI root, enter "module load impi/18.0.2" and then "echo $TACC_IMPI_DIR".
# Based on https://linuxcluster.wordpress.com/2018/06/04/compiling-openfoam-5-with-intel-mpi/
# I added "/intel64" to the end of the path printed to the command line and exported this path
# within the script $WORK/OpenFOAM-5.x/etc/config.sh/mpi using the command...
# export MPI_ROOT=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64
# ..., which I only added to the INTELMPI section of the script. I also modified mplibINTELMPI
# in the linux64Icc and linuxIA64Icc sub-directories of $WORK/OpenFOAM-5.x/wmake/rules so that
# PINC and PLIBS did not have "64" at the end of "include" and "lib".
module load impi/18.0.2 # options: impi/17.0.3, impi/18.0.0, impi/18.0.2

# SET ENVIRONMENT VARIABLES:
# -------------------------
cd $WORK
source OpenFOAM-5.x/etc/bashrc
export WM_SCHEDULER=wmakeScheduler
export WM_NCOMPROCS=48 # make sure this is identical to ntasks in the SBATCH commands
export WM_HOSTS=`hostname`:$WM_NCOMPROCS

# COMPILE ThirdParty-5.x:
# ----------------------
cd ThirdParty-5.x
./Allclean >> ../log.makeThirdParty 2>&1 # make sure nothing is left behind from a previous compilation attempt that shouldn't be here
./Allwmake -j >> ../log.makeThirdParty 2>&1 # -j runs the command in parallel
#./makeParaView >> ../log.makeThirdParty 2>&1
cd ..

# COMPILE OpenFOAM-5.0:
# --------------------
cd OpenFOAM-5.x
wmRefresh >> ../log.makeOpenFOAM 2>&1 # reset environment variables based on newly created directories after compiling ThirdParty-5.x
wcleanPlatform >> ../log.makeOpenFOAM 2>&1 # make sure nothing is left behind from a previous compilation attempt that shouldn't be here
./Allwmake -j >> ../log.makeOpenFOAM 2>&1

My first issue was compiling scotch 6.0.3 in the ThirdParty-5.x directory using the ./Allwmake file, where the compiler was causing issues. To get around this, I made a copy of the file Makefile.inc.i686_pc_linux2.shlib-OpenFOAM, calling it Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-Icc and changing all instances where gcc or mpicc showed up to be icc or mpiicc, respectively. See the bold lines below for these changes.

EXE		=
LIB		= .so
OBJ		= .o

MAKE		= make
AR		= icc
ARFLAGS		= $(WM_CFLAGS) -shared -o
CAT		= cat
CCS		= icc
CCP		= mpiicc
CCD		= mpiicc
CFLAGS		= $(WM_CFLAGS) -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict

ifeq ($(WM_LABEL_OPTION),Int64)
CFLAGS		+= -DINTSIZE64
endif

CLIBFLAGS	= -shared
LDFLAGS		= -Xlinker --no-as-needed $(WM_LDFLAGS) -lz -lm -lrt
CP		= cp
LEX		= flex -Pscotchyy -olex.yy.c
LN		= ln
MKDIR		= mkdir
MV		= mv
RANLIB		= echo
YACC		= bison -pscotchyy -y -b y

To use this file instead of the default one, I changed the Makefile.inc name in $WORK/ThirdParty-5.x/Allwmake in the scotch compilation section, which starts at line 186 of the Allwmake script, as shown below.

# building scotch is still a bit of a pain
echo ========================================
echo "Build Scotch decomposition library $SCOTCH_VERSION"
echo "    $SCOTCH_ARCH_PATH"

# this needs generalizing for any compiler, since it's hard-coded to GCC
#scotchMakefile=../../etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM
scotchMakefile=../../etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-Icc

These changes allowed scotch to compile successfully, but I found a new error after this, namely that the lockfile binary was not able to be found. After browsing around CFD Online a while, I found that I should check to see if the software procmail was installed. I found that it was not available by default in the HPC cluster I'm working with so I downloaded and compiled procmail-3.22 successfully, but to get OpenFOAM to find my installation I had to add $WORK/procmail-3.22/bin to the $PATH variable.

export PATH=$WORK/procmail-3.22/bin:$PATH

Additionally, to get procmail-3.22 to compile successfully, I used the code changes shown in the ".diff" file I found in this LinuxQuestions.org forum post, which required editing the fields.c, formail.c, formisc.h, and formisc.c files to avoid getting the error message error: conflicting types for 'getline'.

Finally, I reached the point I'm stuck at now, where I'm getting two odd results in my log file for running $WORK/OpenFOAM-5.x/Allwmake.

LINE 1:
/work/04697/winter89/stampede2/OpenFOAM-5.x/etc/config.sh/aliases: line 84: wmUnset: command not found
LINES 39 - 53:
Allwmake src
version changed from previous build
removing .o files corresponding to OpenFOAM/global/global.o ...
Allwmake src/Pstream
wmake dummy
Making dependency list for source file UOPwrite.C
Making dependency list for source file UIPread.C
Making dependency list for source file UPstream.C
wmakeScheduler icpc -std=c++11 -fp-trap=common -fp-model precise -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -diag-disable 327,654,1125,2289,2304,11062,11074,11076 -O3  -DNoRepository  -IlnInclude -I. -I/work/04697/winter89/stampede2/OpenFOAM-5.x/src/OpenFOAM/lnInclude -I/work/04697/winter89/stampede2/OpenFOAM-5.x/src/OSspecific/POSIX/lnInclude   -fPIC -c UPstream.C -o /work/04697/winter89/stampede2/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/src/Pstream/dummy/UPstream.o
wmakeScheduler icpc -std=c++11 -fp-trap=common -fp-model precise -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -diag-disable 327,654,1125,2289,2304,11062,11074,11076 -O3  -DNoRepository  -IlnInclude -I. -I/work/04697/winter89/stampede2/OpenFOAM-5.x/src/OpenFOAM/lnInclude -I/work/04697/winter89/stampede2/OpenFOAM-5.x/src/OSspecific/POSIX/lnInclude   -fPIC -c UIPread.C -o /work/04697/winter89/stampede2/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/src/Pstream/dummy/UIPread.o
wmakeScheduler icpc -std=c++11 -fp-trap=common -fp-model precise -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -diag-disable 327,654,1125,2289,2304,11062,11074,11076 -O3  -DNoRepository  -IlnInclude -I. -I/work/04697/winter89/stampede2/OpenFOAM-5.x/src/OpenFOAM/lnInclude -I/work/04697/winter89/stampede2/OpenFOAM-5.x/src/OSspecific/POSIX/lnInclude   -fPIC -c UOPwrite.C -o /work/04697/winter89/stampede2/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/src/Pstream/dummy/UOPwrite.o
Warning: Permanently added 'c506-101.stampede2.tacc.utexas.edu' (ECDSA) to the list of known hosts.
bash: icpc: command not found
bash: icpc: command not found
bash: icpc: command not found

First, I tried checking if icpc existed in the cluster and got the following output, which I believe indicates all is well.

icpc --version
icpc (ICC) 18.0.2 20180210
Copyright (C) 1985-2018 Intel Corporation.  All rights reserved.

Then I tried looking into the first error as it seems to indicate definitions of OpenFOAM commands, environment variables, etc... might be missing or unset somehow, but I can't find anything that clears up this error or the icpc errors.

Does anyone have any ideas about where to go from here and/or experience compiling OpenFOAM-5.x using Intel software?

In case they prove helpful, I've attached the two log files created during running Allwmake in the ThirdParty-5.x and OpenFOAM-5.x directories.

[aow]

I just realized I forgot to mention that I also edited two files in the $WORK/OpenFOAM-5.x/wmake/rules directories. Specifically, in ../rules/linux64Icc/mplibINTELMPI and ../rules/linuxIA64Icc/mplibINTELMPI, I modified both files to read as follows.

PFLAGS     = -DMPICH_SKIP_MPICXX
PINC       = -isystem $(MPI_ARCH_PATH)/include
PLIBS      = -L$(MPI_ARCH_PATH)/lib -lmpi

I did this because of my $MPI_ROOT variable, which pointed to ../mpi/intel64 that has directories bin, include, lib, etc...

[fertinaz]

Hello Andrew

First error:

LINE 1: /work/04697/winter89/stampede2/OpenFOAM-5.x/etc/config.sh/aliases: line 84: wmUnset: command not found

can be resolved by switching off the flag that enforces bash to raise an error when variables are not set. So type

set +u

before you actually source OF environment in your script.

In fact, you can completely avoid using wmUnset and wmRefresh and instead apply them manually when needed.

For the compiler error, you might want to have a look at the Intel's environment variable ${I_MPI_ROOT}. I guess it'd be easier using it before setting up env-vars used by OF such as ${MPI_ROOT} and ${MPI_ARCH_PATH}.

In general the error you get indicates a problem with the way those modules are loaded, but your Slurm script looks correct. Therefore if you can use those compilation nodes interactively, I'd highly suggest you doing that. To me it looks like, after you source OF your entire path gets cleaned and as a result system cannot find the compiler you loaded.

For scotch, I'd use the file called Makefile.inc.x86-64_pc_linux2.shlib but that depends on the platform you're using. I doubt that the cluster you're working on has 32bit hardware architecture though... However if you were able to compile successfully, it shouldn't hurt.

Anyway, hope this helps

Fatih

[aow]

Hey Fatih,

Thanks for your reply! I was able to successfully compile OpenFOAM-5.x as well as OlaFlow last week on Friday, when I had some time to dig into the matter.

Concerning the "First Error", I still don't totally understand why this was an issue, but I ended up just pasting the commands directly instead of using the command aliases.

Concerning the "Second Error", I was able to resolve it by going through the following steps, some of which are the same as I mentioned in my first post.

First, I made sure to install all of the third party software besides scotch that I would need, some of which I think might have been missing during my initial compilation efforts. Note that in all code below $WORK refers to the directory in which I compiled ThirdParty-5.x and OpenFOAM-5.x in the TACC HPC cluster. The following list of software was installed...

Pre-installed in HPC cluster:
cmake 3.10.2
flex 2.5.37
gcc 6.3.0 ---> not used, I used icc instead
icc 18.0.2
impi 18.0.2
qt 4.8.7 ---> not used, ParaView 5.6.0 is already installed in the cluster

Installed by me in $WORK:
gmp 6.1.2
mpfr 4.0.2
mpc 1.1.0
procmail 3.22

After I installed these, I added the following lines to my .bashrc file in my $HOME directory.

export PATH=$WORK/bin:$PATH
export CPATH=$WORK/include:$CPATH
export LD_LIBRARY_PATH=$WORK/lib:$LD_LIBRARY_PATH
export LD_RUN_PATH=$WORK/lib:$LD_RUN_PATH

Second, I modified the same environment variable definitions in ../OpenFOAM-5.x/etc/bashrc as I mentioned before, namely...

export FOAM_INST_DIR=$WORK
export WM_COMPILER_TYPE=system
export WM_COMPILER=Icc
export WM_MPLIB=INTELMPI

Third, I edited ../OpenFOAM-5.x/etc/config.sh/compiler by adding a new compiler case and changing the versions of gmp, mpfr, and mpc to the correct ones

case "$WM_COMPILER_TYPE" in
OpenFOAM | ThirdParty)

    # Default versions of GMP, MPFR and MPC, override as necessary
    #gmp_version=gmp-5.1.2
    #mpfr_version=mpfr-3.1.2
    #mpc_version=mpc-1.0.1
    gmp_version=gmp-6.1.2
    mpfr_version=mpfr-4.0.2
    mpc_version=mpc-1.1.0

    case "$WM_COMPILER" in
    Icc)
        # Using Icc - not gcc
        export WM_CC='mpicc'
        export WM_CXX='mpicxx'
        ;;
    Gcc | Gcc48)
        gcc_version=gcc-4.8.5
        ;;
    Gcc49)
        gcc_version=gcc-4.9.3
        ;;
    Gcc51)
        gcc_version=gcc-5.1.0
        ;;
    Gcc52)
        gcc_version=gcc-5.2.0
        ;;
    Gcc53)
        gcc_version=gcc-5.3.0
        ;;
    Gcc61)
        gcc_version=gcc-6.1.0
        ;;
    Gcc81)
        gcc_version=gcc-8.1.0
        ;;
    Clang)
        # Using clang - not gcc
        export WM_CC='clang'
        export WM_CXX='clang++'
        clang_version=llvm-3.7.0
        ;;
    *)
        echo 1>&2
        echo "Warning in $WM_PROJECT_DIR/etc/config.sh/compiler:" 1>&2
        echo "    Unknown OpenFOAM compiler type '$WM_COMPILER'" 1>&2
        echo "    Please check your settings" 1>&2
        echo 1>&2
        ;;
    esac
    ;;

esac

Next, I removed all modifications I made to ../OpenFOAM-5.x/etc/config.sh/mpi...

INTELMPI)
## Stampede2 location of Intel MPI 18.0.2
#export MPI_ROOT=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64
    
# No trailing slash
    [ "${MPI_ROOT%/}" = "${MPI_ROOT}" ] || MPI_ROOT="${MPI_ROOT%/}"

    export FOAM_MPI="${MPI_ROOT##*/}"
    export MPI_ARCH_PATH=$MPI_ROOT
#_foamAddPath $MPI_ARCH_PATH/bin
#_foamAddPath $MPI_ARCH_PATH/include
#_foamAddPath $MPI_ARCH_PATH/lib

    if [ ! -d "$MPI_ROOT" -o -z "$MPI_ARCH_PATH" ]
    then
        echo "Warning in $WM_PROJECT_DIR/etc/config.sh/settings:" 1>&2
        echo "    MPI_ROOT not a valid mpt installation directory or ending" \
             " in a '/'." 1>&2
        echo "    Please set MPI_ROOT to the mpt installation directory." 1>&2
        echo "    MPI_ROOT currently set to '$MPI_ROOT'" 1>&2
    fi

    if [ "$FOAM_VERBOSE" -a "$PS1" ]
    then
        echo "Using INTEL MPI:" 1>&2
        echo "    MPI_ROOT : $MPI_ROOT" 1>&2
        echo "    FOAM_MPI : $FOAM_MPI" 1>&2
    fi

    _foamAddPath    $MPI_ARCH_PATH/bin64
    _foamAddLib     $MPI_ARCH_PATH/lib64
#_foamAddPath    $MPI_ARCH_PATH/include64
    ;;

...and instead just used the follwing line in my batch scripts for compiling and running OpenFOAM.

export  MPI_ROOT=$I_MPI_ROOT

Then, I edited two files in ../OpenFOAM-5.x/wmake/rules/linux64Icc, particularly c, and c++. In c, I changed to...

cc = mpicc

..and in c++ I changed to...

CC = mpicxx -std=c++11 -fp-trap=common -fp-model precise

Also, I undid the changes I made to mplibINTELMPI so that it now reads as the default setup.

PFLAGS     = -DMPICH_SKIP_MPICXX
PINC       = -isystem $(MPI_ARCH_PATH)/include64
PLIBS      = -L$(MPI_ARCH_PATH)/lib64 -lmpi

Next, for compiling scotch 6.0.3, I used the same modified Makefile.inc as before, which I made by copying the default file ../ThirdParty-5.x/etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM and renaming it as Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-Icc, where I changed the following lines.

AR= icc
CCS= icc
CCP= mpiicc
CCD= mpiicc

To actually use this file instead of the default one provided by openfoam.org, I changed the Makefile.inc name in $WORK/ThirdParty-5.x/Allwmake in the scotch compilation section, which starts at line 186 of the Allwmake script, as shown below.

# this needs generalizing for any compiler, since it's hard-coded to GCC
#scotchMakefile=../../etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM
scotchMakefile=../../etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM-Icc

The last thing I needed to figure out was listed at the bottom of the webpage explaining what the system requirements are for OpenFOAM. On this webpage, the last section titled Using IntelMPI provides an explanation as to why you need to add the following line to your compilation batch script or execute in the terminal prior to running ./Allwmake to compile OpenFOAM.

export LD_PRELOAD="libmpi.so"

However, "libmpi.so" is just a place holder and you need to use the actual path to where the file is located so I filled in the correct path for the HPC cluster I'm working in as follows.

export LD_PRELOAD=$I_MPI_ROOT/lib64/libmpi.so

At this point, all of the necessary edits had been made and I successfully completed running my OpenFOAM compilation batch script listed below.

#!/bin/bash
#SBATCH --job-name=compileOF5x          # job name
##SBATCH --output=compileOF5x.o$j        # standard output file name
##SBATCH --error=compileOF5x.e$j         # standard error file name
#SBATCH --account=DesignSafe-Motley-UW  # project allocation name (required if you have >1)
#SBATCH --partition=skx-dev             # queue name
#SBATCH --time=2:00:00                  # run time (D-HH:MM:SS)
#SBATCH --nodes=1                       # total number of nodes
#SBATCH --ntasks=48                     # total number of MPI tasks
##SBATCH --chdir=$WORK                   # path to job work directory
##SBATCH --mail-type=all                 # send emails when the job begins, ends, or fails
##SBATCH --mail-user=andrew25@uw.edu     # job alert emails are sent to this address

# Remove old log files
for object in 'log.'*; do
if [ -f $object ]; then
rm $object
fi
done

# Remove old slurm output files
for object in 'slurm-'*; do
if [ -f $object ]; then
rm $object
fi
done

# COMPILER:
# --------
# According to https://openfoam.org/download/source/software-for-compilation/ gcc version
# 4.8 or higher and icc version 17.0.4 or higher are required for OpenFOAM-5.x to compile.
module load intel/18.0.2 # options: intel/16.0.3, intel/17.0.4, intel/18.0.0, intel/18.0.2

# PARALLEL PROCESSING:
# -------------------
# To find the Intel MPI root, enter "module load impi/18.0.2" and then "echo $TACC_IMPI_DIR"
# or "echo $I_MPI_ROOT".
module load impi/18.0.2 # options: impi/17.0.3, impi/18.0.0, impi/18.0.2

# SET ENVIRONMENT VARIABLES:
# -------------------------
export MPI_ROOT=$I_MPI_ROOT # should be identical to $TACC_IMPI_DIR
export LD_PRELOAD=$MPI_ROOT/lib64/libmpi.so # see the "Using IntelMPI" discussion at https://www.openfoam.com/documentation/system-requirements.php
source $WORK/OpenFOAM-5.x/etc/bashrc
cd $WORK/ThirdParty-5.x
./Allclean >> ../log.compileOF5x # make sure nothing is left behind from a previous compilation attempt that shouldn't be here
cd $WORK/OpenFOAM-5.x
./wcleanPlatform >> ../log.compileOF5x  # make sure nothing is left behind from a previous compilation attempt that shouldn't be here
export WM_COMPILE_OPTION=Debug
export WM_SCHEDULER=wmakeScheduler
export WM_NCOMPROCS=48 # make sure this is identical to ntasks in the SBATCH commands
export WM_HOSTS=`hostname`:$WM_NCOMPROCS

# COMPILE ThirdParty-5.x:
# ----------------------
cd $WORK/ThirdParty-5.x
./Allwmake -j >> ../log.compileOF5x 2>&1 # -j runs the command in parallel
#./makeParaView >> ../log.compileOF5x 2>&1

# RESET ENVIRONMENT VARIABLES: needs to be done to account for newly created directories after compiling ThirdParty-5.x
# ---------------------------
#wmRefresh >> ../log.compileOF5x 2>&1 # alias for the four command lines below (doesn't work in this script running on skx-dev)
wmProjectDir=$WM_PROJECT_DIR
foamSettings=$FOAM_SETTINGS
#wmUnset # alias for the command line below (doesn't work in this script running on skx-dev)
. $WM_PROJECT_DIR/etc/config.sh/unset
. $wmProjectDir/etc/bashrc $foamSettings

# COMPILE OpenFOAM-5.0:
# --------------------
cd $WORK/OpenFOAM-5.x
./Allwmake -j >> ../log.compileOF5x 2>&1

Furthermore, I successfully compiled OlaFlow using the following batch script.

#!/bin/bash
#SBATCH --job-name=compileOlaFlow       # job name
##SBATCH --output=compileOlaFlow.o$j     # standard output file name
##SBATCH --error=compileOlaFlow.e$j      # standard error file name
#SBATCH --account=DesignSafe-Motley-UW  # project allocation name (required if you have >1)
#SBATCH --partition=skx-dev             # queue name
#SBATCH --time=2:00:00                  # run time (D-HH:MM:SS)
#SBATCH --nodes=1                       # total number of nodes
#SBATCH --ntasks=48                     # total number of MPI tasks

# Remove old slurm output files
for object in 'slurm-'*; do
if [ -f $object ]; then
rm $object
fi
done

# COMPILER:
# --------
# According to https://openfoam.org/download/source/software-for-compilation/ gcc version
# 4.8 or higher and icc version 17.0.4 or higher are required for OpenFOAM-5.x to compile.
module load intel/18.0.2 # options: intel/16.0.3, intel/17.0.4, intel/18.0.0, intel/18.0.2

# PARALLEL PROCESSING:
# -------------------
# To find the Intel MPI root, enter "module load impi/18.0.2" and then "echo $TACC_IMPI_DIR"
# or "echo $I_MPI_ROOT".
module load impi/18.0.2 # options: impi/17.0.3, impi/18.0.0, impi/18.0.2

# SET ENVIRONMENT VARIABLES:
# -------------------------
export MPI_ROOT=$I_MPI_ROOT # should be identical to $TACC_IMPI_DIR
export LD_PRELOAD=$MPI_ROOT/lib64/libmpi.so # see the "Using IntelMPI" discussion at https://www.openfoam.com/documentation/system-requirements.php
source $WORK/OpenFOAM-5.x/etc/bashrc

# CLEANUP POSSIBLE PRIOR INSTALLATION:
# -----------------------------------
if [ -f $WORK/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/lib/libwaveGeneration.so ]; then
rm -f $WORK/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/lib/libwaveGeneration.so
fi
if [ -f $WORK/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/lib/libwaveAbsorption.so ]; then
rm -f $WORK/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/lib/libwaveAbsorption.so
fi
if [ -f $WORK/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/bin/olaDyMFlow ]; then
rm -f $WORK/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/bin/olaDyMFlow
fi
if [ -f $WORK/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/bin/olaFlow ]; then
rm -f $WORK/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt/bin/olaFlow
fi
if [ -d $WORK/olaFlow ]; then
rm -rf $WORK/olaFlow
fi
git clone git://github.com/phicau/olaFlow.git

# REDIRECT COMPILATION OUTPUT: install in $WORK/OpenFOAM-5.x/platforms/linux64IccDPInt32Opt instead of in $HOME/OpenFOAM/winter89-5.x/platforms/linux64IccDPInt32Opt
# ---------------------------
sed -i "s/FOAM_USER_LIBBIN/FOAM_LIBBIN/g" $WORK/olaFlow/genAbs/waveAbsorption/Make/files
sed -i "s/FOAM_USER_LIBBIN/FOAM_LIBBIN/g" $WORK/olaFlow/genAbs/waveGeneration/Make/files
sed -i "s/FOAM_USER_APPBIN/FOAM_APPBIN/g" $WORK/olaFlow/solvers/olaFlowOF500/Make/files
sed -i "s/FOAM_USER_APPBIN/FOAM_APPBIN/g" $WORK/olaFlow/solvers/olaFlowOF500/olaDyMFlow/Make/files
strA=$(grep "ln -s" $WORK/olaFlow/allMake)
strB ='echo Not creating symbolic link to HOME/OpenFOAM/winter89-5.x/platforms/linux64IccDPInt32Opt/bin/olaDyMFlow'
sed -i "s/$strA/$strB/g" $WORK/olaFlow/allMake # prevent unnecessary attempt to create symbolic link

# SOURCE CODE TYPO FIXES: needed for now to avoid a warning/error message until Pablo fixes it (I tried to do it properly, but failed)
# ----------------------
strA='PointPatchFieldMapper'
strB='pointPatchFieldMapper'
sed -i "s/$strA/$strB/g" $WORK/olaFlow/genAbs/waveGeneration/wavemakerMovement/wavemakerMovement.H
sed -i "s/$strA/$strB/g" $WORK/olaFlow/genAbs/waveGeneration/wavemakerMovement/wavemakerMovement.C

# COMPILE olaFlow:
# ---------------
cd $WORK/olaFlow
./allMake > ../log.compileOlaFlow 2>&1

Note that there was not actually an error occurring when I tried to compile OlaFlow where I thought I was making an edit to avoid one. Please see this forum post concerning the matter.

I hope this might be helpful to anyone else trying to work in a similar situation where you need to use Intel compiler and mpi software in an HPC cluster.

[aow]

I had one more thing I forgot to mention...

Fatih, you mentioned that I was using a 32-bit Makefile.inc for compiling scotch 6.0.3, which I did not realize at first since I was not familiar with the i686 notation as opposed to x86-64; however, the file I modified is the one selected by the openfoam.org developers and is setup to be used by default when you run ./Allwmake in the ThirdParty-5.x directory. I'm wondering if it's intentionally 32-bit for some reason or if it's a mistake in the naming of the file and in fact it's actually set to be 64-bit instead. I'm not a pro enough at this sort of thing to be able to tell immediately by looking at the contents of the file so I cannot really say anything for sure other than my compiled OpenFOAM version has successfully run all of the tutorials and models of my own that I've tried out so far.

Maybe Bruno (i.e. wyldckat) has something to say about this...?