[wildfire230]

Hi All,

I am trying to compile OpenFOAM 5.0 on the Titan supercomputer, and I am having lots of problems figuring out what compile options to set, etc... I do not have much of a background in architecture specific things, and the help desk has limited experience with OpenFOAM, so I am trying to DIY it.

Titan is a Cray XK7 supercomputer. This is what I have from the system user guide:

Compiling code on Titan (and other Cray machines) is different than compiling code for commodity or beowulf-style HPC linux clusters. Among the most prominent differences:

• Cray provides a sophisticated set of compiler wrappers to ensure that the compile environment is setup correctly. Their use is highly encourged.
• In general, linking/using shared object libraries on compute partitions is not supported.
• Cray systems include many different types of nodes, so some compiles are, in fact, cross-compiles.

Available Compilers

The following compilers are available on Titan:

• PGI, the Portland Group Compiler Suite (default)
• GCC, the GNU Compiler Collection
• CCE, the Cray Compiling Environment
• Intel, Intel Composer XE

7.1. Cray Compiler Wrappers

• (Back to Top) Cray provides a number of compiler wrappers that substitute for the traditional compiler invocation commands. The wrappers call the appropriate compiler, add the appropriate header files, and link against the appropriate libraries based on the currently loaded programming environment module. To build codes for the compute nodes, you should invoke the Cray wrappers via:

• cc To use the C compiler
• CC To use the C++ compiler
• ftn To use the FORTRAN 90 compiler

So, first question, what should these flags be set to:

export WM_NCOMPROCS=8
export WM_COMPILER=CC                                                                        
export WM_CC=cc
export WM_CXX=CC                                                                                 
export WM_MPLIB=CRAYMPI                                                                        
export WM_MPI_CC=cc
export WM_MPI_CXX=CC
export WM_MPIRUN_PROG=aprun
export WM_LDFLAGS=-dynamic
export LDFLAGS=-dynamic
export LD=CC

and, what, if any, modules do I need to have loaded? I believe I have to use cc and CC as the c and c++ compiler options, since they are the compiler wrappers provided. I am not sure at all what to set WM_MPLIB to, but I tried CRAYMPI. As far as I know, it uses MPICH. The system has a default loaded module of cray-mpich/7.5.2 and it also has the pgi compiler pgi/16.10.0 loaded as default.


Just as an example of the errors I am having, when I run ./Allwmake in ThirdParty-5.0, I get the following (truncated) output:

========================================
Start ThirdParty Allwmake
========================================

========================================
Build MPI libraries if required

========================================
Build Scotch decomposition library scotch_6.0.3
    /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3
    scotch header in /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3/include
    scotch libs   in /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64ccDPInt32/lib

========================================
Build PTScotch decomposition library scotch_6.0.3 (uses MPI)
    /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3

+ cd scotch_6.0.3/src
+ prefixDIR=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3
+ libDIR=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64ccDPInt32/lib/mpich2-1.1.1p1
+ incDIR=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3/include/mpich2-1.1.1p1
+ mkdir -p /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3
+ mkdir -p /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64ccDPInt32/lib/mpich2-1.1.1p1
+ configOpt='prefix=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3 libdir=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64ccDPInt32/lib/mpich2-1.1.1p1 includedir=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3/include/mpich2-1.1.1p1'
+ '[' -f ../../etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM ']'
+ rm -f Makefile.inc
+ ln -s ../../etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM Makefile.inc
+ '[' -f Makefile.inc ']'
+ unset configEnv
+ '[' cc '!=' gcc ']'
+ configEnv='CC=cc CCS=cc'
+ make realclean
(cd libscotch ;      make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
rm -f *~ *.o lib*.so parser_yy.c parser_ly.h parser_ll.c *scotch.h *scotchf.h y.output *dummysizes
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
(cd scotch ;         make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/scotch'
rm -f *~ *.o acpl amk_ccc amk_fft2 amk_grf amk_hy amk_m2 amk_p2 atst gbase gcv *ggath *gmap gmk_hy gmk_m2 gmk_m3 gmk_msh gmk_ub2 gmtst *gord gotst gout *gpart *gscat *gtst mcv mmk_m2 mmk_m3 mord mtst
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/scotch'
(cd libscotchmetis ; make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotchmetis'
rm -f *~ *.o lib*.so
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotchmetis'
(cd check ;          make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
rm -f *~ *.o
rm -f                 \
                    test_common_thread        \
                    test_scotch_graph_color        \
                    test_scotch_graph_map        \
                    test_scotch_graph_order        \
                    test_scotch_graph_part_ovl    \
                    test_scotch_dgraph_band        \
                    test_scotch_dgraph_check    \
                    test_scotch_dgraph_coarsen    \
                    test_scotch_dgraph_grow        \
                    test_scotch_dgraph_redist    \
                    test_strat_par            \
                    test_strat_seq
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
(cd esmumps ;        make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f *~ common.h *.o lib*.so main_esmumps
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f ../bin/* ../include/* ../lib/*
+ make -j 8 CC=cc CCS=cc ptscotch
(cd libscotch ;      make VERSION=6 RELEASE=0 PATCHLEVEL=3 ptscotch && make ptinstall)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
make CC="cc" CCD="cc"    \
                    scotch.h                \
                    scotchf.h                \
                    libscotch.so                \
                    libscotcherr.so            \
                    libscotcherrexit.so
make[2]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
cc -m64 -fPIC -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -shared -c arch.c -o arch.o
cc -m64 -fPIC -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -shared -c arch_cmplt.c -o arch_cmplt.o



...
...


cc -m64 -fPIC -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -shared -c bgraph_bipart_df.c -o bgraph_bipart_df.o
pgcc-Error-Unknown switch: -Xlinker
make[2]: *** [dummysizes] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
make[1]: *** [scotch] Error 2
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
make: *** [ptscotch] Error 2
+ make realclean
(cd libscotch ;      make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
rm -f *~ *.o lib*.so parser_yy.c parser_ly.h parser_ll.c *scotch.h *scotchf.h y.output *dummysizes
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
(cd scotch ;         make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/scotch'
rm -f *~ *.o acpl amk_ccc amk_fft2 amk_grf amk_hy amk_m2 amk_p2 atst gbase gcv *ggath *gmap gmk_hy gmk_m2 gmk_m3 gmk_msh gmk_ub2 gmtst *gord gotst gout *gpart *gscat *gtst mcv mmk_m2 mmk_m3 mord mtst
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/scotch'
(cd libscotchmetis ; make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotchmetis'
rm -f *~ *.o lib*.so
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotchmetis'
(cd check ;          make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
rm -f *~ *.o
rm -f                 \
                    test_common_thread        \
                    test_scotch_graph_color        \
                    test_scotch_graph_map        \
                    test_scotch_graph_order        \
                    test_scotch_graph_part_ovl    \
                    test_scotch_dgraph_band        \
                    test_scotch_dgraph_check    \
                    test_scotch_dgraph_coarsen    \
                    test_scotch_dgraph_grow        \
                    test_scotch_dgraph_redist    \
                    test_strat_par            \
                    test_strat_seq
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
(cd esmumps ;        make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f *~ common.h *.o lib*.so main_esmumps
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f ../bin/* ../include/* ../lib/*

    WARNING: required include file 'ptscotch.h' not found!

========================================
Build Metis decomposition
    optional component Metis was not found
========================================
Build CGAL
    optional component was not found

========================================
Done ThirdParty Allwmake
========================================

I would HUGELY appreciate any help or advice.


Thanks!