OpenFOAM static build on Cray XT5
 

Hi, all.

I want to report the partiallly successful compilation of OpenFOAM on Cray XT5. As you know, Cray XT5
uses CNL for their node exectuion, so I had to compile the OpenFOAM statically. I could compile
almost all solvers and utilities (but some of them didn't work correctly.) I think it's easy to apply
these procedures to other static-linked only platforms.

Here is the recipe:

0. Switch to GNU compiler (for Cray XT5 module system)
$ module swap PrgEnv-pgi PrgEnv-gnu

1. Expand OpenFOAM-1.5.x.tar.gz (fetched from git repo) and ThirdParty.tar.gz into the CNL-accesible
directory and remove the needless libraries (malloc, zlib and openmpi) in ./ThirdParty. Edit
ThirdParty/Allwmake to comment out malloc and zlib build directives.

NOTE: if you want to use openmpi attached in ThirdParty.tar.gz, you must modify openmpi build
configure script in Allwmake.
Change "--enable-shared --disable-static" to "--disable-shared --enable-static",
and add "--without-memory-manager --without-libnuma".
cf. http://www.open-mpi.org/faq/?category=mpi-apps#static-mpi-apps

2. Edit etc/{bashrc,settings.sh}:
bashrc:
foamInstall=$YOUR_INSTALLED_DIR/$WM_PROJECT
WM_PROJECT_USER_DIR=$FOAM_INST_DIR/$USER-$WM_PROJECT_VERSION
WM_MPLIB=MPICH

settings.sh:
comilerInstall=OpenFOAM => System

case "$WM_MPLIB" in
...
MPICH) => MPICH_ORG)
....
MPI) => MPICH)
export MPI_ARCH_PATH=$MPICH_DIR
export FOAM_MPI_LIBBIN=$FOAM_LIBBIN

Finally, read all configurations.
$ . etc/bashrc

3. Change libso to libo in all Allwmake:
$ cd $FOAM_INST_DIR
$ find -name Allwmake -exec ruby -p -i.bak -e"gsub(/libso/, 'libo')" {} \;
And make solvers not to link indivisual libraries:
$ find -name options -exec ruby -p -i.bak -e"gsub(/EXE_LIBS/, 'EXE_LIBS_ORG')" {} \;

4. Comment out all codes relevant with libdl functions:
You can find them by executing this on shell:
$ find -name "*.C" -exec grep -nH dlfcn {} \;
./src/OpenFOAM/db/dlLibraryTable/dlLibraryTable.C:29:#include <dlfcn.h>
./src/OSspecific/Unix/printStack.C:36:#include <dlfcn.h>

And in $WM_PROJECT_DIR/wmake/rules/$WM_ARCH$WM_COMPILER/general,
replace "-ldl" by "-lz".

5. Make all libraries:
$ cd $WM_PROJECT_DIR
Comment out "(cd applications ..." in Allwmake.
$ export WM_NCOMPPROCS=2; ./Allwmake

6. Make the whole OpenFOAM library:
$ cd $FOAM_LIBBIN
$ ar rcvs libOpenFOAM.a lib*.o

7. Make solvers & utilities Edit $WM_PROJECT_DIR/wmake/rules/$WM_ARCH$WM_COMPILER/c++ to change the compiler and environment
variable:
CC = g++ => CC (or mpiCC)
LINK_LIBS = $(c++DBUG) => -static -Wl,--allow-multiple-definition

$ cd $FOAM_APP
$ ./Allwmake

Known Problems:
* The "functionObjectLibs" directive doesn't work because it loads shared library.
* Linker option "--allow-multiple-definition" suppresses errors of multiple definition of the lexers. I
don't have any ideas about what side-effects it bring on.

Cheers,
asaijo

Hi Asaijo,

Can you clarify in step 4 how you comment out those codes? Did you put a double slash comment '//' in Make/files? I did this and after making libOpenFOAM.a I still get errors about undefined reference to dlLibaryTable and printStack.

Thanks,

David


u/home/dhebert/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.a(libOpenFOAM.o): In function `bool Foam::dlLibraryTable::open<Foam::HashTable<Foam::a utoPtr<Foam::GAMGAgglomeration> (*)(Foam::lduMatrix const&, Foam::dictionary const&), Foam::word, Foam::string::hash>*>(Foam::dictionary const&, Foam::word const&, Foam::HashTable<Foam::autoPtr<Foam::GAMGAgglomerat ion> (*)(Foam::lduMatrix const&, Foam::dictionary const&), Foam::word, Foam::string::hash>* const&)':
(.text._ZN4Foam14dlLibraryTable4openIPNS_9HashTabl eIPFNS_7autoPtrINS_17GAMGAgglomerationEEERKNS_9ldu MatrixERKNS_10dictionaryEENS_4wordENS_6string4hash EEEEEbSB_RKSE_RKT_[bool Foam::dlLibraryTable::open<Foam::HashTable<Foam::a utoPtr<Foam::GAMGAgglomeration> (*)(Foam::lduMatrix const&, Foam::dictionary const&), Foam::word, Foam::string::hash>*>(Foam::dictionary const&, Foam::word const&, Foam::HashTable<Foam::autoPtr<Foam::GAMGAgglomerat ion> (*)(Foam::lduMatrix const&, Foam::dictionary const&), Foam::word, Foam::string::hash>* const&)]+0x125): undefined reference to `Foam::dlLibraryTable::open(Foam::fileName const&)'

/u/home/dhebert/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.a(libOpenFOAM.o): In function `Foam::IOerror::abort()':
(.text+0x1ded): undefined reference to `Foam::error::printStack(Foam::Ostream&)'

Hi, David

You should comment out just only the part of the code which is relevant with dynamic library functions.

Regards,
asaijo

Thanks Asaijo, this a big help!

I can compile fine, but when I go to compile an application (incompressible/pisoFoam in this case) I get warnings regarding getpwnam, getpwuid, gethostbyname...

/u/home/dhebert/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.a(libOSspecific.o): In function `Foam::home(Foam::word const&)':
(.text+0x4774): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/u/home/dhebert/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.a(libOSspecific.o): In function `Foam::home()':
(.text+0x46e8): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/u/home/dhebert/OpenFOAM/OpenFOAM-1.6/lib/linux64GccDPOpt/libOpenFOAM.a(libOSspecific.o): In function `Foam::ping(Foam::word const&, int, int)':
(.text+0x433b): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

The issue now is when I try to run the pisoFoam tutorial with kEpsilon turbulence model there is an error and pisoFoam is only finding laminar turbulence model...

Unknown turbulenceModel type RASModel

Valid turbulenceModel types are :

1
(
laminar
)


From function turbulenceModel::New(const volVectorField&, const surfaceScalarField&, transportModel&)
in file turbulenceModel.C at line 102.

FOAM exiting


Have you come across this problem? My first thought is there is a problem linking to the src/tubulenceModels library. Any ideas if this is the case?

Thanks,

David

David,

Yes I also came across similiar errors. As I wrote first, these
static-compilation processes are very partial and still have many
problems.

And I noticed you were using the 1.6 version of OpenFOAM. My
complication case is for version 1.5.x, so maybe there would be some
differences. (I haven't yet tried to compile the version 1.6.)

What I can say is that you should ensure all of object files (*.o) are
in $FOAM_LIBBIN.

Regards,
asaijo

Hi David
I have the same problem now this
+ aprun -n 64 ./pisoFoam -parallel
[2]
[2]
[2] Unknown turbulenceModel type LESModel

Valid turbulenceModel types are :

1
(
laminar
)
[2]
[2]
[2] From function turbulenceModel::New(const volVectorField&, const surfaceScalarField&, transportModel&)
[2] in file turbulenceModel.C at line 102.
[2]
FOAM parallel run exiting

Did u solve it
Thanks

errors with flex lexer
 

Dear all,

I followed the above advise (many thanks by the way!), but while compiling the the application I get lexer errors. Strangely enough, not for dnsFoam, laplacianFoam, potentialFoam, and scalarTransportFoam. Trouble starts when compiling combustion/PDRFoam. The system is Cray XT5 with gcc-4.4.3, flex 2.5.31, and zlib-1.2.3.

I also found this thread on yyFlexLexer issues, from where I changed a line in flex++ of wmake:
The error:
Would anyone have an idea? Many thanks in advance!

Question about number "4" I'm trying to apply this to 1.7.x
 

It looks there are "*.c" files as well as "*.C" that contain the dlfcn.h header file. I just wanted to double check we want to comment out any functions dealing with libdl. This could get ugly.

Hi James,

I tried the static build, but got stuck and built OpenFOAM dynamically. My notes live here, I hope it's of some help.

Hi Gijs

Thanks for the information, I am looking at your information now. I just wanted to let you know I had a problem with lexer also. I had a log of the build including errors and I traced it back to a missing header file located in "/app2/gnu/platforms/x86_64/include" what I did was add information to the Makefile located "~/OpenFOAM-1.7.x/wmake/Makefile" where the necessary header file could be found. Your location would be different.

here's a diff comparison

"
12:08:39$ diff Makefile_orig Makefile_new
74c74
< EXE_INC =
---
> EXE_INC =
142c142,143
< $(SYS_INC)
---
> $(SYS_INC) \
> -I/app2/gnu/platforms/x86_64/include
"

You also stated you did a dynamic build. On this cray in order to take advantage of parallel processing I think it's required I do a "Static" build.

Thanks for the response.

Jim