CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Installation (http://www.cfd-online.com/Forums/openfoam-installation/)
-   -   OpenFOAM build on Intel Xeon Phi (http://www.cfd-online.com/Forums/openfoam-installation/118552-openfoam-build-intel-xeon-phi.html)

asaijo May 29, 2013 17:40

OpenFOAM build on Intel Xeon Phi
 
1 Attachment(s)
Hi, foamers.

Today, I'd like to report my experience on building an OpenFOAM native MPI solver on Xeon Phi (Intel's co-processor and accelalator parts of the next world fastest supercomputer.)

Building OpenFOAM for Xeon Phi is bit tricky, but not difficult.

1. Make Zlib and Flex for Xeon Phi
2. Set OpenFOAM bashrc/setting.sh to linux64Icc and INTEL MPI
3. Make wmake only
4. Add -mmic to wmake/rules/linux64ICC/{c, c++}
5. Compile OpenFOAM libraries. If compilation fails, add Zlib/Flex path to Make/options
6. Compile solver applicaitions. You'll get many undefined function errors. Then use mpiicpc compiler (not icpc) and add -Wl,rpath-link flag for appropirate library's path by manual.

Finally, I got icoFoam for Xeon Phi. Perhaps there is an easier and canonical way.

I use Xeon Phi 5110P model. It has phisically 60 cores appeared as virtually 240 cores by Intel's Hyperthreading technology and 8GB GDDR5 memory.

The benchmark case is 3 dimensional caivty flow (modified from icoFoam's tutorial case) and I varied the MPI processes from 30 to 240.

Benchmark conditions:
Modified icoFoam tutorial cavity case (OpenFOAM-2.2.0)
3D cavity problem, deltaT = 1e-4, 50 time steps
Problem size: SMALL: (90x90x90) LEFT AXIS , LARGE: (175,175,175) RIGHT AXIS

CXXFLAGS = -mmic -O2 -no-prec-div
Domain decomposed by scotch

This is the result:
http://www.cfd-online.com/Forums/att...1&d=1369863305

You can compare figures with your own results.
Running an OpenFOAM solver on Xeon Phi without any source code modification is possible. But if you want the performance coressponding to the price, you should optimize your code.

Cheers.
-- asaijo

luchen2408 May 29, 2013 23:04

hello,asaijo, I am intrested in your computer, can you tell me how do you set the configuration for the computer? is it the same as in normal computer? have you conduct some simulations?what about the performace?
Besides, did you use NVIDIA Tesla K20 Architecture with CUDA-Option for simulation? did you compare the performance of NVIDIA Tesla K20 Architecture with CUDA-Option or the Intel Xeon Phi Technology? thanks

asaijo May 31, 2013 13:49

Hi, luchen2408.
The machine is normal and standard server, bought from local vendor.
Basic configurations has been set by the vendor. But I had to install a compiler, libraries and do extra tweaks for use.

I haven't used the K20X yet.
Regards.

luchen2408 June 2, 2013 08:54

hello,asaijo,thanks for your reply, can I take the Xeon Phi computer as a normal computer with many cores? and can I intall the software openfoam like an normal computer and use the same command in a normal multi-cores computer? thanks

kimbaol June 2, 2013 23:47

hi, asaijo
I have got an Intel Xeon Phi card and want to build an OpenFOAM native MPI solver on it. I just did as your instruction, and got some problems. Can you tell me more details? Such as:
1. Make Zlib and Flex for Xeon Phi ----------This step is install Zlib and Flex on the host PC, am i right?
2. Set OpenFOAM bashrc/setting.sh to linux64Icc and INTEL MPI --i have done this step
3. Make wmake only
4. Add -mmic to wmake/rules/linux64ICC/{c, c++} -----i have done this
5. Compile OpenFOAM libraries. If compilation fails, add Zlib/Flex path to Make/options -------in directory ~/OpenFOAM/src, i run ./Allwmake, and got some errors, how did you add Zlib/Flex path to Make/options? Is the file Make/option refers to the file ~/OpenFOAM/OpenFOAM-2.2.0/src/OpenFOAM/Make/options?

Thanks very much!

sail June 3, 2013 06:14

Dear Asaijo.

thanks for sharing your results. really interesting, and I'm impressed by them.

---


also, for the sake of comparison, are you using the standard fvscheme and fvsolution?

thanks in advance, i'd love to put my hands on one of those beauties...

kokizzu September 13, 2013 00:18

Is 5110p really worth it?
 
Asaijo, can you tell me how the performance using maximum processor core without xeon phi and with xeon phi, i want to know if Xeon Phi really worth it.

lordvon October 15, 2013 13:13

Update:
The 'ld' linker has a problem with the .o files being compiled with -mmic flag. If I get rid of the -mmic flag, the ld command does not give the error. Of course, this is not what we want to do. Do I need to set ld architecture somehow???


I ran into the following " could not read symbols: File in wrong format" problem while trying to compile using the steps in the first post (I am on step 5, and I did add zlib and flex to Make/options to get rid of those errors).
I went into the OSspecific/POSIX/ directory to compile it manually, but it made no difference.

ld -r -o /home/lordvon/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/lib/libOSspecific.o Make/linux64IccDPOpt/sigFpe.o Make/linux64IccDPOpt/sigSegv.o Make/linux64IccDPOpt/sigInt.o Make/linux64IccDPOpt/sigQuit.o Make/linux64IccDPOpt/sigStopAtWriteNow.o Make/linux64IccDPOpt/sigWriteNow.o Make/linux64IccDPOpt/regExp.o Make/linux64IccDPOpt/timer.o Make/linux64IccDPOpt/fileStat.o Make/linux64IccDPOpt/POSIX.o Make/linux64IccDPOpt/cpuTime.o Make/linux64IccDPOpt/clockTime.o Make/linux64IccDPOpt/memInfo.o Make/linux64IccDPOpt/fileMonitor.o Make/linux64IccDPOpt/printStack.o
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
Make/linux64IccDPOpt/sigFpe.o: could not read symbols: File in wrong format
make: *** [/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/lib/libOSspecific.o] Error 1

I am using Red Hat Enterprise Linux Server release 6.4 (Santiago)

Please help!

sigFpe.o compiled fine, by the way via:
SOURCE=signals/sigFpe.C ; icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,654,819,1125,1476,1505,1572 -O2 -no-prec-div -DNoRepository -IlnInclude -I. -I/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/src/OpenFOAM/lnInclude -I/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/src/OSspecific/POSIX/lnInclude -fPIC -mmic -c $SOURCE -o Make/linux64IccDPOpt/sigFpe.o

Dennis3 October 20, 2013 03:58

Cannot compile
 
Hi everyone,

First of all I would like to say thank you for sharing the information.

I have a problem compiling OpenFOAM to work on Xeon Phi. After that I added -mmic to the c and c++ file in wmake/rules/linux64Icc/, the compilation fails and I receive this error message everywhere in the log:

icpc: command line error: option '-xHost' not supported

If I remove -mmic everything works fine, anyone know that the problem is ?

kostyfisik October 24, 2013 13:20

cpu comparison
 
May be I am wrong, by I will try compare results. There is some data
http://www.hector.ac.uk/cse/distribu...am/node17.html and from topic starter

Xeon Phi for 50 time steps
grid size - 90^3 - 175^3
best time - 200s - 1500 s

Hectors 4 core of AMD 2.8GHz dual-core Opteron 5 time steps
grid size - 100^3 - 200^3
time - 795s - 8800 s

Hectors 1024 core of AMD 2.8GHz dual-core Opteron 40 time steps
grid size - 200^3
time - 1490 s

So, single Xeon Phi card for OpenFOAM is compatible with 1024 core cluster (for this benchmark) :)

kokizzu May 12, 2014 01:58

Hi, i have done step 1 and 2..
how to do the third step?
3. Make wmake only

kokizzu May 15, 2014 21:27

Hi, i got an error while doing these steps:

Code:

# CentOS 6.5
# OpenFoam 2.2.2

2. change etc/bashrc
export WM_COMPILER=Icc
export WM_MPLIB=INTEL_MPI

3. clean old compilation
source etc/bashrc
wcleanAll ; wcleanLnIncludeAll ; wcleanMachine

4a. add -mmic
edit wmake/rules/linux64ICC/c
cFLAGS      = $(GFLAGS) $(cWARN) $(cOPT) $(cDBUG) $(LIB_HEADER_DIRS) -fPIC -mmic

4b. add -mmic
edit wmake/rules/linux64ICC/c++
c++FLAGS    = $(GFLAGS) $(c++WARN) $(c++OPT) $(c++DBUG) $(ptFLAGS) $(LIB_HEADER_DIRS) -fPIC -mmic

4c. remove -xHost (conflict with -mmic)
vim wmake/rules/linux64Icc/c++Opt
c++OPT      = -O2 -no-prec-div

5. build for XeonPhi
./Allwmake

the errors:

Code:

icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,654,819,1125,1476,1505,1572 -O2 -no-prec-div  -DNoRepository -I/root/OpenFOAM/OpenFOAM-2.2.2/src/finiteVolume/lnInclude -ItractionDisplacementCorrectionStress -I../solidDisplacementFoam -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.2.2/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.2.2/src/OSspecific/POSIX/lnInclude  -fPIC -mmic -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64IccDPOpt/tractionDisplacementCorrectionFvPatchVectorField.o Make/linux64IccDPOpt/solidEquilibriumDisplacementFoam.o -L/root/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/lib \
            -lfiniteVolume -lOpenFOAM -ldl  -L/lib -lm -o /root/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/bin/solidEquilibriumDisplacementFoam
x86_64-k1om-linux-ld: cannot find -lfiniteVolume
x86_64-k1om-linux-ld: cannot find -lOpenFOAM

make[2]: *** [/root/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/bin/solidEquilibriumDisplacementFoam] Error 1
make[2]: Leaving directory `/root/OpenFOAM/OpenFOAM-2.2.2/applications/solvers/stressAnalysis/solidEquilibriumDisplacementFoam'
make[1]: *** [solidEquilibriumDisplacementFoam] Error 2
make[1]: Target `application' not remade because of errors.
make[1]: Leaving directory `/root/OpenFOAM/OpenFOAM-2.2.2/applications/solvers/stressAnalysis'
make: *** [stressAnalysis] Error 2
make: Target `application' not remade because of errors.


wyldckat May 17, 2014 14:06

Greetings to all!

@kokizzu: I don't have access to a Xeon Phi, so I have no experience with it.

Nonetheless, I have a lot of experience building OpenFOAM from source code, so here's what I can say regarding your latest post: when build OpenFOAM, usually the very first error that occurs is the one that tells the whole story. If you run Allwmake like this:
Code:

./Allwmake > log.make 2>&1
you'll get a complete log file named "log.make" with the output of running this script. Then you can more easily search for the first error or errors. If you provide the output for the first error or errors, it'll be much easier to help.

Best regards,
Bruno

kokizzu May 18, 2014 06:31

undefined reference to `__intel_sse4_atol'
 
Hi, I'm building OpenFOAM for Intel MIC (Xeon Phi), but i got this error:

Code:

icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -xHost -O2 -no-prec-div  -DNoRepository -I.. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/twoPhaseMixture/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/incompressible/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/interfaceProperties/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/turbulenceModels/incompressible/turbulenceModel -I/root/OpenFOAM/OpenFOAM-2.3.0/src/finiteVolume/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/meshTools/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/fvOptions/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/sampling/lnInclude -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude  -fPIC -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64IccDPOpt/porousInterFoam.o -L/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib \
            -ltwoPhaseMixture -linterfaceProperties -ltwoPhaseProperties -lincompressibleTransportModels -lincompressibleTurbulenceModel -lincompressibleRASModels -lincompressibleLESModels -lfiniteVolume -lmeshTools -lfvOptions -lsampling -lOpenFOAM -ldl  -L/lib -lm -o /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/bin/porousInterFoam
ld: warning: libimf.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtwoPhaseMixture.so, not found (try using -rpath or -rpath-link)
ld: warning: libsvml.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtwoPhaseMixture.so, not found (try using -rpath or -rpath-link)
ld: warning: libirng.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtwoPhaseMixture.so, not found (try using -rpath or -rpath-link)
ld: warning: libintlc.so.5, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtwoPhaseMixture.so, not found (try using -rpath or -rpath-link)
/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtriSurface.so: undefined reference to `__intel_sse4_atol'

My setup:
Code:

vim etc/bashrc
export WM_COMPILER=Icc
export WM_MPLIB=

vim wmake/rules/linuxIcc/c
cFLAGS      = $(GFLAGS) $(cWARN) $(cOPT) $(cDBUG) $(LIB_HEADER_DIRS) -KPIC -mmic

vim wmake/rules/linuxIcc/c++
c++FLAGS    = $(GFLAGS) $(c++WARN) $(c++OPT) $(c++DBUG) $(ptFLAGS) $(LIB_HEADER_DIRS) -KPIC -mmic

vim wmake/rules/linuxIcc/c++Opt
c++OPT      = -O2 -no-prec-div

source etc/bashrc
./Allwmake

Is it because I don't have Intel MKL installed?

the libimf.so was found here

Code:

/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/mic/libimf.so
/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/ia32/libimf.so
/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libimf.so
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/libimf.so
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/ia32/libimf.so
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/libimf.so

where should I put the -rpath or -rpath-link flags?

or should i run
Code:

/opt/intel/composer_xe_2013_sp1.2.144/bin/compilervars_arch.sh mic
first?

wyldckat May 18, 2014 13:20

Try running the last command like this instead:
Code:

source /opt/intel/composer_xe_2013_sp1.2.144/bin/compilervars_arch.sh mic
This way the environment variables are loaded onto the shell environment.

Beyond this, it does look like you're reaching the step #6 on the first post by asaijo:
Quote:

Originally Posted by asaijo (Post 430840)
6. Compile solver applicaitions. You'll get many undefined function errors. Then use mpiicpc compiler (not icpc) and add -Wl,rpath-link flag for appropirate library's path by manual.

Try these commands:
Code:

cd $FOAM_SOLVERS/multiphase/interFoam/porousInterFoam

mpiicpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -xHost -O2 -no-prec-div  -DNoRepository -I.. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/twoPhaseMixture/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/incompressible/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/interfaceProperties/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/turbulenceModels/incompressible/turbulenceModel -I/root/OpenFOAM/OpenFOAM-2.3.0/src/finiteVolume/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/meshTools/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/fvOptions/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/sampling/lnInclude -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude  -fPIC -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64IccDPOpt/porousInterFoam.o -L/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib \
-ltwoPhaseMixture -linterfaceProperties -ltwoPhaseProperties -lincompressibleTransportModels -lincompressibleTurbulenceModel -lincompressibleRASModels -lincompressibleLESModels -lfiniteVolume -lmeshTools -lfvOptions -lsampling -lOpenFOAM -ldl  -L/lib -lm -L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/ -o /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/bin/porousInterFoam

This essentially does what asaijo specified, namely:
  1. Go to the solver's source code folder.
  2. The last 2 lines before the error messages you got, have the complete link command that wmake uses.
  3. Use mpiicpc for building the solver, instead of icpc.
  4. I didn't use the "-Wl,rpath-link" option and used instead:
    Code:

    -L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/

kokizzu May 23, 2014 15:32

Hi, I've tried that, but still the same errors occured, when I added

Code:

-I/opt/intel/impi/4.1.3.049/mic/include/
after -mmic, it shows:

Code:

wclean mpi
wmake libso mpi
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file UIPread.C
could not open file mpi.h for source file UIPread.C due to No such file or directory
Making dependency list for source file UOPwrite.C
could not open file mpi.h for source file UOPwrite.C due to No such file or directory
Making dependency list for source file UPstream.C
could not open file mpi.h for source file UPstream.C due to No such file or directory
Making dependency list for source file PstreamGlobals.C
could not open file mpi.h for source file PstreamGlobals.C due to No such file or directory
SOURCE=UOPwrite.C ;  mpiicpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -xHost -O2 -no-prec-div  -DNoRepository  -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude  -fPIC -c $SOURCE -o Make/linux64IccDPOptINTEL_MPI/UOPwrite.o
SOURCE=UIPread.C ;  mpiicpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -xHost -O2 -no-prec-div  -DNoRepository  -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude  -fPIC -c $SOURCE -o Make/linux64IccDPOptINTEL_MPI/UIPread.o
SOURCE=UPstream.C ;  mpiicpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -xHost -O2 -no-prec-div  -DNoRepository  -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude  -fPIC -c $SOURCE -o Make/linux64IccDPOptINTEL_MPI/UPstream.o
SOURCE=PstreamGlobals.C ;  mpiicpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -xHost -O2 -no-prec-div  -DNoRepository  -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude  -fPIC -c $SOURCE -o Make/linux64IccDPOptINTEL_MPI/PstreamGlobals.o
ld: skipping incompatible /opt/intel/impi/4.1.3.049/mic/lib/libmpigc4.so when searching for -lmpigc4
ld: skipping incompatible /opt/intel/impi/4.1.3.049/mic/lib/libmpigc4.a when searching for -lmpigc4
ld: cannot find -lmpigc4
make: *** [/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/dummy/libPstream.so] Error 1

--------------
If I tried without using MPI, it shows something like this:
Code:

ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
Make/linux64IccDPOpt/sigFpe.o: could not read symbols: File in wrong format
make: *** [/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o] Error 1

--------------
Hi, thanks for replying, here's my full configuration for the last error:

Code:

yum install boost-devel

cd ~/OpenFOAM/OpenFOAM-2.3.0

# edit etc/bashrc
export WM_COMPILER=Icc
export WM_MPLIB=

source etc/bashrc

# edit wmake/rules/linux*Icc/c
cFLAGS      = $(GFLAGS) $(cWARN) $(cOPT) $(cDBUG) $(LIB_HEADER_DIRS) -fPIC -mmic -I/opt/intel/impi/4.1.3.049/mic/include/ -I/opt/crtdc/micgnu/include/

# edit wmake/rules/linux*Icc/c++
c++FLAGS    = $(GFLAGS) $(c++WARN) $(c++OPT) $(c++DBUG) $(ptFLAGS) $(LIB_HEADER_DIRS) -fPIC -mmic -I/opt/intel/impi/4.1.3.049/mic/include/ -I/opt/crtdc/micgnu/include/

# edit wmake/rules/linux64Icc/c++Opt
c++OPT      = -O2 -no-prec-div

# edit wmake/rules/linux64Icc/c++
CC          = mpiicpc -std=c++0x

# edit wmake/rules/linuxIA64Icc/c++
CC          = mpiicpc

# edit wmake/rules/linuxIcc/c++
CC          = mpiicpc -std=c++0x

# edit wmake/rules/*Icc/c
# edit wmake/rules/*Icc/c++
# add on LINK_LIBS
-L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/ -L/opt/crtdc/micgnu/lib/

source /opt/intel/composer_xe_2013_sp1.2.144/bin/compilervars_arch.sh mic

source /opt/intel/impi/4.1.3/mic/bin/mpivars.sh

./Allwmake > errors.log 2>&1 &
tail -f errors.log


kokizzu May 24, 2014 04:05

Quote:

Originally Posted by lordvon (Post 457120)
sigFpe.o compiled fine, by the way via:
SOURCE=signals/sigFpe.C ; icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,654,819,1125,1476,1505,1572 -O2 -no-prec-div -DNoRepository -IlnInclude -I. -I/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/src/OpenFOAM/lnInclude -I/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/src/OSspecific/POSIX/lnInclude -fPIC -mmic -c $SOURCE -o Make/linux64IccDPOpt/sigFpe.o

Hi, i got exactly same problem, and I tried using this command:

Code:

cd ~/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX
mkdir Make/linux64IccDPOpt/
SOURCE=signals/sigFpe.C ; icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,654,819,1125,1476,1505,1572 -O2 -no-prec-div -DNoRepository -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/containers/Lists/UList/ -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/containers/Lists/UList/ -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/primitives/bools/bool/ -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/primitives/pTraits  -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/primitives/ints/label/ -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/primitives/direction/ -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/primitives/chars/char/ -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/primitives/ints/int/ -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/primitives/strings/word/ -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/primitives/strings/string/ -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/primitives/hashes/Hasher/ -fPIC -mmic -c $SOURCE -o Make/linux64IccDPOpt/sigFpe.o -I../../OpenFOAM/primitives/ints/uLabel -I../../OpenFOAM/primitives/ints/uint -I../../OpenFOAM/db/error -I../../OpenFOAM/db/IOstreams/Sstreams -I../../OpenFOAM/db/IOstreams/IOstreams -I../../OpenFOAM/primitives/Scalar/scalar -I../../OpenFOAM/primitives/Scalar/floatScalar -I../../OpenFOAM/primitives/Scalar -I../../OpenFOAM/primitives/VectorSpace -I../../OpenFOAM/primitives/Scalar/doubleScalar -I../../OpenFOAM/primitives/strings/fileName -I../../OpenFOAM/primitives/strings/keyType -I../../OpenFOAM/primitives/Swap -I../../OpenFOAM/containers/Lists/List -I../../OpenFOAM/primitives/contiguous -I../../OpenFOAM/db/IOstreams/token -I../../OpenFOAM/memory/refCount -I../../OpenFOAM/db/typeInfo -I../../OpenFOAM/global/debug -I../../OpenFOAM/db/runTimeSelection/construction -I../../OpenFOAM/memory/autoPtr -I../../OpenFOAM/containers/HashTables/HashTable -I../../OpenFOAM/memory/Xfer -I../../OpenFOAM/containers/Lists/FixedList -I../../OpenFOAM/primitives/hashes/Hash -I../../OpenFOAM/primitives/strings/wordRe -I../../OpenFOAM/containers/LinkedLists/user -I../../OpenFOAM/containers/LinkedLists/accessTypes/LList -I../../OpenFOAM/containers/LinkedLists/linkTypes/SLListBase -I../../OpenFOAM/containers/Lists/PtrList -I../../OpenFOAM/memory/tmp -I../../OpenFOAM/containers/LinkedLists/accessTypes/LPtrList -I../../OpenFOAM/containers/Lists/IndirectList -I../../OpenFOAM/containers/Lists/UIndirectList -I../../OpenFOAM/containers/Lists/BiIndirectList -I../../OpenFOAM/global/JobInfo -I../../OpenFOAM/db/dictionary -I../../OpenFOAM/db/dictionary/entry -I../../OpenFOAM/containers/LinkedLists/accessTypes/ILList -I../../OpenFOAM/containers/LinkedLists/accessTypes/UILList -I../../OpenFOAM/containers/LinkedLists/linkTypes/DLListBase -I../../OpenFOAM/db/IOstreams/Tstreams -I../../OpenFOAM/primitives/strings/lists -I../../OpenFOAM/db/dictionary/primitiveEntry -I../../OpenFOAM/db/IOstreams/StringStreams -I../../OSspecific/POSIX/cpuTime -I../../OpenFOAM/include -I../../OpenFOAM/primitives/ints/long -I../../OpenFOAM/db/IOstreams

but got the exact same error:
Code:

ld -r -o /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o Make/linux64IccDPOpt/sigFpe.o Make/linux64IccDPOpt/sigSegv.o Make/linux64IccDPOpt/sigInt.o Make/linux64IccDPOpt/sigQuit.o Make/linux64IccDPOpt/sigStopAtWriteNow.o Make/linux64IccDPOpt/sigWriteNow.o Make/linux64IccDPOpt/regExp.o Make/linux64IccDPOpt/timer.o Make/linux64IccDPOpt/fileStat.o Make/linux64IccDPOpt/POSIX.o Make/linux64IccDPOpt/cpuTime.o Make/linux64IccDPOpt/clockTime.o Make/linux64IccDPOpt/memInfo.o Make/linux64IccDPOpt/fileMonitor.o Make/linux64IccDPOpt/printStack.o
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
Make/linux64IccDPOpt/sigFpe.o: could not read symbols: File in wrong format
make: *** [/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o] Error 1

that seems overwritten by this command:
Code:

SOURCE=signals/sigFpe.C ;  mpiicpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -O2 -no-prec-div  -DNoRepository -DFOAM_USE_INOTIFY -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude  -fPIC -mmic -I/opt/intel/impi/4.1.3.049/mic/include/ -I/opt/crtdc/micgnu/include/ -c $SOURCE -o Make/linux64IccDPOpt/sigFpe.o

kokizzu May 28, 2014 00:22

Since the "-L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/" doesn't work at all..

we're adding all ".so" files on "cFLAGS" part on the "c" and "c++" file, for example:

Quote:

mpiicpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -O2 -no-prec-div -DNoRepository -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude -fPIC -mmic -I/opt/intel/impi/4.1.3.049/mic/include/ -I/opt/crtdc/micgnu/include/ -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64IccDPOpt/expandDictionary.o -L/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib -lOpenFOAM -ldl -L/lib -L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic -L/opt/crtdc/micgnu/lib/ -lm -o /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/bin/expandDictionary -Wl,/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/libimf.so,/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/libsvml.so,/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/libirng.so,/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/libintlc.so.5,/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/intel64/libPstream.so,/opt/crtdc/micgnu/lib/libz.so\
and the error on the ld part was solved by make a symbolic link "/opt/mpss/3.2.1/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-ld" as "/usr/bin/ld"

some warning disappear, but some warning still shows up:

Code:

x86_64-k1om-linux-ld: warning: libirng.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/intel64/libPstream.so, not found (try using -rpath or -rpath-link)
x86_64-k1om-linux-ld: warning: libdynamicMesh.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libautoMesh.so, not found (try using -rpath or -rpath-link)
x86_64-k1om-linux-ld: warning: liblagrangian.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libautoMesh.so, not found (try using -rpath or -rpath-link)
x86_64-k1om-linux-ld: warning: libfileFormats.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libautoMesh.so, not found (try using -rpath or -rpath-link)
...
x86_64-k1om-linux-ld: warning: libdsmc.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libutilityFunctionObjects.so, not found (try using -rpath or -rpath-link)

*snip* very long long mpiipcp command *snip*

/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/postCalc.o: In function `Foam::tryCalc(Foam::argList const&, Foam::Time const&, Foam::fvMesh const&)':
(.text+0x1520): undefined reference to `Foam::calc(Foam::argList const&, Foam::Time const&, Foam::fvMesh const&)'

pastebin for the error file: http://pastie.org/9227330

any possible solution for solving those errors?

kokizzu May 28, 2014 02:51

at last.. (/^__^)/

this one works:

wmake/rules/linux64Icc/c
Code:

.SUFFIXES: .c .h                                                                                                                                                     
                                                                                                                                                                     
cWARN        =                                                                                                                                                       
                                                                                                                                                                     
cc          = icc                                                                                                                                                     
                                                                                                                                                                     
include $(RULES)/c$(WM_COMPILE_OPTION)                                                                                                                               
                                                                                                                                                                     
cFLAGS      = -I/opt/intel/impi/4.1.3.049/mic/include/ -I/opt/crtdc/micgnu/include/ $(GFLAGS) $(cWARN) $(cOPT) $(cDBUG) $(LIB_HEADER_DIRS) -fPIC \                   
-mmic -I/usr/include -Wl,/opt/crtdc/micgnu/lib/libz.so                                                                                                               
                                                                                                                                                                     
ctoo        = $(WM_SCHEDULER) $(cc) $(cFLAGS) -c $$SOURCE -o $@                                                                                                       
                                                                                                                                                                     
LINK_LIBS  = -Wl,-rpath-link,/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/,-rpath-link,/opt/crtdc/micgnu/lib,-rpath-link,/root/OpenFOAM/OpenFOAM-2.3.0/platf
orms/linux64IccDPOpt/lib/intel64/,-rpath-link,/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/ -L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/ -L
/opt/crtdc/micgnu/lib/ $(cDBUG) -L$(IA32ROOT)/lib                                                                                                                     

LINKLIBSO  = $(cc) $(cFLAGS) -shared
LINKEXE    = $(cc) $(cFLAGS) -Xlinker --add-needed -Xlinker -z -Xlinker nodefs

wmake/rules/linux64Icc/c++
Code:

.SUFFIXES: .C .cxx .cc .cpp                                                                                                                                           
                                                                                                                                                                     
c++WARN    = -wd327,525,654,819,1125,1476,1505,1572                                                                                                                 
                                                                                                                                                                     
CC          = mpiicpc -std=c++0x                                                                                                                                     
                                                                                                                                                                     
include $(RULES)/c++$(WM_COMPILE_OPTION)                                                                                                                             
                                                                                                                                                                     
ptFLAGS    = -DNoRepository                                                                                                                                         
                                                                                                                                                                     
c++FLAGS    = -I/opt/intel/impi/4.1.3.049/mic/include/ -I/opt/crtdc/micgnu/include/ $(GFLAGS) $(c++WARN) $(c++OPT) $(c++DBUG) $(ptFLAGS) $(LIB_HEADER_DIRS) -fPIC \   
-mmic -I/usr/include -Wl,/opt/crtdc/micgnu/lib/libz.so                                                                                                               
                                                                                                                                                                     
Ctoo        = $(WM_SCHEDULER) $(CC) $(c++FLAGS) -c $$SOURCE -o $@                                                                                                     
cxxtoo      = $(Ctoo)                                                                                                                                                 
cctoo      = $(Ctoo)                                                                                                                                                 
cpptoo      = $(Ctoo)                                                                                                                                                 
                                                                                                                                                                     
LINK_LIBS  = -Wl,-rpath-link,/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/,-rpath-link,/opt/crtdc/micgnu/lib,-rpath-link,/root/OpenFOAM/OpenFOAM-2.3.0/platf
orms/linux64IccDPOpt/lib/intel64/,-rpath-link,/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/ -L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/ -L
/opt/crtdc/micgnu/lib/ $(c++DBUG) -L$(IA64ROOT)/lib                                                                                                                   

LINKLIBSO  = $(CC) $(c++FLAGS) -shared -Xlinker --add-needed -Xlinker --no-as-needed
LINKEXE    = $(CC) $(c++FLAGS) -Xlinker --add-needed -Xlinker --no-as-needed


manredd June 8, 2014 23:31

OpenFOAM2.3.0 on Xeon Phi with OpenMPI1.6.5 (could not read symbols)
 
Dear All,

I would be very grateful if you tell me how you specifically fixed the following problem:

wmake libso mpi
'/home/manumachu/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/openmpi-1.6.5/libPstream.so' is up to date.
+ OSspecific/POSIX/Allwmake
Found <sys/inotify.h> -- enabling inotify for file monitoring.
ld -r -o /home/manumachu/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o Make/linux64IccDPOpt/sigFpe.o Make/linux64IccDPOpt/sigSegv.o Make/linux64IccDPOpt/sigInt.o Make/linux64IccDPOpt/sigQuit.o Make/linux64IccDPOpt/sigStopAtWriteNow.o Make/linux64IccDPOpt/sigWriteNow.o Make/linux64IccDPOpt/regExp.o Make/linux64IccDPOpt/timer.o Make/linux64IccDPOpt/fileStat.o Make/linux64IccDPOpt/POSIX.o Make/linux64IccDPOpt/cpuTime.o Make/linux64IccDPOpt/clockTime.o Make/linux64IccDPOpt/memInfo.o Make/linux64IccDPOpt/fileMonitor.o Make/linux64IccDPOpt/printStack.o
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
Make/linux64IccDPOpt/sigFpe.o: could not read symbols: File in wrong format
make: *** [/home/manumachu/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o] Error 1

Best Regards
Manredd


All times are GMT -4. The time now is 08:41.