OpenFOAM build on Intel Xeon Phi
 

Hi, foamers.

Today, I'd like to report my experience on building an OpenFOAM native MPI solver on Xeon Phi (Intel's co-processor and accelalator parts of the next world fastest supercomputer.)

Building OpenFOAM for Xeon Phi is bit tricky, but not difficult.

1. Make Zlib and Flex for Xeon Phi
2. Set OpenFOAM bashrc/setting.sh to linux64Icc and INTEL MPI
3. Make wmake only
4. Add -mmic to wmake/rules/linux64ICC/{c, c++}
5. Compile OpenFOAM libraries. If compilation fails, add Zlib/Flex path to Make/options
6. Compile solver applicaitions. You'll get many undefined function errors. Then use mpiicpc compiler (not icpc) and add -Wl,rpath-link flag for appropirate library's path by manual.

Finally, I got icoFoam for Xeon Phi. Perhaps there is an easier and canonical way.

I use Xeon Phi 5110P model. It has phisically 60 cores appeared as virtually 240 cores by Intel's Hyperthreading technology and 8GB GDDR5 memory.

The benchmark case is 3 dimensional caivty flow (modified from icoFoam's tutorial case) and I varied the MPI processes from 30 to 240.

Benchmark conditions:
Modified icoFoam tutorial cavity case (OpenFOAM-2.2.0)
3D cavity problem, deltaT = 1e-4, 50 time steps
Problem size: SMALL: (90x90x90) LEFT AXIS , LARGE: (175,175,175) RIGHT AXIS

CXXFLAGS = -mmic -O2 -no-prec-div
Domain decomposed by scotch

This is the result:
http://www.cfd-online.com/Forums/att...1&d=1369863305

You can compare figures with your own results.
Running an OpenFOAM solver on Xeon Phi without any source code modification is possible. But if you want the performance coressponding to the price, you should optimize your code.

Cheers.
-- asaijo

hello,asaijo, I am intrested in your computer, can you tell me how do you set the configuration for the computer? is it the same as in normal computer? have you conduct some simulations?what about the performace?
Besides, did you use NVIDIA Tesla K20 Architecture with CUDA-Option for simulation? did you compare the performance of NVIDIA Tesla K20 Architecture with CUDA-Option or the Intel Xeon Phi Technology? thanks

Hi, luchen2408.
The machine is normal and standard server, bought from local vendor.
Basic configurations has been set by the vendor. But I had to install a compiler, libraries and do extra tweaks for use.

I haven't used the K20X yet.
Regards.

hello,asaijo,thanks for your reply, can I take the Xeon Phi computer as a normal computer with many cores? and can I intall the software openfoam like an normal computer and use the same command in a normal multi-cores computer? thanks

hi, asaijo
I have got an Intel Xeon Phi card and want to build an OpenFOAM native MPI solver on it. I just did as your instruction, and got some problems. Can you tell me more details? Such as:
1. Make Zlib and Flex for Xeon Phi ----------This step is install Zlib and Flex on the host PC, am i right?
2. Set OpenFOAM bashrc/setting.sh to linux64Icc and INTEL MPI --i have done this step
3. Make wmake only
4. Add -mmic to wmake/rules/linux64ICC/{c, c++} -----i have done this
5. Compile OpenFOAM libraries. If compilation fails, add Zlib/Flex path to Make/options -------in directory ~/OpenFOAM/src, i run ./Allwmake, and got some errors, how did you add Zlib/Flex path to Make/options? Is the file Make/option refers to the file ~/OpenFOAM/OpenFOAM-2.2.0/src/OpenFOAM/Make/options?

Thanks very much!

Dear Asaijo.

thanks for sharing your results. really interesting, and I'm impressed by them.

---


also, for the sake of comparison, are you using the standard fvscheme and fvsolution?

thanks in advance, i'd love to put my hands on one of those beauties...

Is 5110p really worth it?
 

Asaijo, can you tell me how the performance using maximum processor core without xeon phi and with xeon phi, i want to know if Xeon Phi really worth it.

Update:
The 'ld' linker has a problem with the .o files being compiled with -mmic flag. If I get rid of the -mmic flag, the ld command does not give the error. Of course, this is not what we want to do. Do I need to set ld architecture somehow???


I ran into the following " could not read symbols: File in wrong format" problem while trying to compile using the steps in the first post (I am on step 5, and I did add zlib and flex to Make/options to get rid of those errors).
I went into the OSspecific/POSIX/ directory to compile it manually, but it made no difference.

ld -r -o /home/lordvon/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/lib/libOSspecific.o Make/linux64IccDPOpt/sigFpe.o Make/linux64IccDPOpt/sigSegv.o Make/linux64IccDPOpt/sigInt.o Make/linux64IccDPOpt/sigQuit.o Make/linux64IccDPOpt/sigStopAtWriteNow.o Make/linux64IccDPOpt/sigWriteNow.o Make/linux64IccDPOpt/regExp.o Make/linux64IccDPOpt/timer.o Make/linux64IccDPOpt/fileStat.o Make/linux64IccDPOpt/POSIX.o Make/linux64IccDPOpt/cpuTime.o Make/linux64IccDPOpt/clockTime.o Make/linux64IccDPOpt/memInfo.o Make/linux64IccDPOpt/fileMonitor.o Make/linux64IccDPOpt/printStack.o
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
Make/linux64IccDPOpt/sigFpe.o: could not read symbols: File in wrong format
make: *** [/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/lib/libOSspecific.o] Error 1

I am using Red Hat Enterprise Linux Server release 6.4 (Santiago)

Please help!

sigFpe.o compiled fine, by the way via:
SOURCE=signals/sigFpe.C ; icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,654,819,1125,1476,1505,1572 -O2 -no-prec-div -DNoRepository -IlnInclude -I. -I/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/src/OpenFOAM/lnInclude -I/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/src/OSspecific/POSIX/lnInclude -fPIC -mmic -c $SOURCE -o Make/linux64IccDPOpt/sigFpe.o

Cannot compile
 

Hi everyone,

First of all I would like to say thank you for sharing the information.

I have a problem compiling OpenFOAM to work on Xeon Phi. After that I added -mmic to the c and c++ file in wmake/rules/linux64Icc/, the compilation fails and I receive this error message everywhere in the log:

icpc: command line error: option '-xHost' not supported

If I remove -mmic everything works fine, anyone know that the problem is ?

cpu comparison
 

May be I am wrong, by I will try compare results. There is some data
http://www.hector.ac.uk/cse/distribu...am/node17.html and from topic starter

Xeon Phi for 50 time steps
grid size - 90^3 - 175^3
best time - 200s - 1500 s

Hectors 4 core of AMD 2.8GHz dual-core Opteron 5 time steps
grid size - 100^3 - 200^3
time - 795s - 8800 s

Hectors 1024 core of AMD 2.8GHz dual-core Opteron 40 time steps
grid size - 200^3
time - 1490 s

So, single Xeon Phi card for OpenFOAM is compatible with 1024 core cluster (for this benchmark) :)

Hi, i have done step 1 and 2..
how to do the third step?
3. Make wmake only

Hi, i got an error while doing these steps:

the errors:

Greetings to all!

@kokizzu: I don't have access to a Xeon Phi, so I have no experience with it.

Nonetheless, I have a lot of experience building OpenFOAM from source code, so here's what I can say regarding your latest post: when build OpenFOAM, usually the very first error that occurs is the one that tells the whole story. If you run Allwmake like this:
you'll get a complete log file named "log.make" with the output of running this script. Then you can more easily search for the first error or errors. If you provide the output for the first error or errors, it'll be much easier to help.

Best regards,
Bruno

undefined reference to `__intel_sse4_atol'
 

Hi, I'm building OpenFOAM for Intel MIC (Xeon Phi), but i got this error:

My setup:
Is it because I don't have Intel MKL installed?

the libimf.so was found here

where should I put the -rpath or -rpath-link flags?

or should i run first?

Try running the last command like this instead:
This way the environment variables are loaded onto the shell environment.

Beyond this, it does look like you're reaching the step #6 on the first post by asaijo:
Try these commands:
This essentially does what asaijo specified, namely:

1. Go to the solver's source code folder.
2. The last 2 lines before the error messages you got, have the complete link command that wmake uses.
3. Use mpiicpc for building the solver, instead of icpc.
4. I didn't use the "-Wl,rpath-link" option and used instead:
   
   Code:

   ---

   -L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/

   ---

Hi, I've tried that, but still the same errors occured, when I added

after -mmic, it shows:

--------------
If I tried without using MPI, it shows something like this:
--------------
Hi, thanks for replying, here's my full configuration for the last error:

Hi, i got exactly same problem, and I tried using this command:

but got the exact same error:
that seems overwritten by this command:

Since the "-L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/" doesn't work at all..

we're adding all ".so" files on "cFLAGS" part on the "c" and "c++" file, for example:

and the error on the ld part was solved by make a symbolic link "/opt/mpss/3.2.1/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-ld" as "/usr/bin/ld"

some warning disappear, but some warning still shows up:

pastebin for the error file: http://pastie.org/9227330

any possible solution for solving those errors?

at last.. (/^__^)/

this one works:

wmake/rules/linux64Icc/c
wmake/rules/linux64Icc/c++

OpenFOAM2.3.0 on Xeon Phi with OpenMPI1.6.5 (could not read symbols)
 

Dear All,

I would be very grateful if you tell me how you specifically fixed the following problem:

wmake libso mpi
'/home/manumachu/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/openmpi-1.6.5/libPstream.so' is up to date.
+ OSspecific/POSIX/Allwmake
Found <sys/inotify.h> -- enabling inotify for file monitoring.
ld -r -o /home/manumachu/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o Make/linux64IccDPOpt/sigFpe.o Make/linux64IccDPOpt/sigSegv.o Make/linux64IccDPOpt/sigInt.o Make/linux64IccDPOpt/sigQuit.o Make/linux64IccDPOpt/sigStopAtWriteNow.o Make/linux64IccDPOpt/sigWriteNow.o Make/linux64IccDPOpt/regExp.o Make/linux64IccDPOpt/timer.o Make/linux64IccDPOpt/fileStat.o Make/linux64IccDPOpt/POSIX.o Make/linux64IccDPOpt/cpuTime.o Make/linux64IccDPOpt/clockTime.o Make/linux64IccDPOpt/memInfo.o Make/linux64IccDPOpt/fileMonitor.o Make/linux64IccDPOpt/printStack.o
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181)
Make/linux64IccDPOpt/sigFpe.o: could not read symbols: File in wrong format
make: *** [/home/manumachu/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o] Error 1

Best Regards
Manredd