OpenFOAM build on Intel Xeon Phi
1 Attachment(s)
Hi, foamers.
Today, I'd like to report my experience on building an OpenFOAM native MPI solver on Xeon Phi (Intel's co-processor and accelalator parts of the next world fastest supercomputer.) Building OpenFOAM for Xeon Phi is bit tricky, but not difficult. 1. Make Zlib and Flex for Xeon Phi 2. Set OpenFOAM bashrc/setting.sh to linux64Icc and INTEL MPI 3. Make wmake only 4. Add -mmic to wmake/rules/linux64ICC/{c, c++} 5. Compile OpenFOAM libraries. If compilation fails, add Zlib/Flex path to Make/options 6. Compile solver applicaitions. You'll get many undefined function errors. Then use mpiicpc compiler (not icpc) and add -Wl,rpath-link flag for appropirate library's path by manual. Finally, I got icoFoam for Xeon Phi. Perhaps there is an easier and canonical way. I use Xeon Phi 5110P model. It has phisically 60 cores appeared as virtually 240 cores by Intel's Hyperthreading technology and 8GB GDDR5 memory. The benchmark case is 3 dimensional caivty flow (modified from icoFoam's tutorial case) and I varied the MPI processes from 30 to 240. Benchmark conditions: Modified icoFoam tutorial cavity case (OpenFOAM-2.2.0) 3D cavity problem, deltaT = 1e-4, 50 time steps Problem size: SMALL: (90x90x90) LEFT AXIS , LARGE: (175,175,175) RIGHT AXIS CXXFLAGS = -mmic -O2 -no-prec-div Domain decomposed by scotch This is the result: http://www.cfd-online.com/Forums/att...1&d=1369863305 You can compare figures with your own results. Running an OpenFOAM solver on Xeon Phi without any source code modification is possible. But if you want the performance coressponding to the price, you should optimize your code. Cheers. -- asaijo |
hello,asaijo, I am intrested in your computer, can you tell me how do you set the configuration for the computer? is it the same as in normal computer? have you conduct some simulations?what about the performace?
Besides, did you use NVIDIA Tesla K20 Architecture with CUDA-Option for simulation? did you compare the performance of NVIDIA Tesla K20 Architecture with CUDA-Option or the Intel Xeon Phi Technology? thanks |
Hi, luchen2408.
The machine is normal and standard server, bought from local vendor. Basic configurations has been set by the vendor. But I had to install a compiler, libraries and do extra tweaks for use. I haven't used the K20X yet. Regards. |
hello,asaijo,thanks for your reply, can I take the Xeon Phi computer as a normal computer with many cores? and can I intall the software openfoam like an normal computer and use the same command in a normal multi-cores computer? thanks
|
hi, asaijo
I have got an Intel Xeon Phi card and want to build an OpenFOAM native MPI solver on it. I just did as your instruction, and got some problems. Can you tell me more details? Such as: 1. Make Zlib and Flex for Xeon Phi ----------This step is install Zlib and Flex on the host PC, am i right? 2. Set OpenFOAM bashrc/setting.sh to linux64Icc and INTEL MPI --i have done this step 3. Make wmake only 4. Add -mmic to wmake/rules/linux64ICC/{c, c++} -----i have done this 5. Compile OpenFOAM libraries. If compilation fails, add Zlib/Flex path to Make/options -------in directory ~/OpenFOAM/src, i run ./Allwmake, and got some errors, how did you add Zlib/Flex path to Make/options? Is the file Make/option refers to the file ~/OpenFOAM/OpenFOAM-2.2.0/src/OpenFOAM/Make/options? Thanks very much! |
Dear Asaijo.
thanks for sharing your results. really interesting, and I'm impressed by them. --- also, for the sake of comparison, are you using the standard fvscheme and fvsolution? thanks in advance, i'd love to put my hands on one of those beauties... |
Is 5110p really worth it?
Asaijo, can you tell me how the performance using maximum processor core without xeon phi and with xeon phi, i want to know if Xeon Phi really worth it.
|
Update:
The 'ld' linker has a problem with the .o files being compiled with -mmic flag. If I get rid of the -mmic flag, the ld command does not give the error. Of course, this is not what we want to do. Do I need to set ld architecture somehow??? I ran into the following " could not read symbols: File in wrong format" problem while trying to compile using the steps in the first post (I am on step 5, and I did add zlib and flex to Make/options to get rid of those errors). I went into the OSspecific/POSIX/ directory to compile it manually, but it made no difference. ld -r -o /home/lordvon/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/lib/libOSspecific.o Make/linux64IccDPOpt/sigFpe.o Make/linux64IccDPOpt/sigSegv.o Make/linux64IccDPOpt/sigInt.o Make/linux64IccDPOpt/sigQuit.o Make/linux64IccDPOpt/sigStopAtWriteNow.o Make/linux64IccDPOpt/sigWriteNow.o Make/linux64IccDPOpt/regExp.o Make/linux64IccDPOpt/timer.o Make/linux64IccDPOpt/fileStat.o Make/linux64IccDPOpt/POSIX.o Make/linux64IccDPOpt/cpuTime.o Make/linux64IccDPOpt/clockTime.o Make/linux64IccDPOpt/memInfo.o Make/linux64IccDPOpt/fileMonitor.o Make/linux64IccDPOpt/printStack.o ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) Make/linux64IccDPOpt/sigFpe.o: could not read symbols: File in wrong format make: *** [/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/lib/libOSspecific.o] Error 1 I am using Red Hat Enterprise Linux Server release 6.4 (Santiago) Please help! sigFpe.o compiled fine, by the way via: SOURCE=signals/sigFpe.C ; icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,654,819,1125,1476,1505,1572 -O2 -no-prec-div -DNoRepository -IlnInclude -I. -I/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/src/OpenFOAM/lnInclude -I/home/lordvon/OpenFOAM/OpenFOAM-2.2.2/src/OSspecific/POSIX/lnInclude -fPIC -mmic -c $SOURCE -o Make/linux64IccDPOpt/sigFpe.o |
Cannot compile
Hi everyone,
First of all I would like to say thank you for sharing the information. I have a problem compiling OpenFOAM to work on Xeon Phi. After that I added -mmic to the c and c++ file in wmake/rules/linux64Icc/, the compilation fails and I receive this error message everywhere in the log: icpc: command line error: option '-xHost' not supported If I remove -mmic everything works fine, anyone know that the problem is ? |
cpu comparison
May be I am wrong, by I will try compare results. There is some data
http://www.hector.ac.uk/cse/distribu...am/node17.html and from topic starter Xeon Phi for 50 time steps grid size - 90^3 - 175^3 best time - 200s - 1500 s Hectors 4 core of AMD 2.8GHz dual-core Opteron 5 time steps grid size - 100^3 - 200^3 time - 795s - 8800 s Hectors 1024 core of AMD 2.8GHz dual-core Opteron 40 time steps grid size - 200^3 time - 1490 s So, single Xeon Phi card for OpenFOAM is compatible with 1024 core cluster (for this benchmark) :) |
Hi, i have done step 1 and 2..
how to do the third step? 3. Make wmake only |
Hi, i got an error while doing these steps:
Code:
# CentOS 6.5 Code:
icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,654,819,1125,1476,1505,1572 -O2 -no-prec-div -DNoRepository -I/root/OpenFOAM/OpenFOAM-2.2.2/src/finiteVolume/lnInclude -ItractionDisplacementCorrectionStress -I../solidDisplacementFoam -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.2.2/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.2.2/src/OSspecific/POSIX/lnInclude -fPIC -mmic -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64IccDPOpt/tractionDisplacementCorrectionFvPatchVectorField.o Make/linux64IccDPOpt/solidEquilibriumDisplacementFoam.o -L/root/OpenFOAM/OpenFOAM-2.2.2/platforms/linux64IccDPOpt/lib \ |
Greetings to all!
@kokizzu: I don't have access to a Xeon Phi, so I have no experience with it. Nonetheless, I have a lot of experience building OpenFOAM from source code, so here's what I can say regarding your latest post: when build OpenFOAM, usually the very first error that occurs is the one that tells the whole story. If you run Allwmake like this: Code:
./Allwmake > log.make 2>&1 Best regards, Bruno |
undefined reference to `__intel_sse4_atol'
Hi, I'm building OpenFOAM for Intel MIC (Xeon Phi), but i got this error:
Code:
icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -xHost -O2 -no-prec-div -DNoRepository -I.. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/twoPhaseMixture/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/incompressible/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/interfaceProperties/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/turbulenceModels/incompressible/turbulenceModel -I/root/OpenFOAM/OpenFOAM-2.3.0/src/finiteVolume/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/meshTools/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/fvOptions/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/sampling/lnInclude -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude -fPIC -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64IccDPOpt/porousInterFoam.o -L/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib \ Code:
vim etc/bashrc the libimf.so was found here Code:
/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/mic/libimf.so or should i run Code:
/opt/intel/composer_xe_2013_sp1.2.144/bin/compilervars_arch.sh mic |
Try running the last command like this instead:
Code:
source /opt/intel/composer_xe_2013_sp1.2.144/bin/compilervars_arch.sh mic Beyond this, it does look like you're reaching the step #6 on the first post by asaijo: Quote:
Code:
cd $FOAM_SOLVERS/multiphase/interFoam/porousInterFoam
|
Hi, I've tried that, but still the same errors occured, when I added
Code:
-I/opt/intel/impi/4.1.3.049/mic/include/ Code:
wclean mpi If I tried without using MPI, it shows something like this: Code:
ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) Hi, thanks for replying, here's my full configuration for the last error: Code:
yum install boost-devel |
Quote:
Code:
cd ~/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX Code:
ld -r -o /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o Make/linux64IccDPOpt/sigFpe.o Make/linux64IccDPOpt/sigSegv.o Make/linux64IccDPOpt/sigInt.o Make/linux64IccDPOpt/sigQuit.o Make/linux64IccDPOpt/sigStopAtWriteNow.o Make/linux64IccDPOpt/sigWriteNow.o Make/linux64IccDPOpt/regExp.o Make/linux64IccDPOpt/timer.o Make/linux64IccDPOpt/fileStat.o Make/linux64IccDPOpt/POSIX.o Make/linux64IccDPOpt/cpuTime.o Make/linux64IccDPOpt/clockTime.o Make/linux64IccDPOpt/memInfo.o Make/linux64IccDPOpt/fileMonitor.o Make/linux64IccDPOpt/printStack.o Code:
SOURCE=signals/sigFpe.C ; mpiicpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -O2 -no-prec-div -DNoRepository -DFOAM_USE_INOTIFY -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude -fPIC -mmic -I/opt/intel/impi/4.1.3.049/mic/include/ -I/opt/crtdc/micgnu/include/ -c $SOURCE -o Make/linux64IccDPOpt/sigFpe.o |
Since the "-L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/" doesn't work at all..
we're adding all ".so" files on "cFLAGS" part on the "c" and "c++" file, for example: Quote:
some warning disappear, but some warning still shows up: Code:
x86_64-k1om-linux-ld: warning: libirng.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/intel64/libPstream.so, not found (try using -rpath or -rpath-link) any possible solution for solving those errors? |
at last.. (/^__^)/
this one works: wmake/rules/linux64Icc/c Code:
.SUFFIXES: .c .h Code:
.SUFFIXES: .C .cxx .cc .cpp |
OpenFOAM2.3.0 on Xeon Phi with OpenMPI1.6.5 (could not read symbols)
Dear All,
I would be very grateful if you tell me how you specifically fixed the following problem: wmake libso mpi '/home/manumachu/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/openmpi-1.6.5/libPstream.so' is up to date. + OSspecific/POSIX/Allwmake Found <sys/inotify.h> -- enabling inotify for file monitoring. ld -r -o /home/manumachu/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o Make/linux64IccDPOpt/sigFpe.o Make/linux64IccDPOpt/sigSegv.o Make/linux64IccDPOpt/sigInt.o Make/linux64IccDPOpt/sigQuit.o Make/linux64IccDPOpt/sigStopAtWriteNow.o Make/linux64IccDPOpt/sigWriteNow.o Make/linux64IccDPOpt/regExp.o Make/linux64IccDPOpt/timer.o Make/linux64IccDPOpt/fileStat.o Make/linux64IccDPOpt/POSIX.o Make/linux64IccDPOpt/cpuTime.o Make/linux64IccDPOpt/clockTime.o Make/linux64IccDPOpt/memInfo.o Make/linux64IccDPOpt/fileMonitor.o Make/linux64IccDPOpt/printStack.o ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) ld: Make/linux64IccDPOpt/sigFpe.o: Relocations in generic ELF (EM: 181) Make/linux64IccDPOpt/sigFpe.o: could not read symbols: File in wrong format make: *** [/home/manumachu/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libOSspecific.o] Error 1 Best Regards Manredd |
symlinking ld to the correct one solves that problem:
Code:
# ll /usr/bin/ld* |
Quote:
Facundo |
Hello,
currently, I'm testing OF on two Intel Xeon PHI cards using a native OF build but I cannot reproduce the results of the benchmark obtained by asaijo. I get really bad results for PHI architecture. Did anyone know what code modifications should be done to get the performance improvements described in this thread? Best regards Matthias |
Hi
I followed the setup here as shown from kokizzu but still lots of error messages as : ~OpenFOAM/OpenFOAM-2.2.2/wmake/platforms/linux64Icc/wmkdep: cannot execute binary file .. can any one tell me how to fix that? |
Hello:
I also met this problem, but here my "compilervars_arch.sh" don't have option "mic", I am very sure we have several xeon Phi cards on each node here. is it because of installation of compiler? thanks! Quote:
|
Hi everybody, thank you so much for sharing your Phi/OpenFoam sucesses and difficulties.
I'd like to try the same for naval hydrodynamics and have a 8*Xeon Phi machine at hand (Supermicro 4028, Dual Haswell, eight 31S1P mic's, accessible via Teamviewer). I want the solution to run on the Phi's only, cluster-in-a-box-style: The host OS and processors would only care about the virtual networking and storage then. No need to care about the complicated offloading/coprocessing stuff. I'm both an OpenFOAM and C++ nOOb, and already overloaded with other parts of the project, so it's probably hopeless for me to get this setup to work all by myself. So i'm looking for someone with significant experience who would be interested in a small consulting mission. First step would be to compile OpenFOAM for the mic's, maybe perform some tuning along what others have tried (hints here: https://www.nersc.gov/assets/Uploads...5-OpenFOAM.pdf) and then benchmark a reference case (~20 million cells) against the same load running on the two Haswell host processors. I guess this is only a few days of work for someone who knows his stuff, and it would allow to decide if it's worth pursuing in terms of raw machine performance. (I'd be equally interrested in that kind of test for SU2 if you are familiar with that.) If that succeeds (success meaning that one phi card can more or less do the work of a Haswell CPU), the second step would be to engage in workflow setup. Between the discussions about requirements, installation, testing, writing up (nothing formal needed, i just need to memorize what to do and what to keep in mind) and explaining i'd expect this to take around a couple weeks of work. A typical marine case would be 2DOF or 6DOF with or without waves, including some kind of turbulence modelling. Apart from the large hull 15m there will also be small apendages (think .5 to 3m) behaving like airfoils. Results will be graphical pressures, velocities, flowlines on the hull, and tables with resistance, position, accelerations. I don't necessarily need absolute data because i can calibrate against known stuff. Context: I'm a lonesome entrepreneur doing the feasibility homework of my startup project. At this stage i have more or less proven my design (on a commercial cfd package) and am now into optimizing it. Later on i'll have it cross-checked by more knowledgable/better equipped hydrodynamics consultants. So there is no big money involved at this stage, and even when actually launched my project would become a small business with a handful of people. However the months of sheer cost of living and computing expenses start to pile up to significant amounts, and since you could save me months i'd be happy to honor that. We could of course also talk about machine time if you have heavy calculations to do yourself. Also i'll probably have other stuff to do later on if we get on well. The thread is probably not the best place to discuss details so please private message me or drop me a mail at davidtimide <at> gmail <dot> com if interested! Thanks for your kind attention and a nice day to all! |
Hi All,
When i build openfoam for xeonphi, I am getting the following error. Can some one please tell me what could be the problem. ld: Relocatable linking with relocations from format elf64-k1om (Make/linux64IccDPOpt/sigFpe.o) to format elf64-x86-64 (/share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/lib/linux64IccDPOpt/libOSspecific.o) is not supported make: *** [/share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/lib/linux64IccDPOpt/libOSspecific.o] Error 1 + wmake libso OpenFOAM wmakeLnInclude: linking include files to ./lnInclude /bin/sh: /share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/wmake/bin/linux64Icc/wmkdep: cannot execute binary file /bin/sh: /share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/wmake/bin/linux64Icc/wmkdep: cannot execute binary file /bin/sh: /share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/wmake/bin/linux64Icc/wmkdep: cannot execute binary file /bin/sh: /share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/wmake/bin/linux64Icc/wmkdep: cannot execute binary file |
Hi All,
The above linking problem resolved after adding LD flags(-melf_k1om) in make file. Thanks Sivaramakrishna |
Quote:
I appear to be having the same problem (error linking from format elf64-k1om to elf64-x86-64). Are you able to elaborate on how exactly you added that flag? For example, what file it's added to and the syntax? Thanks, Tom |
Hi,
Add -melf_k1om flag to LD_FLAGS in make files c, c++. If that does not work change symbolic link of ld to ld of mic architecture to resolve linking errors. Thanks Sivaramakrishna |
Quote:
Code:
wmake/rules/linux64Icc Doesn't work for me. Neither does it work when I say in Code:
etc/bashrc Code:
export WM_CXXFLAGS=$WM_CFLAGS Code:
./Allclean |
Compiling easy? Not at all!
Hi Guys,
Quote:
Question: Do I need to compile this freaky thirdParty Metis, ParMGridGen, Libccmio and stuff in order to make OpenFoam or Foam-Extend usable? For me, I solved many issues for the compiling procedure (Cross-compilation for Xeon Phi Co-Processor 3110) resulting in a super long manual, but with this Metis and libccmio Stuff I run into strange linker issues that seem to stem from the insufficient configure scripts for qmake that do not implement my -mmic option but use their own with -m64. When I try to compile Foam-Extend without this third-party stuff, libfoam.so compiles incomplete and so does the rest of course. I would like to leave all this thirdparty stuff out because fixing the rpm_make routine --- ahem --- s*cks. Or do I do a mistake? --- Edit: Ok, I left out the ThirdParty stuff and I was able to compile at least the sources /src without scotch, metis and parmidgren. But going for the applications, I also get this error: Code:
k1om-mpss-linux-ld: warning: libimf.so, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libinterfaceProperties.so, not found (try using -rpath or -rpath-link) Code:
#echo $LD_LIBRARY_PATH |
All times are GMT -4. The time now is 21:17. |