CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Programming & Development (http://www.cfd-online.com/Forums/openfoam-programming-development/)
-   -   MPI profiling (http://www.cfd-online.com/Forums/openfoam-programming-development/119369-mpi-profiling.html)

GeraBarna June 15, 2013 17:45

MPI profiling
 
Hello everyone!

I am a student relatively new to OpenFOAM writing my thesis, and for that I need to somehow determine the load-balancing of processors while running parallel solvers in OpenFOAM 2.2. My first idea was to use some MPI profiling tool to measure the time each separate process spends either calculating or communicating, but I cannot get any tool running.

Does anybody have some other idea on how to get some measurement of the load balancing on the processors while running solvers, or some idea on what combinations of mpi implementations and profiling tools might work?

So far I have tried to use: valgrind, fpmpi, Tau, MPE, ipm, mpip with OpenMPI 1.6.3 and MPICH 1.1.1p1 and 1.4.1.1p1.
Valgrind does work, but because of the huge overhead it introduces on the calculations it simply is not feasible.

My problem with fpmpi and Tau is that the MPI versions coming with openFoam ( OpenMPI and MPICH ) disable the mpi profiling interfaces and if I re-enable them the the configure scripts break, thus I cannot use them.
MPE: I canot link the libraries, because I get a linker error:

/usr/bin/ld: /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/liblmpe.a(log_mpi_core.o): relocation R_X86_64_32S against `.bss' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/liblmpe.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status

recompiling MPE and the Pstream library with -fPIC didn't solve the problem.

ipm, mpip: the configure script breaks.
I have also tried using googlePerformance tools but they generate segmentation faults at runtime.

My current idea is to use the linux time command to get a measurement of the CPU time from each process, or to put some time measurements into the PStream library, but I don't know how accurate these can be, thus I would like to have a more precise method.

Any new ideas are much appreciated :)

akidess June 17, 2013 03:17

Did you read this: https://www.hpc.ntnu.no/display/hpc/...filing+Solvers

GeraBarna June 17, 2013 06:59

Thank you for your answer!
Unfortunately, I don't even get that far with ipm, because the configure script breaks, so I cannot compile and install it.

Here is my attempt: ( the script breaks at the same point without the extra options too)
Code:

> ./configure --with-compiler=GNU --with-os=LINUX --with-cpu=XEON 2>&1 | tee log.configure
configuring for unknown w/ OS=LINUX ARCH= COMPILER=GNU CPU=XEON
checking whether the C compiler ( gcc ) works...yes
checking whether the C++ compiler ( g++ ) works...yes
determining how to compile HPM (PAPI) C code ...no
determining how to compile HPM (PMAPI) C code ...no
determining how to compile HPM (DISABLED) C code ...yes
determining how to link HPM (DISABLED) code ( -lpapi)...no
determining how to link HPM (DISABLED) code ( -lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code ( -lpapi -lpfm)...no
determining how to link HPM (DISABLED) code (-lpapi)...no
determining how to link HPM (DISABLED) code (-lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code (-lpmapi)...no
determining how to link HPM (DISABLED) code (64 -lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code (64 -lpapi)...no
determining how to link HPM (DISABLED) code (64 -lpapi -lpfm)...no
*** HPM link files not found, HPM disabled ***
Generating ../include/ipm_calls.h
Generating ../include/ipm_fproto.h
Generating libtiny.c
Generating libipm.c
Generating libipm_io.c
checking for working IPM macros in ipm.h...yes
checking for working gettimeofday...yes
checking for working IPM serial timer macro (RTC) in ipm.h...no
checking for working IPM serial timer macro (READREALTIME) in ipm.h...no
checking for working IPM serial timer macro (RTS) in ipm.h...no
checking for working IPM serial timer macro (ITC) in ipm.h...no
checking for working IPM serial timer macro (RDTSC) in ipm.h...yes
checking for working MPI C compiler (cc) ...no
checking for working MPI C compiler (cc -lmpi) ...no
checking for working MPI C compiler (mpcc) ...no
checking for working MPI C compiler (mpicc) ...yes
checking whether the MPI C compiler (mpicc  -DIPM_DISABLE_EXECINFO -I/home/barna/ipm/include    -L/home/barna/ipm/lib  ) works with IPM macros...yes
checking MPI_STATUS_COUNT (val1)...no
checking MPI_STATUS_COUNT (count)...no
checking MPI_STATUS_COUNT (_count)...no
checking MPI_STATUS_COUNT (size)...no
unknown MPI_STATUS_COUNT


AlmostSurelyRob June 17, 2013 07:27

Barna,

you need to look up the configure file. Apparently the code for MPI_STATUS_COUNT doesn't compile. I made it to work by replacing this bit of code
Code:

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
MPI_Status s;
s.$tag = 0;
return

with this

Code:

cat >> ./$TEST_NAME.c <<EOF
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
MPI_Status s;
s.MPI_TAG = 0;
return

It's on the line 1252 in my configure.

Now the configure works, but make breaks. Let me know how it works for you.

AlmostSurelyRob June 17, 2013 07:49

I compiled ipm on blackbird if you're interested. There's something strange about OpenMPI and IPM. The code of IPM assumed that MPI_Status structure should contain completely different fields than OpenMPI offers. I consulted mpi.h to see it and after a bit of "fun" I replaced

Code:

#define MPI_STATUS_COUNT size
Code:

#define MPI_STATUS_COUNT _ucount
in ipm/include/config.h.

Let me know if this works at all. From the comments it seems that these are internal bits of OpenMPI and shouldn't be touched. I compiled sucessfuly, but I have no idea if it works as intended.

GeraBarna June 17, 2013 08:53

Thank you so much!
I have managed to compile and run a solver linked to ipm with these workarounds. I will now try to do some actual profiling, and see if it works properly.

GeraBarna June 20, 2013 09:34

Thank you for the tipps for IPM compiling. Sadly, even though IPM did compile, I couldn't use it. But since then I have figured out how to fix the Profiling interface in the openMPI version that comes with OpenFOAM 2.2

As it turns out, even though it is specified on the openMPI website that it only uses cuda if one explicitly enables it with the -with-cuda= option, it does use it to compile vampire trace. At cuda 5 they changed the interface, thus vampire trace no longer compiles. Thus the solution to enable the profiling interface was to replace the:
--disable-mpi-profile
with:
--disable-vt
also, I had to remove the -j $WM_NCOMPPROCS
from line 109 : make -j $WM_NCOMPPROCS && make install

After this IVP still didn't work, but I could manage to get TAU to work with a reasonable amount of effort.

Sethu July 9, 2014 23:30

Dear Barnabas,

Could you please tell me the procedure that you have used to implement "tau" or "ipm" in your system and to profile openfoam with it.

Im using OF 22x in centOS 6.4 cluster...

Thanks in advance,


All times are GMT -4. The time now is 13:48.